Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

更健康的语言模型：用于公共卫生问答的检索增强生成技术

Abstract: Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance. Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in an explicitly maintained corpus, but end-to-end performance depends critically on retrieval configuration and on evaluation beyond multiple-choice formats.

摘要： 大型语言模型（LLMs）在医学问答基准测试中取得了令人瞩目的成果，但其在公共卫生领域的应用受到“幻觉”问题以及官方指南快速更新的限制。检索增强生成（RAG）通过将回答建立在明确维护的语料库基础上，缓解了这些风险；然而，端到端的性能在很大程度上取决于检索配置，以及对多项选择格式之外的评估能力。

We extend PubHealthBench, a question answering (QA) benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluate retrieval and generation choices. We compare dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, and show that hybrid retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance.

我们将 PubHealthBench（一个包含 7,929 个源自英国政府公共卫生指南的问题的问答基准）扩展到了检索增强环境，并系统地评估了检索和生成的选择。我们比较了多种嵌入模型和语料库变体下的稠密（dense）、稀疏（sparse）和混合（hybrid）检索，结果表明混合检索能持续提高召回率和排序质量，且分块长度（chunk length）和主题与排序性能之间存在相互作用。

Providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or outperform larger models used without retrieval, with gains primarily driven by retrieval quality and careful context selection.

提供检索到的上下文显著提高了各类 LLM 在多项选择题上的准确率，使较小的开源权重模型能够达到甚至超过不使用检索的大型模型，其性能提升主要得益于检索质量和精细的上下文选择。

To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, and validate it against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale.

为了评估真实的自由形式回答，我们引入了一种基于准则的“LLM 作为裁判”（LLM-as-a-judge）方法，涵盖了忠实度、完整性、清晰度和事实一致性，并将其与双重人工标注进行了验证。结果显示，裁判与人类在忠实度和完整性上的共识度最高，而事实一致性和清晰度的复现可靠性较低，这提醒我们在大规模解释这些维度时需保持谨慎。

Overall, our results highlight retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.

总体而言，我们的研究结果强调了检索是实现可靠公共卫生问答的主要杠杆，并为构建和评估基于官方指南的 RAG 系统提供了实践指导。