Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
Faithful by Design: Evaluating and Improving LLM-Generated Clinical Trial Summaries for Multi-Stakeholder Audiences
设计忠实度：评估并改进面向多利益相关方的临床试验大模型摘要
Abstract: Large language models are increasingly used to summarize clinical trial results for healthcare providers, patients, and payers, but their tendency to hallucinate poses significant risks in this high-stakes context. This study introduces a benchmark evaluation framework for measuring the faithfulness of LLM-generated clinical trial summaries across three stakeholder audiences.
摘要： 大型语言模型（LLM）正越来越多地被用于为医疗服务提供者、患者和支付方总结临床试验结果，但其产生“幻觉”的倾向在这种高风险背景下构成了重大隐患。本研究引入了一个基准评估框架，用于衡量大模型生成的临床试验摘要在针对三类利益相关方时的忠实度。
The framework consists of 200 stratified trials drawn from the Aggregate Analysis of ClinicalTrials.gov database, evaluated using audience-specific prompt templates and a six-dimension faithfulness annotation schema. Baseline measurements were established for GPT-4o, Claude Sonnet 4.6, and Gemini 2.5 Flash across 1,800 generated summaries scored using a cross-encoder natural language inference (NLI) model.
该框架包含从 ClinicalTrials.gov 数据库的汇总分析中抽取的 200 项分层试验，并使用针对特定受众的提示词模板和六维度忠实度标注方案进行评估。研究为 GPT-4o、Claude Sonnet 4.6 和 Gemini 2.5 Flash 建立了基准测量，涵盖了 1,800 份生成的摘要，并使用交叉编码器自然语言推理（NLI）模型进行评分。
Unsupported Claims was identified as the dominant failure mode across all three models, with a mean annotation score of 1.55 out of three. A knowledge-graph-augmented retrieval system was developed and evaluated against the baseline, producing statistically significant improvements in NLI-based faithfulness scores (entailment +0.0125, faithfulness +0.0130, p < 0.0001).
“无根据的声明”（Unsupported Claims）被确定为所有三个模型中最主要的失败模式，平均标注得分仅为 1.55 分（满分 3 分）。研究开发并评估了一个知识图谱增强检索系统，与基准相比，该系统在基于 NLI 的忠实度得分上产生了统计学意义上的显著提升（蕴含度 +0.0125，忠实度 +0.0130，p < 0.0001）。
Improvement pathways were model-dependent, with GPT-4o improving primarily through contradiction reduction while Claude Sonnet 4.6 and Gemini 2.5 Flash improved through increased entailment.
改进路径因模型而异：GPT-4o 的提升主要通过减少矛盾来实现，而 Claude Sonnet 4.6 和 Gemini 2.5 Flash 的提升则通过增加蕴含度来实现。