PathReportEval: A Systematic Benchmark for Pathology Report Generation

PathReportEval：病理报告生成的系统性基准

Pathology report generation from whole-slide images (WSIs) is a rapidly growing multimodal learning problem, yet progress is difficult to measure because existing studies use heterogeneous datasets, model settings, visual encoders, and evaluation protocols. 从全切片图像（WSI）生成病理报告是一个快速发展的多模态学习问题，但由于现有研究使用了异构的数据集、模型设置、视觉编码器和评估协议，其进展难以衡量。

Moreover, commonly used natural language generation metrics, including BLEU, ROUGE, and METEOR, primarily reward lexical similarity and often fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes. 此外，常用的自然语言生成指标（包括 BLEU、ROUGE 和 METEOR）主要奖励词汇相似度，往往无法检测出具有临床后果的错误，例如漏诊、幻觉发现或不一致的肿瘤属性。

We present a standardized benchmark and evaluation framework for pathology report generation. The benchmark evaluates four representative methods across three datasets (TCGA, HistAI, and REG 2025) using three pathology foundation encoders (CONCHv1.5, UNI2-h, and H-Optimus-1). 我们提出了一个用于病理报告生成的标准化基准和评估框架。该基准使用三种病理基础编码器（CONCHv1.5、UNI2-h 和 H-Optimus-1），在三个数据集（TCGA、HistAI 和 REG 2025）上评估了四种代表性方法。

Our framework standardizes preprocessing, feature extraction, training, decoding, and evaluation, enabling fair comparison across models while providing a modular platform for integrating new methods, datasets, and encoders. 我们的框架对预处理、特征提取、训练、解码和评估进行了标准化，在实现模型间公平比较的同时，提供了一个用于集成新方法、数据集和编码器的模块化平台。

A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness. CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance, producing both an overall score and interpretable sub-scores. 该研究的核心贡献是临床报告质量评分（CRQS），这是一种用于评估事实正确性的临床基础指标。CRQS 将参考报告和生成报告映射为结构化的临床属性，并衡量四个互补维度：临床事实覆盖率、关键信息召回率、幻觉率和临床不一致性，从而得出总体评分和可解释的子评分。

Experiments demonstrate that conventional language-generation metrics are weakly aligned with clinical correctness and frequently overestimate report quality. In contrast, CRQS reveals clinically meaningful differences between models and encoders that lexical metrics fail to capture. 实验表明，传统的语言生成指标与临床正确性的一致性较弱，且经常高估报告质量。相比之下，CRQS 揭示了模型和编码器之间词汇指标无法捕捉到的临床意义差异。

Together, the benchmark, public plug-and-play framework, and CRQS establish a reproducible foundation for rigorous evaluation of pathology report generation. 总之，该基准、公共即插即用框架和 CRQS 为病理报告生成的严格评估奠定了可复现的基础。