Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark

评估设计决定了专家与自动标注 MeSH 的差距：Cohen 基准测试中词袋模型与 BiomedBERT 的对照研究

Abstract: A systematic review begins with someone reading thousands of abstracts to identify the few that are relevant, and classifiers are used to prioritise that reading. Their inputs are often augmented with Medical Subject Headings (MeSH), assigned either by expert indexers weeks or months after publication or by automatic tools at once. 摘要： 系统综述通常始于研究人员阅读数千篇摘要以筛选出少数相关文献，而分类器则被用于优化这一阅读优先级。这些分类器的输入数据通常会通过医学主题词（MeSH）进行增强，这些主题词要么由专家索引员在论文发表数周或数月后分配，要么由自动工具即时生成。

To our knowledge the two have not been compared directly as classifier features, and no previous work has asked whether that comparison’s outcome depends on how the classifier is evaluated. Using the Cohen et al. (2006) drug-class benchmark on three topics, we characterise a bag-of-words logistic regression classifier (seven reruns) and BiomedBERT (five seeds), then examine how the Statins result changes under alternative designs. 据我们所知，此前尚未有研究直接将这两者作为分类器特征进行对比，也没有研究探讨过该对比结果是否取决于分类器的评估方式。我们使用 Cohen 等人（2006 年）的药物类别基准测试，涵盖三个主题，对词袋逻辑回归分类器（运行七次）和 BiomedBERT（五个随机种子）进行了表征，并考察了在不同设计下“他汀类药物”（Statins）的结果如何变化。

Under the canonical 5-fold full-corpus design, the bag-of-words expert-vs-auto gap on Statins is +0.096 WSS@95%. Matching the corpus size to the smaller topics (n = 803) reduces it to +0.033 (95% bootstrap CI includes zero), and 10-fold cross-validation at full size to +0.021 (CI narrowly excludes zero). Under canonical evaluation BiomedBERT gives +0.020, within sampling noise of the bag-of-words 10-fold result. 在标准的 5 折全语料库设计下，词袋模型在“他汀类药物”主题上专家标注与自动标注的差距为 +0.096 WSS@95%。将语料库规模调整至较小主题的水平（n = 803）后，该差距缩小至 +0.033（95% 自助法置信区间包含零），而在全规模下进行 10 折交叉验证时，差距缩小至 +0.021（置信区间勉强排除零）。在标准评估下，BiomedBERT 的结果为 +0.020，处于词袋模型 10 折结果的采样噪声范围内。

A power analysis indicates a Statins-sized effect would not have been detectable at the Opioids or ADHD variance, so those nulls are design-limited rather than informative. A representation asymmetry remains: 15.1% of Statins inputs exceed BiomedBERT’s 512-token limit when expert MeSH terms are appended, so truncation may contribute to the smaller transformer gap, although this cannot be separated from training volume here. 功效分析表明，在“阿片类药物”或“多动症”的方差水平下，无法检测到“他汀类药物”规模的效应，因此这些零结果是受设计限制的，而非具有信息量。此外，存在一种表征不对称性：当附加专家 MeSH 术语时，15.1% 的“他汀类药物”输入超过了 BiomedBERT 的 512 个 Token 限制，因此截断可能导致了 Transformer 模型差距的缩小，尽管这一点在此处无法与训练数据量区分开来。

In screening pipelines using transformers or 10-fold bag-of-words, the gap on the topics tested is about 0.02 WSS@95%, with CIs spanning zero on at least one bound. More broadly, benchmark conclusions about feature sources can change substantially under reasonable changes to the evaluation design. 在使用 Transformer 或 10 折词袋模型的筛选流程中，所测试主题的差距约为 0.02 WSS@95%，且置信区间至少在一端跨越了零。更广泛地说，关于特征来源的基准测试结论可能会随着评估设计的合理调整而发生实质性变化。