Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

用于大模型辅助系统综述筛选的辅助不确定性信号：基于八项 Cohen 药物类别综述的基准测试

Abstract: Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty. We show that an auxiliary BERT+GCN classifier supplies a structured uncertainty signal that improves LLM screening efficiency, and we identify the prompt-delivery strategy that maximises the benefit-to-cost ratio.

摘要： 大语言模型（LLMs）正越来越多地被用于系统综述中的标题与摘要筛选，但其决策缺乏经过校准的不确定性评估。我们证明，一种辅助的 BERT+GCN 分类器可以提供结构化的不确定性信号，从而提高大模型筛选的效率；同时，我们确定了能够最大化性价比的提示词交付策略。

We evaluate five LLM prompt-delivery conditions on eight drug-class datasets from the Cohen (2006) benchmark using 3 seeds x 5-fold stratified cross-validation (600 fold-level results). A BERT+GCN model trained per fold classifies each test paper as INCLUDE, EXCLUDE, or MAYBE via two spectral tests (algebraic radical and categorical paradox). Conditions vary information content (none / label / full scores), selectivity (all papers vs. MAYBE only), and timing (proactive vs. reactive two-pass).

我们使用 3 个随机种子和 5 折分层交叉验证（共 600 个折级结果），在 Cohen (2006) 基准测试的八个药物类别数据集上评估了五种大模型提示词交付条件。每个折叠中训练的 BERT+GCN 模型通过两个谱测试（代数根和分类悖论）将每篇测试论文分类为“纳入”（INCLUDE）、“排除”（EXCLUDE）或“可能”（MAYBE）。实验条件涵盖了不同的信息内容（无/标签/完整分数）、选择性（所有论文 vs. 仅 MAYBE）以及时序（主动 vs. 被动两阶段）。

Three findings: (i) Full-context delivery yields significant gains in F1 (+0.011, paired Wilcoxon p=0.008) and WSS@95 (+0.050, p=0.039) at a 1.28x token-cost premium, while preserving recall. (ii) MAYBE-only routing is Pareto-optimal: highest mean recall (0.92) and AUC-ROC (0.54) at only 1.05x baseline cost — one sixth of full-context overhead. (iii) The two-pass design escalates 22.2% +/- 8.8% of records yet never revises its decision (0% flip rate across all datasets and folds), giving decisive evidence that current instruction-tuned LLMs cannot self-triage.

研究发现有三点：(i) 全上下文交付在保持召回率的同时，显著提升了 F1 值（+0.011，配对 Wilcoxon p=0.008）和 WSS@95（+0.050，p=0.039），但 token 成本增加了 1.28 倍。(ii) “仅 MAYBE 路由”策略是帕累托最优的：在仅为基准成本 1.05 倍的情况下（仅为全上下文开销的六分之一），实现了最高的平均召回率（0.92）和 AUC-ROC（0.54）。(iii) 两阶段设计虽然将 22.2% +/- 8.8% 的记录进行了升级处理，但从未修改过其决策（在所有数据集和折叠中翻转率为 0%），这提供了确凿证据，表明当前经过指令微调的大模型无法实现自我分诊。

The cross-model pilot shows an identical +0.8% recall uplift for both LLM generations. A per-paper ablation across 20,796 observations shows the dual paradox test reduces empirically to a one-line logit-gap criterion. We release the full pipeline; the 600-run experiment replays in under one hour from cached LLM responses.

跨模型试点显示，两种大模型代际的召回率均提升了 0.8%。针对 20,796 个观测样本的逐篇消融研究表明，双重悖论测试在经验上可简化为单行 logit 间距准则。我们已发布完整流程；基于缓存的大模型响应，600 次实验的重现时间不到一小时。