Calibrated Selective Fact-Checking via Evidence Chain Evaluation

通过证据链评估实现校准的选择性事实核查

Large language models (LLMs) can achieve strong fact-checking accuracy, yet forced binary decisions conceal a critical reliability problem: systems may issue confident verdicts even when supporting evidence is weak, sparse, or internally inconsistent.

大型语言模型（LLMs）能够实现强大的事实核查准确性，但强制性的二元决策掩盖了一个关键的可靠性问题：即使在支持证据薄弱、稀疏或内部不一致的情况下，系统也可能给出自信的结论。

We address this issue through Evidence Chain Evaluation (ECE), a selective fact-checking framework that permits abstention via an uncertain verdict instead of requiring a true/false decision for every claim. The evaluated system is a tool-using verification agent that gathers evidence through web search, scholarly search, and executable checks, and then returns a structured verdict with confidence and source-level metadata.

我们通过“证据链评估”（Evidence Chain Evaluation, ECE）解决了这一问题。这是一个选择性事实核查框架，它允许通过“不确定”的结论进行弃权，而不是要求对每一项声明都做出真/假判断。该评估系统是一个使用工具的验证代理，通过网络搜索、学术搜索和可执行检查来收集证据，并返回带有置信度和来源级元数据的结构化结论。

On ECE-Bench, ECE achieves 91.6% standard accuracy, 93.7% coverage, and 97.8% selective accuracy on answered claims. Although ECE does not outperform the strongest retrieval baseline on aggregate calibration metrics such as Expected Calibration Error, Brier score, or AURC, it delivers a clear selective-prediction trade-off: the system maintains very high accuracy on answered claims while deferring 6 of 95 cases.

在 ECE-Bench 测试中，ECE 在已回答的声明上达到了 91.6% 的标准准确率、93.7% 的覆盖率和 97.8% 的选择性准确率。尽管 ECE 在预期校准误差（ECE）、Brier 分数或 AURC 等总体校准指标上并未超过最强的检索基准，但它提供了一种清晰的选择性预测权衡：系统在保持已回答声明极高准确性的同时，在 95 个案例中弃权了 6 个。

These deferred cases are concentrated in lower-reliability evidence settings (5/6 at source level L4), supporting the view that abstention functions as a safety-oriented mechanism for handling epistemically weak evidence. Code is available at this https URL cheshireyang/ECE.git.

这些弃权案例集中在可靠性较低的证据环境中（6 个中有 5 个处于 L4 来源级别），这支持了这样一种观点：弃权作为一种面向安全的机制，能够有效处理认知上薄弱的证据。代码可在以下链接获取：https://github.com/cheshireyang/ECE.git。