Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

Computer Science > Computation and Language arXiv:2610.20634 (cs) [Submitted on 30 Sep 2026] Title: Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM Authors: Alden Do Rosario, Kirollos Atef.

计算机科学 > 计算与语言 arXiv:2610.20634 (cs) [提交于 2026 年 9 月 30 日] 标题：Jev-as-a-Judge 用于 RAG 声明验证：非生成式验证器与前沿大语言模型之间的置信度门控级联。作者：Alden Do Rosario, Kirollos Atef。

Abstract: Retrieval-augmented generation asks one question at volume: does this evidence support this claim? The answer is one bit, yet most pipelines obtain it from a text generator and pay generation prices for the prose wrapped around it. We compare a non-generative decision model, which returns a typed value from a declared enumeration together with a native probability and no text, against a frontier LLM configured as an engineer would deploy it for binary classification.

摘要：检索增强生成（RAG）在大规模应用中提出了一个核心问题：该证据是否支持该声明？答案仅为一位（one bit），然而大多数流水线却通过文本生成器获取该答案，并为随之生成的文本支付生成费用。我们将一个非生成式决策模型与一个配置为二分类任务的前沿大语言模型（LLM）进行了对比；前者返回一个来自预定义枚举的类型化值及原生概率，且不包含任何文本。

Both answer the same 495 human-labelled claims from LLM-AggreFact, so every comparison is within-item. We report three things. First, the two systems are statistically indistinguishable on the metric this task is conventionally scored by, a paired difference of -0.56 points, Jev minus Astra, at a 187-fold cost difference.

两者均回答了来自 LLM-AggreFact 的 495 个经人工标注的声明，因此每一项对比均在同一条目内进行。我们报告了三点发现。首先，在常规评分指标上，这两个系统在统计学上没有显著差异，Jev 与 Astra 的配对差值为 -0.56 分，而成本差异却高达 187 倍。

Second, that tie carries almost no information about deployment: the systems disagree on 15.2% of claims, and the cheaper one admits 23.2% of the claims annotators marked unsupported against the frontier model’s 14.1%. A component whose job is to catch unsupported claims should be judged on that rate, and the conventional headline hides it.

其次，这种平局几乎不包含关于部署的信息：两个系统在 15.2% 的声明上存在分歧，且较便宜的模型将 23.2% 被标注员标记为“不支持”的声明误判为支持，而前沿模型这一比例为 14.1%。一个旨在捕捉不支持声明的组件应根据该比率进行评估，而传统的头条指标掩盖了这一点。

Third, because the errors differ they compose: routing the claims the cheap model is confident about and escalating the rest reaches 74.10 at USD 2.92 per thousand claims against 73.83 at USD 8.93, with the threshold chosen out of fold. But a matched random-escalation control at in-sample thresholds, spending the same budget by coin flip, does as well, so what buys the saving is the escalation share and not the probability that picks what to escalate.

第三，由于错误类型不同，它们可以组合使用：将廉价模型有把握的声明进行路由，其余部分升级处理，其准确率为 74.10（每千条声明成本 2.92 美元），对比前沿模型的 73.83（每千条成本 8.93 美元），且阈值是在样本外选择的。然而，在样本内阈值下进行的随机升级对照实验显示，通过抛硬币分配预算也能达到同样效果，因此节省成本的原因在于升级的比例，而非用于选择升级对象的概率。

Measured against the cheap model it is built from, the cascade costs 61-fold more for 0.83 points, 95% CI [-3.36, 4.86], and it is the more permissive guard of the two. Equal accuracy conceals unequal errors. All predictions, prompts, pricing and code are released.

与构建该级联所基于的廉价模型相比，级联成本高出 61 倍，仅提升了 0.83 个百分点（95% 置信区间 [-3.36, 4.86]），且它是两者中更宽松的防护机制。相等的准确率掩盖了不相等的错误。所有预测结果、提示词、定价和代码均已发布。