Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG

Computer Science > Computation and Language arXiv:2610.40966 (cs) [Submitted on 29 Oct 2026] Title: Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG Authors: Ivan Kochergin, Vasily Konovalov.

计算机科学 > 计算与语言 arXiv:2610.40966 (cs) [提交于 2026 年 10 月 29 日] 标题：更好的检索器，更差的标签：不完整的相关性判断如何扭曲对话式 RAG 中的弃权评估 作者：Ivan Kochergin, Vasily Konovalov。

Abstract: Conversational RAG benchmarks such as MTRAG and RETECO evaluate retrieval and generation end to end: each system answers from its own retrieval, so abstention must be scored against whether each retrieved context is sufficient. A convenient shortcut reads sufficiency off the relevance judgments and treats a context without gold documents as unanswerable.

摘要：MTRAG 和 RETECO 等对话式 RAG 基准测试对检索和生成进行端到端评估：每个系统根据其自身的检索结果进行回答，因此必须根据每个检索到的上下文是否充足来对弃权进行评分。一种便捷的捷径是根据相关性判断来读取充足性，并将没有黄金文档的上下文视为无法回答。

We re-run the official baselines exactly and test this shortcut on RECOR (RETECO track 2) with four retrievers (nDCG@10 from .18 to .59) and three Qwen3 readers. Two open LLM judges and a panel of four small open models label each context by content, and a model of another family audits 250 contexts blind, with adjudication.

我们完全重新运行了官方基线，并使用四个检索器（nDCG@10 从 .18 到 .59）和三个 Qwen3 阅读器在 RECOR（RETECO 赛道 2）上测试了这种捷径。两个开源大语言模型评判员和一个由四个小型开源模型组成的专家组根据内容对每个上下文进行标注，另一个系列的模型则对 250 个上下文进行盲审和裁决。

The stronger the retriever, the larger the share of no-gold contexts that still contain the answer (28% to 72%) and the lower the shortcut’s precision (.68 to .10; .70 to .21 under the audit). A reader agrees: Qwen3-4B answers 62—67% of these false misses correctly, against 23—30% of no-gold contexts judged insufficient and 1% with no documents.

检索器越强，仍包含答案的无黄金文档上下文比例就越高（28% 到 72%），而该捷径的精确度就越低（.68 到 .10；在审计下为 .70 到 .21）。阅读器证实了这一点：Qwen3-4B 能正确回答 62—67% 的此类“假漏”问题，而对于被判定为不足的无黄金文档上下文，其正确率仅为 23—30%，对于完全没有文档的上下文，正确率仅为 1%。

In most false misses, the answer comes from a hard negative mined for the same question, another turn’s gold or a neighbouring chunk of the gold page. On MTRAG, whose judgments come from annotators, precision drops from .72 to .50. Under the shortcut, answerability AUROC is up to .19 too low, and the best sufficiency detector appears to be a retrieval score that never reads the documents.

在大多数“假漏”情况中，答案来自为同一问题挖掘出的困难负样本、另一轮的黄金文档或黄金页面相邻的片段。在判断来自标注者的 MTRAG 上，精确度从 .72 下降到 .50。在这种捷径下，可回答性 AUROC 最多偏低 .19，而最好的充足性检测器似乎是一个从不阅读文档的检索分数。

We release code, runs and labels, and recommend scoring refusals against content labels, not the presence of gold.

我们发布了代码、运行结果和标签，并建议根据内容标签而非黄金文档的存在与否来对拒绝回答进行评分。