RAG Evaluation: How to Know Your Retrieval Isn't Broken

查看原文 / View Original 编辑 / Edit

RAG Evaluation: How to Know Your Retrieval Isn’t Broken

RAG 评估：如何判断你的检索系统是否“坏了”

The metric taxonomy for RAG — retrieval metrics, generation metrics, the eval sets you build, and the numbers that tell you which layer is failing. I was reviewing a team’s RAG demo at a meetup last year. The presenter asked a question, the system retrieved a chunk, and the LLM produced a fluent, well-cited answer. It looked perfect. I asked the question nobody in the room could answer: “What is your hit rate?” The presenter blinked. “What do you mean, hit rate?” RAG 的指标分类包括：检索指标、生成指标、你构建的评估集，以及那些能告诉你哪一层出现故障的数据。去年在一次聚会上，我正在审查一个团队的 RAG 演示。演示者提出了一个问题，系统检索到了一个数据块，LLM 随后给出了一个流畅且引用规范的回答。看起来非常完美。我问了一个在场没人能回答的问题：“你们的命中率（Hit Rate）是多少？”演示者愣住了，反问：“命中率是什么意思？”

We checked their logs. The retrieval was returning the correct chunk about 30% of the time — and because the demo questions were chosen for answers the system could find, nobody had noticed. That is the whole story of RAG evaluation in one incident: fluent output hides broken retrieval. The LLM is a confident text generator, so a wrong chunk in the context produces an answer that sounds exactly as good as a right one. The only way to know which is which is to measure the layers independently, with metrics, on data you control. 我们检查了他们的日志。结果发现，该系统的检索准确率仅为 30% 左右——由于演示问题是专门挑选的、系统能够找到答案的问题，所以没人注意到这一点。这一个案例就道尽了 RAG 评估的全部真相：流畅的输出掩盖了失效的检索。LLM 是一个自信的文本生成器，因此即使上下文中的数据块是错误的，它生成的回答听起来也和正确的一样好。要区分好坏，唯一的办法就是利用指标，在受控数据上对各个层级进行独立测量。

Why You Cannot Just “Look at Answers”

为什么你不能只靠“看回答”来评估

The trap is that humans are terrible at evaluating RAG output by eyeballing ten examples. Three reasons: 陷阱在于，人类通过粗略查看十个示例来评估 RAG 输出的效果是非常糟糕的。原因有三：

  1. Fluency is not correctness. The model writes with equal confidence whether the retrieved chunk is right or wrong. A human reader cannot distinguish “grounded in the right source” from “plausible-sounding” without checking the source themselves — and nobody does that for every answer.

  2. 流畅不代表正确。 无论检索到的数据块是否正确，模型都会以同样的自信进行写作。如果不亲自核对来源，人类读者无法区分“基于正确来源”和“听起来合理”的区别——而没人会为每一个回答都去核对来源。

  3. You cannot see the failure that did not happen. A demo question that happens to retrieve well tells you nothing about the 90% of real questions that do not. The only way to see the failure distribution is to run a fixed, labelled set repeatedly.

  4. 你无法看到那些“未发生”的故障。 一个恰好检索成功的演示问题，无法反映出 90% 检索失败的真实问题。要观察故障分布，唯一的办法是反复运行一个固定的、已标注的测试集。

  5. The failure lives in the retrieval layer, not the output. If retrieval returns the wrong chunk, no amount of prompt or model tuning fixes the answer. To know where to invest, you must be able to score retrieval and generation separately. A single “did the answer look good” number tells you nothing about where to fix things.

  6. 故障存在于检索层，而非输出层。 如果检索返回了错误的数据块，无论如何调整提示词或模型，都无法修正回答。要明确投入方向，你必须能够分别对检索和生成进行评分。单一的“回答看起来好不好”的指标，无法告诉你该从哪里着手修复。

The Metric Taxonomy

指标分类

RAG evaluation splits into two families, and the biggest mistake I see is teams measuring one family and calling it done. RAG 评估分为两大类，我见过的最大错误就是团队只测量其中一类就认为大功告成了。

Retrieval Metrics — “Did we find the right source?” 检索指标——“我们找到正确的来源了吗？”

  • Hit rate / Recall@k: Out of all queries, what fraction had the correct document in the top-k retrieved results? This is the single most important number in all of RAG. If your hit rate at k=5 is below 80%, your retriever is the problem, full stop.
  • 命中率 / Recall@k： 在所有查询中，有多少比例的正确文档出现在前 k 个检索结果中？这是 RAG 中最重要的单一指标。如果你的 k=5 命中率低于 80%，那么问题就在检索器上，毫无疑问。
  • MRR (Mean Reciprocal Rank): Where in the ranked list did the correct document appear? A system that always returns the right chunk at position 1 has MRR 1.0; one that buries it at position 4 is worse even if it still “hits.” Latency and context budget reward high ranks.
  • MRR（平均倒数排名）： 正确文档出现在排名列表的什么位置？一个总是将正确数据块排在第 1 位的系统，其 MRR 为 1.0；而将其排在第 4 位的系统，即使“命中”了，效果也更差。延迟和上下文预算都要求高排名。
  • Precision@k: Of the k retrieved chunks, what fraction were relevant? This matters because irrelevant chunks cost tokens and confuse the model.
  • Precision@k（精确率）： 在检索到的 k 个数据块中，有多少比例是相关的？这一点很重要，因为不相关的数据块会消耗 Token 并干扰模型。

Generation Metrics — “Did the answer use the source correctly?” 生成指标——“回答是否正确使用了来源？”

  • Faithfulness: What fraction of the claims in the answer are supported by the retrieved context? This is the “is it grounded or hallucinating” number.
  • 忠实度（Faithfulness）： 回答中的主张有多少比例是由检索到的上下文支持的？这是衡量“是否有据可依还是在产生幻觉”的指标。
  • Answer relevancy: Does the answer actually address the user’s question?
  • 回答相关性： 回答是否真正解决了用户的问题？
  • Context precision/recall: These measure whether the retrieved chunks were actually needed and sufficient to produce the answer.
  • 上下文精确率/召回率： 这些指标衡量检索到的数据块是否确实必要且足以生成回答。

Building the Eval Set (The Part Everyone Skips)

构建评估集（每个人都会跳过的部分）

Metrics are meaningless without a labelled set. The rule I use: start with 100–200 question–document pairs built from your real logs, not from happy-path examples a developer wrote. 没有标注集，指标就毫无意义。我遵循的原则是：从 100 到 200 个基于真实日志构建的问题-文档对开始，而不是使用开发者编写的“理想路径”示例。

  1. Mine real queries: Pull the last month of actual user questions.
  2. 挖掘真实查询： 提取过去一个月真实的各种用户问题。
  3. Label the ground truth: For each query, identify the document chunks that contain the correct answer.
  4. 标注真值： 对于每个查询，识别出包含正确答案的文档块。
  5. Include the hard cases: Multi-hop questions, questions with exact codes or SKUs.
  6. 包含困难案例： 多跳问题、带有精确代码或 SKU 的问题。
  7. Version the set: It lives in your repo, in git.
  8. 版本化测试集： 将其存放在你的代码仓库（Git）中。
← 返回新闻列表 / Back to news list