Loop Engineering for Listing Questions: When the Answer Is Every Passage, Not the Top One
Loop Engineering for Listing Questions: When the Answer Is Every Passage, Not the Top One
列表问题的循环工程：当答案存在于每一段落而非单一顶层段落时
Enterprise Document Intelligence [Vol.1 #12] – The category of question most RAG pipelines silently fail on, and the pipeline shape that handles them 企业文档智能 [第1卷 #12] —— 大多数 RAG 流水线会静默失败的问题类别，以及处理此类问题的流水线架构
Ask your pipeline to “list every exclusion in this policy” and watch what comes back: a clean, confident list of five exclusions, nicely formatted, each one real. The policy has nine. Nothing in the answer hints that four are missing, and the user has no reason to double-check a list that looks this tidy. Listing questions break the one assumption retrieval is built on, that the answer is the top passage. Here the answer is every passage. 让你的流水线“列出本政策中的每一项除外责任”，看看会返回什么：一份干净、自信的五项除外责任列表，格式精美，每一项都是真实的。但该政策实际上有九项。答案中没有任何迹象表明缺失了四项，用户也没有理由去复核一份看起来如此整洁的列表。列表类问题打破了检索系统所依赖的一个核心假设，即“答案存在于最相关的段落中”。而在这种情况下，答案存在于每一个相关段落中。
This article is part of Part III of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks: document parsing, question parsing, retrieval, and generation. It handles listing questions: detection, three aggregation strategies, and the completeness signal that says when the list is done. 本文是《企业文档智能》第三部分的内容，该系列旨在通过四个基石构建企业级 RAG 系统：文档解析、问题解析、检索和生成。本文将探讨如何处理列表类问题：包括检测、三种聚合策略，以及用于判断列表何时完成的完整性信号。
Most RAG benchmarks measure performance on factual lookup questions: “what is the effective date?”, “what is the BLEU score?”, “who is the policyholder?”. One question, one passage, one answer. The pipeline retrieves the right chunk, the LLM extracts the value, done. One category of question doesn’t fit this shape: “What are all the subcategories of GOVERN?” “What are all the regularization techniques used to train the Transformer?” “What are all the obligations of the seller in this contract?” “What are all the conditions under which this clause does not apply?” These are listing questions. The answer isn’t in one passage. It’s distributed across the document. 大多数 RAG 基准测试衡量的是事实查询问题的表现：“生效日期是什么？”、“BLEU 分数是多少？”、“投保人是谁？”。一个问题，一个段落，一个答案。流水线检索到正确的块，大模型提取数值，任务完成。但有一类问题不符合这种模式：“GOVERN 的所有子类别是什么？”、“训练 Transformer 使用的所有正则化技术有哪些？”、“合同中卖方的所有义务是什么？”、“该条款不适用的所有条件是什么？”这些都是列表类问题。答案不在某一个段落中，而是分布在整个文档里。
The pipeline that returns the top-k most similar chunks misses items because the top-k doesn’t span the whole list. The LLM that reads the top-k produces an answer that looks complete but isn’t, because it confidently lists what it sees and stays silent about what it doesn’t. This article is about building pipelines that handle listing questions explicitly. The retrieval shape is different, and so is the completeness check. 返回最相似的前 K 个块的流水线会遗漏项目，因为前 K 个块无法覆盖整个列表。读取这前 K 个块的大模型生成的答案看起来很完整，实则不然，因为它自信地列出了它所看到的内容，却对未看到的内容保持沉默。本文旨在构建能够显式处理列表类问题的流水线。其检索模式与完整性检查方式都与传统方法不同。
1. Why listing breaks naive RAG
1. 为什么列表类问题会击垮简单的 RAG
1.1 The silent failure: five out of six, full confidence
1.1 静默失败：六中之五，满怀自信
Take a simple listing question on the NIST CSF: “What are all the categories under the GOVERN function?” The right answer is a list of six categories: Organizational Context (GV.OC), Risk Management Strategy (GV.RM), Roles, Responsibilities, and Authorities (GV.RR), Policy (GV.PO), Oversight (GV.OV), and Cybersecurity Supply Chain Risk Management (GV.SC). 以 NIST CSF（网络安全框架）的一个简单列表问题为例：“GOVERN 功能下的所有类别是什么？”正确答案应包含六个类别：组织背景 (GV.OC)、风险管理策略 (GV.RM)、角色、责任与权限 (GV.RR)、政策 (GV.PO)、监督 (GV.OV) 以及网络安全供应链风险管理 (GV.SC)。
A naive RAG pipeline does this: Embeds the question. Retrieves top-5 chunks by similarity. Sends them to the LLM with a generation prompt. What comes back from retrieval is often the introduction to GOVERN (page 17, the bulleted list of category descriptions) plus a few paragraphs around it. That’s enough for the LLM to list four or five categories, but it might miss GV.SC if the supply chain mentions are clustered in a different chunk that didn’t make the top-k. 一个简单的 RAG 流水线会这样做：嵌入问题，通过相似度检索前 5 个块，将它们连同生成提示词一起发送给大模型。检索返回的结果通常是 GOVERN 的介绍（第 17 页，带有类别描述的列表）以及周围的几个段落。这足以让大模型列出四到五个类别，但如果关于供应链的提及集中在另一个未进入前 K 名的块中，它可能会漏掉 GV.SC。
The LLM then produces: “The categories under GOVERN are: Organizational Context, Risk Management Strategy, Roles and Responsibilities, Policy, and Oversight.” Five out of six. Looks like a complete answer. The user has no way to know that GV.SC is missing unless they cross-check against the document. 大模型随后生成：“GOVERN 下的类别包括：组织背景、风险管理策略、角色与责任、政策和监督。”六中之五。看起来像是一个完整的答案。除非用户对照文档进行核对，否则无法知道 GV.SC 被遗漏了。
This failure isn’t an artefact of top-k retrieval. The Needle-in-a-Haystack benchmark measures one needle, one haystack, one verbatim sentence to find. Frontier models score near-perfectly with long context, which is real and useful. A listing question is six needles, scattered through the corpus, none verbatim because each item is named differently in different places. The benchmark doesn’t test that shape. 这种失败并非前 K 个检索的特有产物。“大海捞针”（Needle-in-a-Haystack）基准测试衡量的是在一个干草堆中寻找一根针、一个逐字句子的能力。前沿模型在长上下文处理上得分近乎完美，这确实很有用。但列表类问题是“六根针”，散布在整个语料库中，且没有一个是逐字相同的，因为每个项目在不同地方的命名方式各异。该基准测试并未测试这种模式。
Long-context-only models hit the same wall as naive top-k: they return four or five, miss the one phrased differently, and present the truncated list with full confidence. Long-context models and top-k fail here for the same reason: the question asks for every item, not the top one. 仅依赖长上下文的模型会遇到与简单前 K 检索相同的障碍：它们返回四或五个项目，漏掉表述不同的那一个，并自信地呈现出截断后的列表。长上下文模型和前 K 检索在这里失败的原因相同：问题要求的是“每一项”，而不是“最相关的一项”。
1.2 Top-k is structurally wrong for listing
1.2 前 K 检索在结构上不适用于列表问题
The retrieval task has a different shape for listing. 对于列表类问题，检索任务具有完全不同的形态。