I designed a Harness to fix my agent's quality problem — then found 6 flaws in my own design
I designed a Harness to fix my agent’s quality problem — then found 6 flaws in my own design
我设计了一个“质量控制框架”来解决 AI Agent 的质量问题，结果发现了 6 个致命缺陷
In my previous article (I tested 3 models as AI agent quality inspectors: the stronger the model, the more valid work it rejects - DEV Community), I measured three model tiers as agent output quality inspectors across 8 scenarios (4 valid, 4 garbage). The result was a clean precision-recall tradeoff: 在上一篇文章中，我测试了三个不同层级的模型作为 AI Agent 的输出质量检查员，涵盖了 8 种场景（4 个有效，4 个垃圾）。结果呈现出清晰的“精确率-召回率”权衡：
- qwen3:0.5b (weak model): 25% garbage pass-through, 50% false rejections qwen3:0.5b（弱模型）： 25% 的垃圾内容漏过，50% 的误拒率。
- GLM-5.2 (strong model): 0% garbage pass-through, 75% false rejections GLM-5.2（强模型）： 0% 的垃圾内容漏过，75% 的误拒率。
The honest conclusion: a quality gate isn’t a solution — it’s a risk-transfer layer. Each layer catches some failures and introduces new ones. 坦诚的结论是：质量门禁（Quality Gate）并不是解决方案，它只是一个“风险转移层”。每一层都在拦截部分错误的同时，引入了新的错误。
I didn’t stop there. I asked myself: if you accept the human-in-the-loop cost and design a proper Harness — not an automatic fix, but a system that makes human review efficient — what does it look like? I sketched a 4-module architecture: 我没有就此止步。我问自己：如果你愿意承担“人在回路”（Human-in-the-loop）的成本，并设计一个合适的框架（Harness）——不是自动修复，而是一个让人工审核更高效的系统——它会是什么样？我勾勒了一个四模块架构：
- Batch clustering: compress 750 flagged items into 100 groups by failure vector, review one representative per group 批量聚类： 通过失败向量将 750 个被标记的项目压缩为 100 个组，每组审核一个代表。
- Closed-loop calibration: human verdicts → sample pool → scheduled few-shot updates → inspector gets smarter 闭环校准： 人工判定 → 样本池 → 定时少样本（Few-shot）更新 → 检查员变得更聪明。
- Human as gold standard: final arbitration by a trained reviewer 人工作为黄金标准： 由受过培训的审核员进行最终仲裁。
- Asynchronous batching: accumulate flagged items, review in batches 异步批处理： 积累被标记的项目，分批审核。
It looked complete. It looked like progress beyond the “it’s all tradeoffs” conclusion. Then I picked up the same ruler I used on the original production-agent articles, and measured this design. Six flaws. Not one less. 它看起来很完美，似乎超越了“一切皆权衡”的结论。然后，我拿起了在最初的生产环境 Agent 文章中使用的那把尺子，重新衡量了这个设计。结果发现了六个缺陷，一个不少。
Flaw 1: batch clustering — mathematically elegant, operationally dangerous
缺陷 1：批量聚类——数学上优雅，操作上危险
The proposal: “cluster 750 flagged items into 100 groups by failure vector; review one representative per group.” This assumes that “failure modes” can be correctly grouped by embedding clustering. But the 3 falsely-rejected scenarios from the GLM-5.2 experiment had three different reasons: 该方案提出：“通过失败向量将 750 个被标记项目聚类为 100 组；每组审核一个代表。”这假设“失败模式”可以通过 Embedding 聚类正确分组。但 GLM-5.2 实验中被误拒的 3 个场景原因各不相同：
- Scenario A (research brief): content too short (title + one sentence) 场景 A（研究简报）： 内容太短（标题 + 一句话）。
- Scenario B (draft): missing structure (no chapter divisions) 场景 B（草稿）： 缺少结构（没有章节划分）。
- Scenario C (chapter files): too many placeholders (all TODO) 场景 C（章节文件）： 占位符太多（全是 TODO）。
These three have low embedding similarity — one about a research brief, one about a draft, one about file structure. They wouldn’t cluster into the same group. I ran a quick embedding experiment to confirm (Qwen3-embedding:0.6b on the actual 8 scenarios from Experiment E): 这三者的 Embedding 相似度很低——一个是关于研究简报，一个是草稿，一个是文件结构。它们不会被聚类到同一组。我运行了一个快速的 Embedding 实验来验证（使用 Qwen3-embedding:0.6b 对实验 E 的 8 个实际场景进行测试）：
| Pair | Cosine | Meaning |
|---|---|---|
| L1-L2 (brief vs draft, both valid) | 0.610 | Moderate |
| L1-L3 (brief vs chapter, both valid) | 0.295 | Unrelated |
| L2-L3 (draft vs chapter, both valid) | 0.341 | Unrelated |
| Average across all three | 0.415 | Won’t cluster together |
L1 and L3 share a cosine of 0.295 — they’re both false rejects of the strong model, but that shared label doesn’t pull two different topics closer in embedding space. The 750→100 compression ratio has no embedding basis. A practical clustering run would place these three into separate groups, each requiring its own review — the promised compression evaporates. L1 和 L3 的余弦相似度为 0.295——它们虽然都是强模型的误拒案例，但这种共同的标签并不会在 Embedding 空间中拉近两个不同主题的距离。750 到 100 的压缩比没有任何 Embedding 依据。实际聚类运行会将这三者放入不同的组，每一组都需要单独审核——承诺的压缩效果瞬间蒸发。
Worse: if the clustering algorithm accidentally groups 3 real-garbage items with 1 false-rejected valid item, and the reviewer clicks “approve” to correct the false rejection — all 3 garbage items slip through. 更糟糕的是：如果聚类算法意外地将 3 个真正的垃圾项目与 1 个被误拒的有效项目归为一组，而审核员为了纠正误拒点击了“批准”——那么这 3 个垃圾项目就会被漏过。
I found exactly this in the data: scenario L4 (valid test log, “42 passed”) and scenario G4 (garbage zero-case log, “0 passed (no tests collected)”) have a cosine of 0.861 — both are test-log formatted strings, and embedding reads format, not semantics. Clustering will absolutely group them together. A reviewer correcting L4’s false rejection would simultaneously approve G4, pushing the false-positive rate back to 25% — the same as without the strong model. 我在数据中发现了这种情况：场景 L4（有效的测试日志，“42 passed”）和场景 G4（垃圾零用例日志，“0 passed (no tests collected)”）的余弦相似度为 0.861——两者都是测试日志格式的字符串，而 Embedding 读取的是格式而非语义。聚类绝对会将它们归为一组。审核员在纠正 L4 的误拒时会同时批准 G4，将误报率推回 25%——这与不使用强模型时的结果一样。
Verdict: the clustering compression ratio has zero experimental support. It might reduce false rejections from 75% to 0% — but it could simultaneously raise the false-positive rate from 0% to 30%. 结论： 聚类压缩比没有任何实验支持。它或许能将误拒率从 75% 降至 0%，但同时可能将误报率从 0% 提高到 30%。
Flaw 2: closed-loop calibration — shifting the problem to “tuning”
缺陷 2：闭环校准——将问题转移到“调优”上
The proposal: “human clicks → sample pool → scheduled few-shot updates → inspector gradually gets smarter.” This uses a second meta-LLM (the one that selects and formats few-shot examples) to correct the first LLM’s bias. But that meta-LLM has the same temperature-0 instability — Experiment 2 showed open-ended output is only 70% consistent. 该方案提出：“人工点击 → 样本池 → 定时少样本更新 → 检查员逐渐变聪明。”这使用第二个元 LLM（负责选择和格式化少样本示例的 LLM）来纠正第一个 LLM 的偏差。但该元 LLM 同样存在 Temperature=0 的不稳定性——实验 2 表明，开放式输出的一致性仅为 70%。
The “common patterns” it extracts from 100 valid samples might look like: “contains a heading,” “length > 100 characters,” “has paragraph breaks.” If the business requirement changes tomorrow to “write a one-line summary,” these historically learned patterns become a new source of false rejections. A closed loop can only adapt to past data distribution. It cannot handle distribution shift. 它从 100 个有效样本中提取的“通用模式”可能是：“包含标题”、“长度 > 100 字符”、“有段落换行”。如果明天的业务需求变为“写一行摘要”，这些历史学习到的模式就会成为误拒的新来源。闭环只能适应过去的数据分布，无法处理分布偏移（Distribution Shift）。
More fundamentally: there is zero evidence that feeding more few-shot examples linearly reduces false-rejection rates. I tested this. 更根本的是：没有任何证据表明，输入更多的少样本示例能线性降低误拒率。我对此进行了测试。
Setup: qwen3:0.5b, same 8 scenarios (4 valid + 4 garbage), N=5 runs each. 设置： qwen3:0.5b，相同的 8 个场景（4 有效 + 4 垃圾），每项运行 5 次。 Baseline: original prompt. 基准： 原始提示词。 Treatment: same prompt with 3 few-shot examples prepended (including “short but valid content → PASS”). 处理： 相同的提示词，但在前面加上 3 个少样本示例（包括“内容短但有效 → 通过”）。
| Scenario | Baseline false-rejection rate | +Few-shot false-rejection rate | Change |
|---|---|---|---|
| L1 (brief, valid) | 100% | 40% | ✅ improved |
| L2 (draft, valid) | 0% | 100% | ❌ worse |
| L3 (chapter, valid) | 80% | 80% | = |
| L4 (test log, valid) | 20% | 100% | ❌ worse |
| Aggregate false-rejection | 50% | 80% | ❌ |
| Garbage pass-through | 20% | 15% | — |
L1 improved (the brief was exactly the kind of “short but valid” the examples taught). But L2 and L4 — scenarios that were correctly accepted at baseline — both jumped to 100% rejection. G2 (period character) went from 0% to 40% false positive — new holes opened. Few-shot is whack-a-mole: every fix trades off somewhere else. You might feed 500 samples and GLM-5.2 still kills “short but valid” outputs. Its “strictness” bias is at the model-weight level — not something a few in-context examples can overwrite. L1 得到了改善（简报正是示例所教导的“短但有效”类型）。但 L2 和 L4——在基准测试中被正确接受的场景——误拒率都跳升到了 100%。G2（句号字符）的误报率从 0% 升至 40%——出现了新的漏洞。少样本学习就像“打地鼠”：每一次修复都会在其他地方产生权衡。你可能输入 500 个样本，GLM-5.2 依然会杀掉“短但有效”的输出。它的“严格”偏差存在于模型权重层面——这不是几个上下文示例就能覆盖的。
Verdict: I promised the closed loop would calibrate. That promise rests on an unvalidated assumption — that LLM bias is correctable through in-context examples. Experiment 2 already showed that temperature 0 is fundamentally unstable; adding few-shot just adds another layer of instability. 结论： 我承诺闭环可以进行校准。这个承诺建立在一个未经证实的假设之上——即 LLM 的偏差可以通过上下文示例来纠正。实验 2 已经表明 Temperature=0 本质上是不稳定的；添加少样本只会增加另一层不稳定性。
Flaw 3: the reviewer is the “gold standard” — the most subtle lie
缺陷 3：审核员是“黄金标准”——最隐蔽的谎言
Every hu… 每一个人类……