The Model Wrote the Right Rule and My Replay Rejected It: The Extraction-vs-Replay Split

模型写出了正确的规则，但我的重放机制却拒绝了它：提取与重放的分裂

Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. 更新——v0.3.0 版本已发布。CauterRule 现已在 GitHub 和 PyPI 上线。它能将重复的智能体故障转化为永久的固定规则——即提取、重放测试、推广。通过 pip install cauterule，你可以获得完整的 CLI、框架适配器、规则生命周期、包生态系统以及官方规则包。v0.3.0 的实地测试报告评估了 2 个云端模型，涵盖 40 个语料库和 4,768 次轨迹运行，下文中的所有数据均源于此。

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization. For two releases we treated “the pass rate is low” as one problem and reached for the extractor. It turned out to be two problems we had fused into a single number. This is the split, and it changed where we point next. CauterRule 是一个开源的辅助工具（sidecar），旨在从重复的智能体故障中学习固定规则。它从轨迹中提取经验，进行重放测试，并试图将可复用的指导与嘈杂的过度泛化区分开来。在过去两个版本中，我们将“通过率低”视为一个单一问题，并试图通过改进提取器来解决。事实证明，这是两个被我们合并成一个数字的问题。这就是所谓的分裂，它改变了我们接下来的改进方向。

The assumption: one metric, two questions. The pipeline has two halves that answer two different questions: Extraction: given a failure, does the model produce the right rule? Replay / evaluation: given a rule, can we verify it against history? We measured only the second, and we read its score as a verdict on the first. 假设：一个指标，两个问题。该流水线包含两部分，分别回答两个不同的问题：提取（Extraction）：给定一个故障，模型是否产生了正确的规则？重放/评估（Replay / evaluation）：给定一个规则，我们能否根据历史记录验证它？我们之前只测量了后者，却将其得分解读为对前者的判定。

The replay gate that decides promotion computes: precision = prevented / (prevented + broken), recall = prevented / total_failures, where prevented and broken come from simulate(), which calls rule_matches() — a text matcher. So prevented means “the trigger’s prose reached a similarity threshold against a reference failure’s prose,” and broken means “…against a reference success’s prose.” Nothing in that path asks whether applying the rule’s directive would have changed the trajectory’s outcome. The gate is a lexical resemblance check wearing a validation costume. 决定是否推广的重放门控计算公式为：精确率 = 阻止数 / (阻止数 + 破坏数)，召回率 = 阻止数 / 总故障数。其中“阻止数”和“破坏数”来自 simulate() 函数，该函数调用了 rule_matches()——一个文本匹配器。因此，“阻止”意味着“触发器的文本与参考故障文本达到了相似度阈值”，“破坏”意味着“……与参考成功文本达到了相似度阈值”。在这个路径中，没有任何环节询问“应用该规则的指令是否会改变轨迹的结果”。这个门控本质上是一个披着验证外衣的词汇相似度检查。

The F-001 example: the rule was right, the grader said no. In the failures/positive corpus, trajectory F-001 is the canonical git case: expected_rule (ground truth) “when git push fails with non-fast-forward, pull latest changes before pushing” extracted when “when git push fails with non-fast-forward” extracted do “pull latest changes before pushing”. The model reproduced the reference rule almost verbatim. Extraction did its job. Then replay scored it: failures_prevented = 5, successes_broken = 3, near_misses = 1, precision = 0.625, recall = 0.625, verdict = INCONCLUSIVE. F-001 示例：规则正确，但评分器拒绝了它。在故障/正面语料库中，轨迹 F-001 是典型的 git 案例：预期规则（真值）为“当 git push 因非快进（non-fast-forward）失败时，在推送前拉取最新更改”；提取的触发条件为“当 git push 因非快进失败时”，提取的动作指令为“在推送前拉取最新更改”。模型几乎逐字复现了参考规则。提取工作完成得很好。但重放评分显示：阻止故障 = 5，破坏成功 = 3，接近失误 = 1，精确率 = 0.625，召回率 = 0.625，判定结果 = 不确定（INCONCLUSIVE）。

Three “broken” successes were S-022-git-pull, S-023-git-status, and NM-044-git-commit-hook — a successful git status counted as broken by a git-push rule, because the two share the token git. A correct rule was demoted to inconclusive by three unrelated successes it merely rhymes with. The model wrote the right rule. The grader rejected it on the prose. 三个被“破坏”的成功案例分别是 S-022-git-pull、S-023-git-status 和 NM-044-git-commit-hook——一个成功的 git status 被 git-push 规则判定为“破坏”，仅仅是因为两者都包含“git”这个词。一个正确的规则因为三个仅仅在词汇上“押韵”的无关成功案例而被降级为“不确定”。模型写出了正确的规则，但评分器却因为文本表述拒绝了它。

The assumption we made (and where it broke): Assumption: the replay verdict tells us whether the extracted rule is good. It tells us whether the extracted trigger’s surface form resembles stored reference surface forms. Those are different claims. A correct rule phrased differently scores low; a wrong rule that happens to share vocabulary scores high. The metric is a text-similarity proxy, and it is the gate. 我们所做的假设（以及它失效的地方）：假设：重放判定告诉我们提取的规则是否良好。实际上，它只告诉我们提取的触发器表面形式是否与存储的参考表面形式相似。这是两个不同的主张。一个表述不同的正确规则得分很低；一个恰好共享词汇的错误规则得分却很高。该指标只是文本相似度的代理，但它却成为了门控标准。

The telling part: the undercount is not unique to replay. The corpus carries a ground-truth rule (expected_rule) on 23/50 failures/positive trajectories and 288 reference-expansion trajectories — a ready-made extraction-accuracy instrument we were never scoring against. When we do score it naively (token-F1 of the extracted rule vs expected_rule), we get only ~0.58 (llama-3.1-8b) / ~0.50 (gpt-4o-mini). Not because the model is wrong — F-001 is near-verbatim — but because it rewords, and a token comparator can’t see through rewording. The same paraphrase gap that breaks replay also undercounts extraction. We had one brittle text-matching lens in two places. 关键之处在于：这种低估并非重放机制所独有。语料库在 50 个故障/正面轨迹中的 23 个以及 288 个参考扩展轨迹中都带有真值规则（expected_rule）——这是一个现成的提取准确度评估工具，我们却从未利用它进行评分。当我们天真地进行评分（提取规则与预期规则的 token-F1 对比）时，得分仅为 ~0.58 (llama-3.1-8b) / ~0.50 (gpt-4o-mini)。这并非因为模型错了——F-001 几乎是逐字正确的——而是因为它进行了改写，而 token 比较器无法识别改写。导致重放失败的同一种“释义鸿沟”也导致了提取结果的低估。我们在两个地方都使用了同一个脆弱的文本匹配透镜。

What worked: Naming the two halves separately stopped the misattribution. “Low pass rate” is a symptom; “extraction accuracy” and “replay fidelity” are the two variables. Once separated, the model stops taking the blame for the grader. The ground truth was already in the corpus. expected_rule needs no new labeling on failures/positive — it was being parsed out and dropped. Picking it up is the cheapest high-signal metric we have. One worked example did more than any average. F-001 — right rule, inconclusive verdict, three token-sharing successes — is the argument in one line. Averages hid it; the single case exposed the mechanism. 有效的方法：将两部分分开命名，停止了错误的归因。“通过率低”是一个症状；“提取准确度”和“重放保真度”才是两个变量。一旦分开，模型就不再为评分器的缺陷背锅。真值其实已经在语料库中。expected_rule 在故障/正面案例中不需要新的标注——它之前只是被解析出来后丢弃了。重新拾起它，是我们拥有的成本最低且信号最强的指标。一个成功的案例比任何平均值都更有说服力。F-001——正确的规则、不确定的判定、三个共享 token 的成功案例——用一行就说明了问题。平均值掩盖了真相，而单个案例揭示了机制。

What didn’t work: We had a ground-truth rule and never scored against it. expected_rule is dropped at parse (Trajectory has no such field), so the only quality number was the replay verdict. We graded the homework with the wrong rubric for a full release. The replay metric masquerades as validation. “Tested against history before promotion” is the product’s core promise, but the test is lexical resemblance. A reader who believes the promise and inspects the gate finds prose matching. Token-F1 undercounts extraction too. Reaching only ~0.5–0.6 against known-good rules, a token-F1 extraction score would also wrongly suggest the model is mediocre. The comparator needs a semantic or signature signal, or it repeats the replay’s error on the extraction side. Both metrics share one brittle primitive. 无效的方法：我们拥有真值规则却从未针对它进行评分。expected_rule 在解析时被丢弃（轨迹中没有该字段），因此唯一的质量指标就是重放判定。我们在正式发布时使用了错误的评分标准来批改作业。重放指标伪装成了验证手段。“在推广前根据历史记录进行测试”是产品的核心承诺，但测试方式却是词汇相似度。一个相信该承诺并检查门控机制的读者，会发现这只是文本匹配。Token-F1 也低估了提取效果。在已知正确规则的情况下得分仅为 ~0.5–0.6，token-F1 提取分数也会错误地暗示模型表现平庸。比较器需要语义或特征信号，否则它会在提取端重复重放机制的错误。这两个指标都共享同一个脆弱的原始逻辑。