Loop Engineering: How to Stop Your Agent Reward-Hacking Its Own Checks

Loop Engineering：如何防止你的 AI Agent 对其检查机制进行“奖励作弊”

You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug. 你给 Agent 一个失败的测试用例，并要求它让测试通过。它确实让测试变绿了。但当你查看代码差异（diff）时发现：它根本没动被测试的代码，而是修改了测试本身。原本断言 == 9000 的地方现在变成了 == 10000，这恰好是那个有 Bug 的函数返回的值。测试之所以变绿，是因为它被修改得去迎合那个 Bug 了。

This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor’s own engineering team published a piece titled “reward hacking is swamping model intelligence gains.” There is a benchmark built to measure it in long-horizon coding agents, SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. 这有一个专门的术语，叫“奖励作弊”（reward hacking），而且这绝非边缘情况下的罕见故障。Cursor 的工程团队曾发表过一篇文章，题为《奖励作弊正在淹没模型智能的提升》。目前已经有一个名为 SpecBench 的基准测试，专门用于衡量长周期编程 Agent 中的这种现象。每一个曾让 Agent 处理失败测试集的开发者，都见过它的各种变体：被删除的断言、被添加的 @pytest.mark.skip、硬编码的返回值，或是被悄悄削弱的关联测试。

The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed. That gap, between the check and what the check stands for, is what this piece is about. Agent 被要求让检查通过，于是它就让检查通过了。没人告诉它“检查”只是“代码正确性”的替代指标，所以它优化的是它实际拿到的那个“检查”。这种“检查”与“检查所代表的真实目标”之间的鸿沟，正是本文要探讨的主题。

A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; three pieces since took the check that decides good enough, stop, the gate that refuses a bad write, and the surface every rule loads from. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model. 一个循环包含五个环节：生成（generate）、检查（check）、引导（steer）、重试（retry）和停止（stop）。本系列的第一篇文章定义了这些环节；随后的三篇文章分别探讨了决定“是否足够好”的检查机制、拒绝错误写入的“门控”（gate），以及规则加载的界面。本文将聚焦于决定 Agent 下一次尝试目标的环节：引导（steer），以及引导机制如何导致模型产生奖励作弊。

What the steer is

什么是“引导”（Steer）

The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check’s output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it: “引导”是将检查结果转化为下一条指令的环节。当检查结果为失败（红灯）时，系统会根据检查的输出汇总成一行文本，并将其输入到下一次生成中。以下是“门控”一文中构建的循环，用于重构 src/ 直到守卫（guard）通过。其中“引导”就是其中的一个环节：

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
    run_agent --task "$prompt" # GENERATE
    if bash no-mocks.sh; then # CHECK
        echo "stop: guard holds after $i retries"; exit 0
    fi
    prompt="The last attempt still tripped the guard; fix it: $(bash no-mocks.sh 2>&1)" # STEER: only the new signal
    i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking. 模型永远看不到完整的历史记录。在每次重试时，它只能看到一个提示词（prompt），而这个提示词是由“引导”环节决定带回来的内容。在第一轮中，提示词是你的原始目标。但在随后的每一轮中，引导环节都会覆盖它。因此，模型在第三次重试时瞄准的目标，并不是你最初写下的目标，而是“引导”环节最后说的话——而这行字是循环在你没注意时自己生成的。

The steer gets none of the attention

“引导”环节往往被忽视

Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer. 奖励作弊的原因不止一个，大多数关注点集中在两个方面：检查机制过于宽松导致可被利用，以及 Agent 对评分标准拥有写入权限。第三个原因几乎无人问津，而这正是本文的主题。它就是循环在每次重试时交给模型的目标，即“引导”。

The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back “make the test pass,” it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back. 模型并不是直接优化检查本身，它优化的是它收到的指令，而指令的内容取决于“引导”环节写了什么。当循环反馈“让测试通过”时，它实际上是将“检查”本身定义为了目标。从那一刻起，优化指令和作弊测试就成了同一件事，因为让测试通过的最廉价方式，就是让测试去迎合代码当前的行为。“引导”说目标是变绿，于是结果就变绿了。

None of this touches the other roads to a gamed result, and it is worth being honest about that, because the Cursor piece above documents one of them. A large share of the reward hacking it found was answer-retrieval: agents pulling a fix straight from a public pull request or the repository’s own bundled git history, with 63% of one model’s successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. 这并没有涵盖所有导致作弊的途径，我们必须诚实地承认这一点，因为前面提到的 Cursor 文章就记录了其中一种。他们发现很大一部分奖励作弊是“答案检索”：Agent 直接从公开的 Pull Request 或仓库自带的 Git 历史中提取修复方案，其中一个模型 63% 的成功解决案例是“检索”出来的，而非“推导”出来的。这种情况即使在目标完全正确的情况下也会发生。这是一个权限问题，而非引导问题，无论如何措辞“引导”都无法阻止它。

The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly. 本文之所以选择“引导”作为切入点，是因为它完全被忽视了，而且它是最容易修复的。你可以在每次重试时亲自编写它，但大多数循环在处理这一步时都写得很糟糕。

The good steer holds the goal

优秀的“引导”应始终保持目标

Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard’s own output and nothing else: 看看上面那个循环带回了什么。目标在循环前只声明了一次，run_agent 调用从未重新发送它。而“引导”环节重写了 prompt，只带回了守卫（guard）的输出，别无他物：

prompt="The last attempt still tripped the guard; fix it: $(bash no-mocks.sh 2>&1)" # STEER: only the new signal

That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check’s own words. 这才是你想要的结构：先给指令，再给证据。修复守卫标记的行，这里是检查机制原封不动提供的错误信息。目标没有改变，因为“引导”从未重述目标，它只是将差异（delta）附加到目标上。模型得到的是原始目标，加上用检查机制自己的话描述的、上一次尝试出错的精确记录。

“Pass the test” is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line. A good steer is a reduction of the check’s output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into “make it pass,” it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game. “让测试通过”永远不是它优化的全部，因为它所服务的原始目标依然在页面上，就在失败行的旁边。一个好的“引导”是对检查输出的精简。它提取结论和产生该结论的最少证据，并将其原样传回。一旦“引导”将失败总结为“让它通过”，它就不再是精简，而变成了一个新的目标——而这个新目标，正是 Agent 将会进行作弊的对象。