I picked a coding agent off a leaderboard. It flopped on our codebase.

我从排行榜上选了一个编程智能体，结果在我们的代码库上翻车了。

Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner. 去年，我的团队需要选择一个编程智能体（coding agent），我主动请缨负责评估工作。当时我感觉良好：我调出了公开的基准测试分数，列出了候选名单，选了排名第一的那一个，并告诉大家我们找到了赢家。

Then we actually pointed it at our repo. It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. 然而，当我们真正把它接入代码库时，它并没有发生什么惊天动地的崩溃，只是不断地犯一些小错，浪费我们的时间。它生成的代码差异（diff）无法通过我们的代码审查；它重命名了一个函数，却破坏了三个它从未打开过的文件。它运行的测试通过了，但代码库实际上已经坏了。

I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation. That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back. 我基于一个数字自信地推荐了一个工具，结果发现这个数字对我们的实际情况几乎毫无参考价值。这让我感到非常尴尬，于是我开始探究原因。经过几周的阅读和几次错误的尝试，我终于找到了一套行之有效的方法。我将这些经验写在这里，希望它能让你免于召开那种不得不收回推荐建议的尴尬会议。

Why the benchmark score lied to me

为什么基准测试分数欺骗了我

The score was not fake. It was just measuring somebody else’s code. Once I looked properly, four gaps explained the whole thing: 分数本身并不虚假，它只是在衡量别人的代码。当我仔细审视后，发现四个差距解释了所有问题：

The agent might have already seen the answers. The problems in these public benchmarks are old. Models were very likely trained on the actual fixes used to grade them. So the score partly measures memory, not problem-solving.
智能体可能已经看过答案了。 这些公开基准测试中的问题都很陈旧。模型很可能在训练过程中接触过用于评分的实际修复方案。因此，分数在一定程度上衡量的是记忆力，而非解决问题的能力。
The setup is nothing like real work. A benchmark gives the agent a clean repo, one clear issue, and one command to run the tests. My engineers give it a half-open editor, a messy branch, a Slack thread, and a reviewer comment. Completely different job.
环境与实际工作完全不同。 基准测试给智能体提供的是一个干净的代码库、一个明确的问题和一个运行测试的命令。而我的工程师给它的是一个半打开的编辑器、一个混乱的分支、一段 Slack 讨论记录和一条审查意见。这是完全不同的工作。
Our codebase has its own habits. Our internal libraries, our wrappers, our test style, the imports we ban. No benchmark knows any of that, so an agent can write textbook-perfect code that our reviewers still reject on sight.
我们的代码库有自己的习惯。 我们的内部库、封装方式、测试风格以及禁止使用的导入项。没有任何基准测试了解这些，因此智能体写出的代码可能在教科书上是完美的，但我们的审查人员一眼就会拒绝。
The bar for passing is way lower. A benchmark passes a patch if the broken test now passes. My team passes a patch if it does that, and does not break unrelated tests, does not reformat the whole file, does not add stray imports, and is small enough to actually review. Benchmarks grade the first thing and none of the rest.
通过标准低得多。 基准测试只要修复了损坏的测试就算通过。而我的团队要求补丁不仅要修复问题，还不能破坏无关测试、不能重排整个文件、不能添加多余的导入，且必须足够小以便于审查。基准测试只评估第一点，而忽略了其余所有要求。

So I stopped treating the leaderboard as a decision and started treating it as a sniff test. Fine for narrowing the field. Useless for the actual call. 因此，我不再把排行榜作为决策依据，而是将其视为一种初步筛选工具。它适合缩小选择范围，但对于最终决定毫无用处。

The test set was in our git history all along

测试集一直就在我们的 Git 历史记录里

Here is the bit I wish I had thought of on day one. The best possible test data for a coding agent is the work we had already merged. Our last fifty merged pull requests came with everything a benchmark cannot fake: the real intent, the reviewer’s comments, a passing test suite, and a known-good final diff. 这是我希望在第一天就想到的点。对于编程智能体来说，最好的测试数据就是我们已经合并的代码。我们最近合并的 50 个 Pull Request 包含了基准测试无法模拟的一切：真实的意图、审查者的评论、通过的测试套件以及已知的正确最终差异。

So I replayed them: Take the last fifty merges, skip version bumps and generated files, keep the mix close to what we really ship. For each one, note the commit it started from and the issue text. Roll the repo back to that starting point, hand the agent the issue, let it work in a sandbox. Compare what it produced against what we actually merged. 所以我重演了它们：选取最近的 50 次合并，跳过版本号更新和自动生成的文件，保持组合与我们实际发布的内容接近。对于每一个 PR，记录其起始提交和问题描述。将代码库回滚到起始点，把问题交给智能体，让它在沙盒中工作，然后将它的产出与我们实际合并的代码进行对比。

I got this wrong on my first attempt, so learn from it: I scored on how closely the text matched. That is both unfair and useless. A human reviewer would happily merge a differently-worded solution that behaves the same way. The question is not “does this look like our diff.” It is “would this have passed the same review our diff passed.” 我第一次尝试时搞错了，所以请吸取教训：我当时是根据文本匹配度来评分的。这既不公平也毫无意义。人类审查者完全乐意合并一个措辞不同但行为一致的解决方案。问题的关键不在于“它看起来像不像我们的 diff”，而在于“它能否通过我们那个 diff 所通过的审查”。

The five things I ended up scoring

我最终确定的五个评分维度

After running enough replays I stopped guessing and settled on five things. Each one catches something a benchmark never sees. 在进行了足够的重演测试后，我不再盲目猜测，而是确定了五个维度。每一个维度都能捕捉到基准测试从未发现的问题。

Would the diff pass our review? Behavior, not text. I mark it down for reformatting files nobody asked it to touch, for breaking a test the merged version passed, and for taking twice the diff to reach the same outcome.
补丁能否通过审查？ 看行为而非文本。如果它重排了没人要求改动的文件、破坏了原本能通过的测试，或者用了两倍的代码量才达到同样的效果，我都会扣分。
Did it use its tools in a sane order? A coding agent is a loop, not a text generator. It reads, edits, runs tests, reacts, commits. I check four things: did it read a file before editing it, did the edit land where it said it would, did it actually run the tests afterward or just commit blind, and did it commit only what it changed.
它是否按合理的顺序使用工具？ 编程智能体是一个循环，而不是文本生成器。它读取、编辑、运行测试、反馈、提交。我检查四点：编辑前是否读取了文件、编辑位置是否正确、之后是否真的运行了测试还是盲目提交、以及是否只提交了改动的部分。
Did it update everything the change touched? This is the one that burned us. The agent renames a helper in one file and leaves broken call sites scattered across others. So now I follow the imports outward from every changed file and check that the downstream files still line up.
它是否更新了所有受影响的地方？ 这是让我们吃过亏的地方。智能体在一个文件中重命名了辅助函数，却在其他文件中留下了损坏的调用点。所以现在我会顺着每个改动文件的导入路径向外检查，确保下游文件依然正常。
Did the plan survive the work? Modern agents write a plan before touching anything. Two questions: did the plan match what was actually asked, and did the final diff still match the plan. Agents drift halfway through and quietly contradict the original request.
计划是否在执行中保持一致？ 现代智能体在动手前会写计划。有两个问题：计划是否符合要求，最终的 diff 是否符合计划。智能体有时会在中途偏离，悄悄违背最初的请求。
Does it know when to give up on a bad idea? When tests fail after an edit, a disciplined agent reverts and tries another approach. An undisciplined one piles more code on top of the failure. You can see it plainly in the trace: how many edits does it make per failed test run, and does it ever actually revert.
它是否知道何时放弃错误的想法？ 当编辑后测试失败时，有纪律的智能体会回滚并尝试其他方法。没纪律的则会在错误之上堆砌更多代码。你可以从追踪记录中清楚地看到：每次测试失败后它进行了多少次编辑，以及它是否真的回滚过。

One more check I added after a scare

在一次惊吓后我增加的额外检查

Coding agents invent credentials and echo risky shell patterns more often than I expected. So now every generated diff gets scanned for made-up secrets, dangerous constructs like raw eval or shell interpolation, and whatever patterns our own repo has banned. It takes milliseconds and it has already caught things. Cheap insurance. 编程智能体编造凭证和输出危险 Shell 命令的频率比我预想的要高。因此，现在每一个生成的 diff 都会被扫描，检查是否存在虚构的密钥、像 eval 或 Shell 插值这样的危险结构，以及我们代码库中禁止的任何模式。这只需要几毫秒，而且已经抓住了不少问题。这是廉价的保险。