The artifact worth reviewing when agents write the code

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

GitHub’s Security Lab reported 24 Android vulnerabilities found by an AI security agent, and the part worth copying is what ended up version-controlled. Not the exploit code. The set of YAML taskflows that guided the model through the audit, checked into a public repository, runnable by anyone with a Copilot license. The agent did the finding. The taskflows are the thing a team can read, diff, and argue about. That is a small shift with a large consequence for review. When a person writes a patch, the diff is a decent summary of the judgment that went into it, because the person held the reasoning and the code is the residue. When an agent writes the patch, the reasoning lives somewhere else entirely: a prompt, a rule file, a model version, a sequence of tool calls. Reviewing only the residue means reading the output of a process you never see. The diff stops being the whole artifact.

GitHub 安全实验室报告称，一个 AI 安全代理发现了 24 个安卓漏洞，而真正值得效仿的部分是那些最终被纳入版本控制的内容。这指的不是漏洞利用代码，而是引导模型进行审计的一系列 YAML 任务流（taskflows）。这些任务流被签入公共仓库，任何拥有 Copilot 许可证的人都可以运行。发现漏洞的是代理，但任务流才是团队可以阅读、进行差异对比（diff）并展开讨论的对象。这对代码审查而言是一个微小但影响深远的转变。当人类编写补丁时，差异对比是其判断过程的良好总结，因为推理过程存在于人的脑海中，而代码只是残留物。当代理编写补丁时，推理过程完全存在于别处：提示词、规则文件、模型版本或一系列工具调用中。仅审查残留物意味着你是在阅读一个从未见过的过程的产出，差异对比不再是完整的产物。

Ian Bicking’s post on what a serious AI product would look like puts the failure precisely. Checking for mistakes, he writes, is pushed off into code review, which encourages the author to hand the work to a reviewer without ever looking. He wants a two-column worksheet where every claim has a checkbox you only tick after you have verified it, and he notes that the vendors’ own warning labels (“AI can make mistakes, double-check responses”) are fine print designed to move responsibility onto the user. His complaint is about products. The same structure shows up in review. If the process that produced a change is invisible, then the reviewer becomes the only place where checking happens, and the volume of changes is exactly what makes that unworkable. The fix is not to ask reviewers to work harder. It is to make the process itself a reviewable object, so that some of the checking happens where it can be repeated instead of where it cannot.

Ian Bicking 关于严肃的 AI 产品应具备何种形态的文章精准地指出了这一缺陷。他写道，错误检查被推卸给了代码审查，这助长了作者在不进行任何检查的情况下就将工作交给审查者的风气。他主张使用一种双栏工作表，每一项主张都配有一个复选框，只有在验证后才能勾选。他指出，供应商自己的警告标签（“AI 可能会出错，请仔细核对回复”）只是旨在将责任转嫁给用户的细则。他的抱怨针对的是产品，但同样的结构也出现在审查中。如果产生变更的过程是不可见的，那么审查者就成了检查发生的唯一场所，而变更的数量恰恰使其变得不可行。解决办法不是要求审查者更加努力，而是将过程本身变成一个可审查的对象，从而使部分检查可以在可重复的地方进行，而不是在无法重复的地方进行。

What a reviewable workflow actually looks like: The GitHub Security Lab agent is a concrete example. The taskflows are YAML files with names like gather_mobile_entry_point_info.yaml and classify_application_local.yaml. One splits entry points into mobile and non-mobile so the agent understands the right attack surface. Another enumerates vulnerability classes and asks the model to consider them against each entry point, so an intent-based entry point gets checked against known intent-based bug classes rather than whatever the model feels like noticing. The team also ran both a strict prompt and a broad prompt across multiple runs, because the model is non-deterministic. Output lands in a SQLite table with a has_vulnerability column you filter with a checkmark. There is a real cost attached, and the post is upfront about it: a GitHub Copilot license, premium model requests, many tool calls, and one to two hours on a medium-sized repository. Every one of those attributes is reviewable before you run anything. You can read the vulnerability class list and ask whether it covers your stack. You can see the run is repeated. You can see where the findings land and what they claim. Compare that with a code review comment that says “this looks suspicious” and moves on.

可审查的工作流究竟是什么样子的？GitHub 安全实验室的代理就是一个具体的例子。这些任务流是 YAML 文件，名称如 gather_mobile_entry_point_info.yaml 和 classify_application_local.yaml。前者将入口点分为移动端和非移动端，以便代理理解正确的攻击面；后者列举了漏洞类别，并要求模型针对每个入口点进行考量，这样基于意图的入口点就会针对已知的相关漏洞类别进行检查，而不是由模型随意发挥。由于模型具有非确定性，团队还在多次运行中同时使用了严格提示词和宽泛提示词。输出结果存入 SQLite 表中，其中包含一个可以通过复选框过滤的 has_vulnerability 列。这确实存在成本，文章对此直言不讳：需要 GitHub Copilot 许可证、高级模型请求、大量的工具调用，以及在中型仓库上耗费一到两个小时。在运行任何东西之前，所有这些属性都是可审查的。你可以阅读漏洞类别列表，询问它是否覆盖了你的技术栈；你可以看到运行过程是可重复的；你可以看到发现结果存放在哪里以及它们的主张是什么。将此与那种只写一句“这看起来很可疑”就草草了事的代码审查评论相比，高下立判。

Cloudflare’s CI-native reviewer takes a different shape to the same end. They built it on OpenCode and put several reviewers behind a deduplication coordinator, so a finding that two reviewers agree on surfaces once. There is also a check on whether the repo’s AGENTS.md file is current. The coordinator is the reviewable part. Without it, the same issue appears four times and the reviewer learns to skim.

Cloudflare 的 CI 原生审查器采取了不同的形式来实现相同的目标。他们基于 OpenCode 构建了该系统，并在多个审查者之后设置了一个去重协调器，因此两个审查者达成一致的发现只会显示一次。此外，系统还会检查仓库的 AGENTS.md 文件是否为最新。协调器就是那个可审查的部分。如果没有它，同一个问题会出现四次，审查者最终只会学会走马观花。

Rules are part of the workflow: The rules a reviewer applies live in the same category. Stack Overflow’s post on building shared coding guidelines for AI is written mostly from the generation side, how to get agents to write to your standards, and it contains the line that matters for review: code review will be most engineers’ first look at the code, since they did not write it. Vish Abrams, chief architect at Heroku, makes the point that seasoned engineers assume things like DRY as common knowledge, and that assumption does not transfer to an agent. A rule that only exists in a config file the reviewer never reads is decoration. What matters is whether the reviewer ingests the files your team already keeps, AGENTS.md, CLAUDE.md, .cursorrules, Copilot instruction files, and scopes them so a rule for the payments service does not fire on the CLI tool. CodeRabbit reads repo config, Copilot reads instruction files, and Kodus ingests repo rule files as scoped rules rather than as loose context. Those are three shapes of the same idea, and the useful test is whether you can point at the file and the line that produced a comment.

规则是工作流的一部分：审查者应用的规则也属于同一范畴。Stack Overflow 关于构建 AI 共享编码指南的文章主要从生成端出发，探讨如何让代理按照你的标准编写代码，其中包含了一句对审查至关重要的话：代码审查将是大多数工程师第一次看到这些代码的机会，因为他们并没有亲自编写它。Heroku 的首席架构师 Vish Abrams 指出，经验丰富的工程师会将 DRY（不要重复自己）等原则视为常识，但这种假设并不能转移到代理身上。如果一条规则只存在于审查者从不阅读的配置文件中，那它只是装饰。真正重要的是审查者是否吸收了团队现有的文件（如 AGENTS.md、CLAUDE.md、.cursorrules、Copilot 指令文件），并对其进行范围限定，从而确保支付服务的规则不会在 CLI 工具上触发。CodeRabbit 读取仓库配置，Copilot 读取指令文件，而 Kodus 将仓库规则文件作为限定范围的规则而非松散的上下文来吸收。这是同一理念的三种表现形式，而有效的测试标准是：你是否能指出产生某条评论的具体文件和行号。

A test you can run before you trust the workflow: Take any agent-produced change your team merged last week and try to answer these without opening the diff. Where is the prompt or taskflow, and is it in the repository or in someone’s chat history? Which model version ran, and would a rerun use the same one? Which rules were in scope for this file, and where do those rules live? What did the workflow check that a test suite would not catch? If the workflow found nothing, is that because there was nothing, or because the run was cut short? If the answers are all “somewhere in a chat window,” the review process is carrying work it cannot carry at volume. If the answers are files, you have something a second engineer can review before anyone merges anything, which is the point.

在信任工作流之前可以进行的测试：拿走你团队上周合并的任何由代理生成的变更，试着在不打开差异对比的情况下回答以下问题。提示词或任务流在哪里？它是在仓库里还是在某人的聊天记录里？运行的是哪个模型版本，重新运行会使用同一个版本吗？该文件适用的规则有哪些，这些规则存放在哪里？工作流检查了哪些测试套件无法捕捉到的内容？如果工作流什么也没发现，是因为确实没有问题，还是因为运行被中断了？如果答案全是“在某个聊天窗口里”，那么审查过程承担了它在大规模环境下无法承受的工作。如果答案是文件，那么你就拥有了可以在任何人合并之前由第二位工程师进行审查的内容，这才是重点所在。

What this does to review time: None of this makes review faster in the sense of reading fewer lines. It moves a specific category of checking out of the reviewer’s head and into an artifact that can be read once and rerun every time. That is the difference between reviewing a change and reviewing a process, and only the second one scales with the number of changes an agent can produce. There is a cost, and it is worth naming. Versioning prompts and taskflows adds config files to maintain, and they drift, which is why Cloudflare che

这对审查时间的影响：这一切并不会通过减少阅读行数来加快审查速度。它将特定类别的检查从审查者的脑海中转移到了一个可以被阅读一次并每次重复运行的产物中。这就是审查变更与审查过程之间的区别，只有后者才能随着代理产生的变更数量进行扩展。这确实有成本，而且值得一提。对提示词和任务流进行版本控制增加了需要维护的配置文件，而且它们会发生偏移，这就是为什么 Cloudflare……