本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

I Built a Benchmark That Grades AI Code Reviewers on Planted Defects

Kaggle Benchmarking Challenge Submission This is a submission for the Kaggle Benchmarking Challenge.* Why I built it I spent the last month running fourteen independent audit rounds on a clean-room codebase. Outside reviewers — human and AI — kept catching defects that survived my own builder, my own test suite, and my own verifier. The experience left me with a question I couldn’t shake: can an AI model do what those auditors did? Not fix the bug. Find it. Name the lines. Say what’s wrong and why.

Kaggle 基准测试挑战赛提交作品。这是我为 Kaggle 基准测试挑战赛提交的作品。* 我构建它的原因：过去一个月里，我在一个洁净室代码库上进行了十四轮独立的审计。外部审查员（包括人类和人工智能）不断发现那些连我自己的构建器、测试套件和验证器都未能察觉的缺陷。这段经历让我产生了一个挥之不去的疑问：人工智能模型能做到那些审计员所做的事吗？不是修复漏洞，而是发现它、指出具体行数、说明错误所在及其原因。

Existing benchmarks don’t isolate this. SWE-bench measures repair — a model can stumble into a passing patch without ever identifying the defect. Devign does binary “vulnerable / not vulnerable” classification — no localization, no explanation, no severity. A model that says “vulnerable” for the wrong reason scores the same as one that found the actual flaw. I haven’t found a benchmark that combines all three of: planted defects with controlled difficulty tiers, mandatory line localization and explanation, and false-positive measurement on paired clean variants. So I built Red-Pen Bench.

现有的基准测试无法孤立地评估这一点。SWE-bench 衡量的是修复能力——模型可能在从未识别出缺陷的情况下偶然生成一个通过测试的补丁。Devign 进行的是“易受攻击/不易受攻击”的二元分类——没有定位、没有解释、也没有严重程度评估。一个因错误原因判定为“易受攻击”的模型，其得分与真正发现缺陷的模型相同。我没有找到一个能同时结合以下三点的基准测试：带有受控难度等级的植入缺陷、强制性的行定位与解释，以及针对配对洁净变体的误报测量。因此，我构建了 Red-Pen Bench。

What it tests 44 planted defects drawn from the mutation-testing operator taxonomy. The approach follows the selective-mutation tradition: Offutt, Lee, Rothermel, Untch, and Zapf (“An Experimental Determination of Sufficient Mutant Operators,” ACM TOSEM 5(2):99–118, 1996) showed that five operators (ABS, UOI, LCR, AOR, ROR) capture 99.5% of the mutation score of the full operator set. That 99.5% figure belongs to the complete five-operator set — this corpus uses four of the five (AOR, ROR, UOI, LCR; no ABS) plus statement-deletion defects and higher-order mutants (two interacting faults) for the hard tier, which that study doesn’t cover. The operator family is the method; the exact coverage number is theirs, not mine.

它测试了 44 个取自变异测试算子分类法的植入缺陷。该方法遵循选择性变异的传统：Offutt、Lee、Rothermel、Untch 和 Zapf（《充分变异算子的实验测定》，ACM TOSEM 5(2):99–118, 1996）的研究表明，五个算子（ABS、UOI、LCR、AOR、ROR）可以捕获完整算子集 99.5% 的变异分数。99.5% 这个数字属于完整的五个算子集——本语料库使用了其中四个（AOR、ROR、UOI、LCR；不含 ABS），并在困难等级中增加了语句删除缺陷和高阶变异体（两个相互作用的故障），这些是该研究未涵盖的内容。算子家族是方法论；确切的覆盖率数字是他们的，而非我的。

Every defect ships in four variants: Original — the defect in its natural setting; Disguised — the same mechanism transplanted to a different domain, different names (reduces the chance that a model can rely on surface familiarity or template recognition); Fixed — the repaired code (false-positive check); Clean — correct code written from scratch, independent of the defective version (control).

每个缺陷都包含四种变体：原始版——缺陷在其自然环境中的状态；伪装版——将相同的机制移植到不同的领域并更改名称（减少模型依赖表面熟悉度或模板识别的可能性）；修复版——修复后的代码（用于误报检查）；洁净版——从零开始编写的正确代码，独立于缺陷版本（作为对照）。

Three difficulty tiers: Easy (18 items): single-operator defects — off-by-one loops, wrong arithmetic operators, inverted guards; Medium (18 items): wrong logical connectors, missing edge-case checks, ceiling-vs-floor errors, the mutable-default-argument trap; Hard (8 items): higher-order mutants — an auth check that’s both inverted and miswired, a withdrawal that’s both overdraft-permissive and audit-evasive.

三个难度等级：简单（18 项）：单算子缺陷——差一错误循环、错误的算术运算符、反转的守卫条件；中等（18 项）：错误的逻辑连接词、缺失的边界情况检查、上限与下限错误、可变默认参数陷阱；困难（8 项）：高阶变异体——既被反转又接线错误的身份验证检查、既允许透支又规避审计的提款操作。

Every item passed a killability gate before entering the corpus: an automated test that fails on the defect variants and passes on the fixed/clean ones. And the corpus survived its own red pen — my first build had 44 defects that an independent verifier proved were really 10 templates with renamed variables. I threw it out, regenerated 44 genuinely distinct defects, and the verifier signed off on the second build: 44/44 structurally distinct, 44/44 killability, 176-snippet integrity. The benchmark grades reviewers; it had to survive review itself.

每一项在进入语料库之前都通过了“可杀灭性”门槛：即一项在缺陷变体上失败、在修复/洁净变体上通过的自动化测试。该语料库也经受住了自身的严格审查——我最初构建的 44 个缺陷被独立验证器证明实际上只是 10 个重命名变量的模板。我将其废弃，重新生成了 44 个真正不同的缺陷，验证器对第二次构建表示认可：44/44 结构各异，44/44 可杀灭，176 个代码片段完整性达标。该基准测试用于评估审查员，它自身也必须经得起审查。

Scoring (no LLM judge) Models must output three things: DEFECT: yes/no, LINES: , EXPLANATION: . Scored automatically, with no model grading a model: Detection rate — fraction of the 2n defective variants (original + disguised) where the verdict matched ground truth; Line-level F1 — mean over all 4n variants of line-level F1 against the planted fault lines (on fixed/clean variants: 1 if the model named no lines, 0 otherwise); Keyword recall — mean over the 2n defective variants of planted-keyword hits in the explanation; False-positive rate — fraction of the 2n non-defective control variants (fixed + independently clean) flagged DEFECT: yes.

评分（无 LLM 裁判）：模型必须输出三项内容：DEFECT: yes/no, LINES: , EXPLANATION: 。采用自动评分，不使用模型来评估模型：检测率——在 2n 个缺陷变体（原始 + 伪装）中，判定结果与事实相符的比例；行级 F1——在所有 4n 个变体中，针对植入故障行的行级 F1 平均值（在修复/洁净变体上：如果模型未指出任何行，则为 1，否则为 0）；关键词召回率——在 2n 个缺陷变体中，解释中命中植入关键词的平均值；误报率——在 2n 个非缺陷对照变体（修复 + 独立洁净）中，被标记为 DEFECT: yes 的比例。

Tier score = (detection + line-F1 + keyword-recall) / 3 − false-positive rate. The denominators scale exactly to the variant counts. Easy and medium tiers (n=18) average the four terms over 36, 72, 36, and 36 measurements respectively; the hard tier (n=8) averages over 16, 32, 16, and 16. The “Average” column is the unweighted mean of the three tier scores, recomputed from the displayed tier values — each tier weighted equally despite the different item counts.

等级得分 = (检测率 + 行级 F1 + 关键词召回率) / 3 − 误报率。分母与变体数量完全对应。简单和中等等级（n=18）分别对 36、72、36 和 36 次测量取四个项的平均值；困难等级（n=8）则对 16、32、16 和 16 次测量取平均值。“平均”列是三个等级得分的未加权平均值，根据显示的等级数值重新计算——尽管项目数量不同，但每个等级的权重相同。

Note what the baselines expose about the metric itself: the composite’s zero point is arbitrary. A model that attempts detection and errs can score below one that abstains — the meaningful threshold is the always-no baseline (16.7%), not zero. Anything below 16.7% is losing to doing nothing at all. The models Four models, four labs, all run live on Kaggle Benchmarks on October 10, 2026, agai

注意基准线揭示了指标本身的特性：复合指标的零点是任意的。一个尝试检测但出错的模型，其得分可能低于一个选择放弃的模型——有意义的阈值是“总是回答否”的基准线（16.7%），而不是零。任何低于 16.7% 的得分都意味着表现不如什么都不做。这些模型：四个模型，四个实验室，均于 2026 年 10 月 10 日在 Kaggle Benchmarks 上实时运行……