How well do agents use test/verification techniques?

智能体在使用测试/验证技术方面的表现如何？

We previously noted that, while it’s easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse, indicating that whatever defaults developers are using may not work very well. Here, we test if simple instructions to agents to use particular techniques or libraries improve implementation correctness. 我们之前曾指出，虽然通过让编程智能体使用有效的测试技术来达到特定的质量标准比以往任何时候都容易，但软件质量似乎正在变差，这表明开发者目前使用的默认设置可能并不奏效。在此，我们测试了向智能体发出使用特定技术或库的简单指令，是否能提高实现的正确性。

Here, we re-use the Zstd implementation eval discussed in this comparison of agentic programming language effectiveness and, instead, compare different testing techniques and testing libraries when agents are given a prompt to implement Zstd with different addendums, such as “Use test-driven development”, “Use Lean 4”, “Use QuickCheck”, “Use property-based testing”, etc. I also ran some other evals, such as on the IMAP RFC, which are briefly discussed. All implementations were in Rust. 在此，我们重新使用了在“智能编程语言有效性比较”中讨论过的 Zstd 实现评估，并对比了当智能体在收到实现 Zstd 的提示词时，附加不同指令（如“使用测试驱动开发”、“使用 Lean 4”、“使用 QuickCheck”、“使用基于属性的测试”等）所带来的不同测试技术和库的效果。我还进行了一些其他评估（例如针对 IMAP RFC 的评估），并在文中进行了简要讨论。所有实现均使用 Rust 语言。

The 26 prompt conditions tested were ACL2, Adaptive (agents asked to use the best technique), Alloy, “Audit and fuzz risky areas”, “Audit first”, Creusot, Default (no additional instructions), Differential testing, Fuzzing, Hegel, Insta, Kani, Lean 4, “Make no mistakes”, Metamorphic testing, Mutation testing, Property-based testing, Proptest, QuickCheck, rstest, Rust built-in test framework, SMT solvers (with Z3, cvc5, and Yices, all available), Spin, TDD, TLA+, and Verus. 测试的 26 种提示词条件包括：ACL2、自适应（要求智能体使用最佳技术）、Alloy、“审计并模糊测试风险区域”、“先审计”、Creusot、默认（无额外指令）、差异测试、模糊测试、Hegel、Insta、Kani、Lean 4、“不出错”、变异测试、基于属性的测试、Proptest、QuickCheck、rstest、Rust 内置测试框架、SMT 求解器（提供 Z3、cvc5 和 Yices）、Spin、TDD、TLA+ 以及 Verus。

Additional, 4 skills were tested: Hegel with the official Hegel skill, the ECC Rust test skill (ECC is a collection of skills with 250k GitHub stars and 38k forks), the Trail of Bits property test skill, and a test skill I wrote (I’m a luddite who uses prompts instead of skills and have no feel for how to write a good skill). Other than my skill, the skills were chosen because those were the top skills codex turned up when asked to find relevant skills. 此外，还测试了 4 种技能：带有官方 Hegel 技能的 Hegel、ECC Rust 测试技能（ECC 是一个拥有 25 万 GitHub 星标和 3.8 万次分支的技能集合）、Trail of Bits 属性测试技能，以及我自己编写的一个测试技能（我是个只用提示词不用技能的“卢德分子”，对如何编写好的技能毫无感觉）。除了我自己的技能外，其他技能的选择是因为它们在 Codex 搜索相关技能时排名最高。

Predictions

预测

I pre-registered some guesses on how conditions will do: 我预先登记了一些关于各条件表现的猜测：

TDD will underperform (55% confidence): I actually added TDD specifically because I thought it would underperform. My confidence is low here because I don’t know what agents will do when instructed to do TDD; perhaps agents won’t do TDD and will do something that doesn’t underperform (or perhaps I’m wrong about TDD underperformance). TDD 将表现不佳（55% 置信度）： 我专门加入 TDD 正是因为我认为它会表现不佳。我对此信心不足，因为我不知道当被要求使用 TDD 时智能体会做什么；也许智能体根本不会执行 TDD，而是做了一些不会导致表现不佳的事情（或者也许我对 TDD 表现不佳的判断是错的）。
Formal methods will not overperform (52% confidence): My thought here is that formal methods are effective and useful (more so now than ever), good test methods are also effective and useful and, on simple problems, formal methods shouldn’t outperform if used at a similar level of competence. As with the above, but even more so, my confidence is low here because I don’t know what agents will do when instructed to do anything, and formal methods have been more hyped than effective test techniques for agentic coding, so it’s entirely plausible that labs have trained agents with RL environments with synthetic data which trains them to be very effective with formal methods without having trained agents to be effective with good test techniques. 形式化方法不会表现超群（52% 置信度）： 我的想法是，形式化方法既有效又有用（现在比以往任何时候都更有用），好的测试方法同样有效且有用。在简单问题上，如果使用水平相当，形式化方法不应表现得更好。与上述情况一样，我对此信心更低，因为我不知道智能体在收到指令时会做什么。在智能编程领域，形式化方法受到的炒作多于有效的测试技术，因此完全有可能实验室通过带有合成数据的强化学习环境训练了智能体，使其在形式化方法上非常有效，却未训练它们掌握好的测试技术。
“Make no mistakes” will not outperform no instructions (95% confidence): It’s a joke, and one that a lot of people have tried. If it worked, surely people would’ve noticed? “不出错”不会优于无指令（95% 置信度）： 这只是个笑话，很多人都试过。如果它真的有效，人们肯定早就注意到了吧？
The ECC test skill will not outperform (65% confidence): It’s somewhat big and doesn’t have any information I’d expect to be useful. It instructs agents to use TDD; to the extent that it gets agents to use TDD, I’d expect this to make things worse. ECC 测试技能不会表现超群（65% 置信度）： 它有点臃肿，且不包含任何我认为有用的信息。它指示智能体使用 TDD；如果它真的让智能体使用了 TDD，我预计这反而会让情况变糟。
Hegel’s skill will not outperform (65% confidence): It’s very big (the SKILL.md plus the linked Rust reference are over 20k tokens) and reads more like a tutorial than agent instructions. Hegel 的技能不会表现超群（65% 置信度）： 它太大了（SKILL.md 加上链接的 Rust 参考文档超过 2 万个 token），读起来更像教程而不是智能体指令。
The Trail of Bits test skill will not outperform (55% confidence): It has what looks like it might be useful information, but it’s also fairly big. Trail of Bits 测试技能不会表现超群（55% 置信度）： 它包含了一些看起来可能有用的信息，但它也相当臃肿。

Overall results

总体结果

Below, we have a very messy graph which shows the results for the conditions tested. One thing we can see is that nothing really wildly outperforms. However, Default (no additional instructions) does well above average. Looking at xhigh, on average, the fuzzing and PBT-related conditions did a little better than formal methods on average, with the situation being a lot more mixed at medium. The testing-related skills codex recommended we try underperformed, although our quick custom skill did ok. TDD didn’t do well, as predicted. 下方是一张非常杂乱的图表，展示了所测试条件的实验结果。我们可以看到，没有任何一种方法表现出显著的超群优势。然而，默认设置（无额外指令）的表现远高于平均水平。观察 xhigh（高努力）数据，平均而言，模糊测试和基于属性的测试（PBT）相关条件比形式化方法稍好，而在 medium（中等努力）水平下情况则更为复杂。Codex 推荐我们尝试的测试相关技能表现不佳，尽管我们快速编写的自定义技能表现尚可。正如预测的那样，TDD 的表现并不理想。