Benchmarking Opus 5 on SlopCodeBench

在 SlopCodeBench 上对 Opus 5 进行基准测试

Benchmarking Opus 5 on SlopCodeBench we got better benchmarks I’ve written before something along the lines of: THERE ARE NO GOOD BENCHMARKS for a model’s ability to maintain codebase quality That wasn’t entirely true. I love nothing more than burying a good lede. Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski’s lab at UW Madison. 在 SlopCodeBench 上对 Opus 5 进行基准测试后，我们得到了更好的基准数据。我之前曾写过类似这样的话：“目前没有能够衡量模型维护代码库质量能力的优秀基准测试。”但这并不完全正确。我最喜欢在文章开头埋下伏笔。上周五，我深入研究了 SlopCodeBench，这是一个由威斯康星大学麦迪逊分校 @GOrlanski 实验室推出的较新（2026 年 3 月）的长周期编码基准测试。

It addresses the thing that bothers me most about coding benchmarks - that even “larger” more complex benchmarks still divulge the whole problem up front: In contrast, each challenge in SlopCodeBench has multiple “checkpoints” - the model doesn’t know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged. It’s a good paper. It’s not that long. You should read it. What’s cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively. 它解决了编码基准测试中最让我困扰的问题——即便是“更大”、更复杂的基准测试，也会在一开始就泄露整个问题。相比之下，SlopCodeBench 中的每个挑战都有多个“检查点”——模型无法预知整个问题，它必须随着新需求的披露，逐步演进代码库。这是一篇很好的论文，篇幅不长，值得一读。这个基准测试最酷的地方在于它尚未饱和——在测试时，当时最强的模型 GPT-5.4 和 Opus 4.6 的严格通过率分别仅为 11% 和 17%。

On Friday I ran three claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench and watched it live for six hours. Opus 5 wins technically but none of them did a very good job IMO. Will post more results soon with Fable and 5.6 Sol in the mix. The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6’s 17% strict pass rate in the original paper. 周五，我让三个 Claude 模型（Opus 4.8、Sonnet 5 和 Opus 5）运行了 SlopCodeBench 的一个子集，并现场观察了六个小时。从技术上讲，Opus 5 赢了，但在我看来，它们表现得都不怎么样。我很快会发布包含 Fable 和 5.6 Sol 的更多结果。最值得关注的是，Opus 5 在我测试的这个小型基准子集上获得了 24% 的通过率——这比原论文中 Opus 4.6 的 17% 严格通过率并没有高出多少。

All of the tested models showed a pretty significant increase in verbosity, complexity, and a bunch of other code smell metrics over the course of each challenge, with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges. My personal read of this 23% pass rate is that SlopCodeBench finally gives some signal for a thing I’ve only ever been able to argue from vibes - that for real-shaped software engineering work, building one issue at a time, today’s models can’t be relied on to run lights-off without steering. 所有被测模型在每个挑战过程中，在冗余度、复杂性以及其他一系列“代码异味”（code smell）指标上都表现出显著增加。在同一组挑战中，Opus 5 编写的函数/可调用对象数量是 Opus 4.8 的五倍。我个人对这个 23% 通过率的解读是：SlopCodeBench 终于为我一直以来只能凭“感觉”论证的观点提供了证据——即在处理真实的软件工程任务时，由于需要逐个解决问题，目前的模型还无法在无人值守的情况下独立完成工作。

I had claude pick out 3 problems from the repo, 17 checkpoints total, a mix of easy/medium/hard labeled problems: circuit_eval — easy (8 checkpoints), database_migration — medium (5 checkpoints), dynamic_config_service_api — hard (4 checkpoints). There’s an appendix at the end with all 17 checkpoints explained in detail but I won’t put that all here. And then I ran them across all three models, in parallel, with a fresh context window per checkpoint. All models got the same prompts and ran in the claude code harness. 我让 Claude 从仓库中挑选了 3 个问题，总计 17 个检查点，涵盖了简单/中等/困难三种难度：circuit_eval（简单，8 个检查点）、database_migration（中等，5 个检查点）、dynamic_config_service_api（困难，4 个检查点）。文末附录详细解释了所有 17 个检查点，这里就不一一列举了。随后，我让三个模型并行运行这些任务，每个检查点都使用全新的上下文窗口。所有模型接收相同的提示词，并在 Claude 代码测试框架中运行。

The metric I decided I care about is the strict pass: everything new is green including every regression test that was inherited from previous checkpoints. A model fails a checkpoint if the solution has a defect - defects are detected by taking the models output, a CLI to run or in some cases e.g. an api server to poke at, and running a set of held-out black-box tests against the produced entrypoint. 我决定关注的指标是“严格通过”：所有新功能必须通过测试，且继承自先前检查点的所有回归测试也必须通过。如果解决方案存在缺陷，模型即判定为检查点失败——缺陷检测方法是获取模型的输出，通过 CLI 运行，或在某些情况下（如 API 服务器）进行探测，并针对生成的入口点运行一组预留的黑盒测试。

Again, the strict pass criteria means that if a model bungles something in checkpoint 4, it can’t pass the following checkpoints because that failing part of the code carries forward. For all 9 test runs, none of the models made it to the end of any challenge with everything passing, even on the problem marked as “easy” difficulty. 再次强调，严格通过标准意味着如果模型在第 4 个检查点搞砸了，它就无法通过后续检查点，因为那部分失败的代码会延续下去。在全部 9 次测试运行中，没有任何模型能完整通过任何挑战，即使是在标记为“简单”难度的问题上也是如此。

While it ran, Sonnet’s first checkpoint was more expensive, but by the end of problem 1, Sonnet became the cheapest of the three. (It seems like once the basics were built and the work turned into maintenance, then the cost savings started to take over). For the first challenge, last generation’s models accumulated defects steadily; Opus 5 had a defect each on checkpoints 4 and 5. For the first two hours, Opus 5 was the only model with any strict passes at all — three in a row to start. 在运行过程中，Sonnet 的第一个检查点成本较高，但到第 1 个问题结束时，Sonnet 成了三个模型中最便宜的一个。（看起来一旦基础架构搭建完成，工作转为维护性质，成本优势就开始显现了）。在第一个挑战中，上一代模型稳步积累缺陷；Opus 5 在第 4 和第 5 个检查点各出现了一个缺陷。在最初的两个小时里，Opus 5 是唯一获得严格通过的模型——开局连续通过了三个。

Compared against the other models, Opus 5 was technically better on problem 1 (circuit_eval). But after acing the first three checkpoints, every subsequent solution had at least one defect (failing test case). 与其他模型相比，Opus 5 在第 1 个问题（circuit_eval）上技术表现更好。但在顺利通过前三个检查点后，后续的每一个解决方案都至少存在一个缺陷（测试用例失败）。

If our definition of success is “reached the final checkpoint with no defects” then Opus 5 failed all three problems, but it failed slightly-less-badly than the other models. For the costs vs defects report, I really hate claude-isms but this one I decided to leave in: every dollar bought correctness. Nobody bought enough of it. (Obviously this small subset of the bench cannot tell us definitively that spending more $$ will lead to higher pass rates). 如果我们将成功定义为“无缺陷到达最终检查点”，那么 Opus 5 在所有三个问题上都失败了，但它比其他模型失败得稍微“体面”一点。关于成本与缺陷的报告，我真的很讨厌那些 Claude 式的套话，但我决定保留这一句：每一美元都换来了正确性，只是没人买得足够多。（显然，这个基准测试的小子集无法明确告诉我们，投入更多资金是否一定会带来更高的通过率）。

As far as strict passes go, Opus 5 got four of them (24% pass rate) (the first three ck of circuit_eval, plus database_migration ck1). Opus 4.8 and Sonnet 5 both got one strict pass (6% pass rate), the same database_migration ck1 that Opus 5 got. So the winner cleared 4/17, and 3 of those were the opening checkpoints of one problem. It would appear we have an unsaturated benchmark for the next frontier of models. Nice work @GOrlanski and team. 就严格通过率而言，Opus 5 获得了四次（24% 的通过率）（circuit_eval 的前三个检查点，加上 database_migration 的第 1 个检查点）。Opus 4.8 和 Sonnet 5 都只获得了一次严格通过（6% 的通过率），即 Opus 5 也通过的 database_migration 第 1 个检查点。所以获胜者完成了 17 个检查点中的 4 个，其中 3 个还是同一个问题的初始检查点。看来我们拥有了一个针对下一代模型的前沿、未饱和的基准测试。干得好，@GOrlanski 和团队。

I’m not totally sold on “linting the slop away” just yet, because I don’t think it’s yet possible to deterministically parse the “maintainability” of a particular codebase checkpoint. But code quality metrics are interesting to keep an eye on, and they’re probably directionally correct. With SlopCodeBench, you get the results after each checkpoint across various quality metrics. There are 41 of them in the results file. Roughly grouped: size (source lines, files, functions, methods, classes, statements, and lines added and removed at that checkpoint) and complexity (cyclomatic complexity mean, max, and spread, how many functions land in the “high” and “extreme” band). 我目前还不太相信“通过 Lint 工具消除垃圾代码”这一套，因为我认为目前还无法确定性地解析特定代码库检查点的“可维护性”。但关注代码质量指标很有趣，而且它们在方向上可能是正确的。通过 SlopCodeBench，你可以在每个检查点后获得各种质量指标的结果。结果文件中共有 41 个指标，大致分为：规模（源代码行数、文件、函数、方法、类、语句，以及在该检查点添加和删除的行数）和复杂性（圈复杂度平均值、最大值和分布，以及有多少函数处于“高”和“极端”区间）。