SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are created at evaluation time, so no content exists in advance to leak, and its difficulty is set by the participating agents rather than fixed at release.

静态编码基准测试在发布时是固定的。随着时间的推移，它们的任务会泄露到训练数据中，更强大的智能体最终会使这些基准测试饱和。我们认为，基准测试应该旨在实现自我扩展：其任务和预言机是在评估时创建的，因此不存在预先泄露的内容，且其难度由参与的智能体设定，而不是在发布时固定。

We introduce SWE-Duel, an arena in which two coding agents evaluate each other over mirrored Red-Blue turns. In each turn, the Red agent works on a real repository across three context-isolated sessions. It proposes and implements a new feature with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests.

我们引入了 SWE-Duel，这是一个竞技场，两个编码智能体通过镜像的红蓝回合相互评估。在每一回合中，红色智能体在三个上下文隔离的会话中处理一个真实的存储库。它提出并实现了一个带有功能测试的新特性，然后在该特性中嵌入了一个微妙的错误，并附带了密封的错误触发测试。

The resulting pull request is admitted only if (1) it passes the feature tests and the repository’s existing tests while failing exactly the bug tests, and (2) a fresh session of the same agent can repair the bug blind, which rules out unsolvable tasks. The Blue agent then reviews the pull request without being told a bug exists. It wins only if its revision passes all three test suites, and then the two agents swap roles.

产生的拉取请求只有在满足以下条件时才会被接纳：(1) 它通过了功能测试和存储库现有的测试，同时恰好未能通过错误测试；(2) 同一智能体的新会话可以在盲测中修复该错误，这排除了不可解的任务。随后，蓝色智能体在不知情的情况下审查该拉取请求。只有当其修订版本通过所有三套测试时，它才会获胜，之后两个智能体交换角色。

We run a tournament among 11 model-harness participants on 15 repositories in five languages for 1,500 turns. The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviews. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate.

我们在 15 个存储库和五种语言上进行了 1,500 个回合的锦标赛，共有 11 个模型参与者。领先者以不同的方式获胜。配备 Codex CLI 的 GPT-5.5 作为最强的求解器排名第一，修复了其审查缺陷的 98.3%。配备 mini-swe-agent 的 Claude Opus 4.8 作为最强的任务作者排名第二，红色胜率为 27.3%。

Finally, within one model family, tasks written by the newest version defeat a fixed-reference solver far more often than those of older versions (42.9% vs. 4.0%), giving preliminary evidence that task difficulty scales with task authoring capability.

最后，在同一个模型系列中，由最新版本编写的任务击败固定参考求解器的频率远高于旧版本（42.9% 对 4.0%），这提供了初步证据，表明任务难度随着任务编写能力的提高而扩展。