CWE-bench v2: More coverage, quality, and transparency

CWE-bench is designed to test the limits of frontier cyber capabilities. In our continued pursuit to push the horizon at Collinear AI, CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on. The setup is the same as before. An agent gets a checkout of a real open-source repository and a short instruction, with the goal of finding and fixing a security weakness without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost. All our programmatic checks have been curated with the help of LLM judge panels that shape them towards the direction of hardness and fairness. We also maintain a held-out set, so that we can keep tracking progress on tasks no model has seen. To evaluate your model on the held-out set, contact cyber@collinear.ai.

CWE-bench 旨在测试前沿网络安全能力的极限。为了在 Collinear AI 继续拓展边界，CWE-bench v2 公开了一套包含 100 个任务的测试集，任何人都可以运行并评估自己的模型。设置与之前相同：智能体（agent）会获得一个真实的开源代码库检出版本和简短指令，目标是在不破坏现有功能的前提下发现并修复安全漏洞。评分完全通过程序化实现，因此结果是确定性的，任何人都可以零额外成本进行复现。我们所有的程序化检查都在 LLM 评判小组的帮助下进行了精心策划，以确保其难度和公平性。我们还维护了一个保留测试集，以便持续追踪模型未曾见过的任务进展。如需在保留测试集上评估您的模型，请联系 cyber@collinear.ai。

Our goal is to be exhaustive. A defender has to anticipate the ways an attacker could get in, and get it right every single time. These 100 tasks cover 91 different CWEs, spanning a wide range of vulnerability classes. A model that does well here has to understand the multiple weakness surfaces that a potential attacker can exploit. Recognising a familiar pattern is not enough. As part of our meticulous effort to keep improving quality, for every task we checked that the following questions were true: Does the instruction read like a realistic instruction and not an evaluation setting? Can every graded behaviour be inferred from the instruction? Does the verifier check for correct behaviour and not for specific structure of one fix? Does the starter code fail and the reference fix pass? Does the original functionality still work with the new fix? Do the verifiers fail memorized solutions like the known upstream patch? Every task was also reviewed by hand. Open-sourcing the set is part of the same effort to improve quality. We welcome feedback on any task, and contributions to the public set.

我们的目标是做到详尽无遗。防御者必须预判攻击者可能入侵的所有方式，并且每次都必须做到万无一失。这 100 个任务涵盖了 91 种不同的 CWE，跨越了广泛的漏洞类别。在此类测试中表现良好的模型，必须理解潜在攻击者可以利用的多种弱点表面。仅仅识别熟悉的模式是不够的。作为我们不断提高质量的严谨工作的一部分，我们针对每个任务核实了以下问题：指令读起来是否像真实的指令而非评估设置？每个评分行为是否都能从指令中推断出来？验证器检查的是正确行为而非某种特定修复结构吗？初始代码是否会失败而参考修复方案是否通过？原始功能在应用新修复后是否依然有效？验证器是否会拒绝像已知上游补丁那样的记忆化解决方案？每个任务也都经过了人工审核。将测试集开源也是提高质量工作的一部分。我们欢迎对任何任务提供反馈，并欢迎对公共测试集做出贡献。

Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open-weight models and updates on its own: labs and the community add a result by opening a PR on the model’s repo with a results file (how). For the full comparison, including API-only models such as Gemini, Claude and GPT, see the CWE-bench leaderboard Space, where new models are added through a PR on the Space. Open-weight models, from the dataset page: All evaluated models, including API-only ones: Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). 95 tasks are solved by at least one model, an indication that the tasks are indeed solvable. However, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. The public set, the grader and the harness are open. Run your model, and tell us where it stands. On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.

结果在 Hub 上的两个地方进行追踪。CWE-bench 数据集页面上的排行榜涵盖了开放权重模型并自动更新：实验室和社区可以通过在模型仓库中提交包含结果文件的 PR 来添加结果。如需进行完整比较（包括 Gemini、Claude 和 GPT 等仅提供 API 的模型），请查看 CWE-bench 排行榜 Space，新模型通过在该 Space 提交 PR 来添加。来自数据集页面的开放权重模型：所有已评估模型（包括仅提供 API 的模型）：在评估的 11 个模型中，Gemini 4 Argon 以 67.8% 的 pass@1 和 81.0% 的 pass@4 领先，略高于 Claude Opus 5.5（67.3% 的 pass@1，78.0% 的 pass@4）。至少有一个模型解决了 95 个任务，这表明这些任务确实是可以解决的。然而，即使是领先者在每次尝试中也会有 19% 的任务失败，因此长尾问题远未解决。公共测试集、评分器和测试工具均已开放。运行您的模型，看看它的表现如何。在成本方面，DeepSeek V4.1 Flash (1017)、Claude Opus 5.5 和 Gemini 4 Argon 构成了帕累托前沿：DeepSeek 以每次部署 0.04 美元的成本达到 57.0% 的 pass@1，Opus 以 1.83 美元提升了 10 个百分点，而 Gemini 以 3.92 美元略微领先。

Every task is a self-contained Harbor task (a Dockerized environment, an instruction, and a deterministic verifier), and the dataset ships a registry.json. Harbor can therefore run the whole benchmark straight from the Hugging Face repo, without cloning anything yourself: harbor run —repo “https://huggingface.co/datasets/collinear-ai/cwe-bench” -d cwe-bench@2.0 -k 4 —agent —model —env docker —allow-agent-host . -k 4 runs 4 attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated: accept its access conditions on the dataset page first. Harbor fetches the tasks with git, so also save your HF token to git’s credential helper once: hf auth login —token $HF_TOKEN —add-to-git-credential. The tasks are also tagged rl-environment: the same environments and verifiers can provide rewards for training defensive agents, not just for evaluating them.

每个任务都是一个独立的 Harbor 任务（包含 Docker 环境、指令和确定性验证器），数据集附带了 registry.json。因此，Harbor 可以直接从 Hugging Face 仓库运行整个基准测试，无需自行克隆任何内容：harbor run —repo “https://huggingface.co/datasets/collinear-ai/cwe-bench” -d cwe-bench@2.0 -k 4 —agent —model —env docker —allow-agent-host 。-k 4 表示每个任务运行 4 次尝试，报告的分数是所有 400 次试验的平均奖励。该数据集是受限的：请先在数据集页面接受其访问条件。Harbor 通过 git 获取任务，因此请务必将您的 HF 令牌保存到 git 的凭据助手中一次：hf auth login —token $HF_TOKEN —add-to-git-credential。这些任务还被标记为 rl-environment：相同的环境和验证器不仅可以用于评估防御性智能体，还可以为训练它们提供奖励。