CWE-bench v2 Expands Coverage, Quality, and Transparency

CWE-bench v2 Expands Coverage, Quality, and Transparency

Published November 3, 2026

CWE-bench is designed to evaluate advanced cyber capabilities. CWE-bench v2 provides a public set of 100 tasks that can be used to evaluate security-focused models. The benchmark uses the same basic setup as its previous version: an agent receives a checkout of a real open-source repository and a short instruction, then must identify and fix a security weakness without breaking existing functionality.

Scoring is fully programmatic, making results deterministic and reproducible without additional evaluation costs. The programmatic checks were curated with the assistance of LLM judge panels to improve both difficulty and fairness. A separate held-out set is maintained to track progress on tasks that models have not previously encountered. Evaluation on that set is available by contacting cyber@collinear.ai.

Broad Vulnerability Coverage

The 100 tasks cover 91 different Common Weakness Enumerations (CWEs) and span a wide range of vulnerability classes. This breadth is intended to test whether a model can understand multiple weakness surfaces that attackers might exploit, rather than merely recognize familiar patterns.

Each task was reviewed against the following criteria:

• The instruction should resemble a realistic development or security request, rather than an evaluation prompt.
• Every graded behavior should be inferable from the instruction.
• The verifier should test correct behavior, not the specific structure of one possible fix.
• The starter code should fail, while the reference fix should pass.
• The repository's original functionality should continue to work after the fix.
• The verifier should reject memorized solutions, including known upstream patches.

Every task was also reviewed manually. The public release is intended to support further quality improvements through feedback and contributions.

Results

Results are tracked in two locations. The leaderboard on the CWE-bench dataset page covers open-weight models and updates through pull requests to model repositories containing results files. The CWE-bench leaderboard Space includes the full comparison, including API-only models such as Gemini, Claude, and GPT. New results are added through pull requests to the Space.

Among the 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows with 67.3% pass@1 and 78.0% pass@4. At least one model solves 95 of the tasks, indicating that the tasks are solvable. However, the leading model fails 19% of the set on every attempt, showing that many challenges remain unresolved.

In the cost comparison, DeepSeek V4.1 Flash (1017), Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier. DeepSeek reaches 57.0% pass@1 at $0.04 per rollout. Claude Opus 5.5 reaches a result ten percentage points higher at $1.83 per rollout, while Gemini 4 Argon achieves a slightly higher score at $3.92 per rollout.

Running CWE-bench from the Hub

Each task is a self-contained Harbor task consisting of a Dockerized environment, an instruction, and a deterministic verifier. The dataset includes a registry.json file, allowing Harbor to run the benchmark directly from the Hugging Face repository:

harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 \
  --agent <agent> --model <model> --env docker \
  --allow-agent-host <model API host>

The -k 4 option runs four attempts per task. The reported score is the mean reward across all 400 trials. The dataset is gated, so its access conditions must be accepted on the dataset page before use.

Harbor retrieves the tasks with Git. An Hugging Face token can be saved to Git's credential helper with:

hf auth login --token $HF_TOKEN --add-to-git-credential

The tasks are also tagged rl-environment. This allows the same environments and verifiers to provide rewards for training defensive agents as well as evaluating them.