Collinear AI has released CWE-bench v2, a public benchmark of 100 tasks designed to test how well AI agents identify and fix security weaknesses in real open-source repositories. Each task gives an agent a repository checkout and a short instruction; the agent must address the vulnerability without breaking existing functionality.
The benchmark covers 91 Common Weakness Enumerations (CWEs), spanning a range of vulnerability types. Its scoring is programmatic and deterministic, allowing results to be reproduced. Collinear AI says the checks were refined with help from panels of language-model judges and that every task received a manual review. The company also maintains a held-out set for tracking performance on tasks models have not encountered; access to that set requires contacting cyber@collinear.ai.
Among 11 evaluated models, Gemini 4 Argon recorded the top results, with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 followed with 67.3% and 78.0%, respectively. At least one model solved 95 of the 100 tasks, but the leading model still failed 19% of the set on every attempt.
The benchmark’s cost comparison identifies DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon as the Pareto frontier. DeepSeek achieved 57.0% pass@1 at $0.04 per rollout; Opus scored ten percentage points higher at $1.83, while Gemini edged ahead at $3.92.
CWE-bench v2 is available through Hugging Face, with results tracked on the dataset page and a separate leaderboard Space. The tasks are packaged as self-contained Harbor environments, each with a Dockerized setup, an instruction and a deterministic verifier. The dataset is gated, so users must accept its access conditions before running it. The same environments and verifiers are also tagged for use as reward sources when training defensive agents.
Sources: Hugging Face Blog.