CWE-bench v2: Expanded Coverage, High Quality, and Transparency

Collinear AI has introduced the second version of CWE-bench, an open benchmark for evaluating AI agents in cybersecurity. Each task is based on a real open-source repository: the agent receives a brief instruction and must eliminate the vulnerability while preserving the project's functionality.

The set includes tasks covering 91 vulnerability classes. Result verification is automated and reproducible, and the tasks and evaluation tools have undergone manual review. To track progress, there is also a closed set of tasks that models have not encountered.

According to published results, among the 11 tested models, Gemini 4 Argon leads with scores of 67.8% in pass@1 mode and 81.0% in pass@4 mode. Results for models with open weights are published on the dataset page, while comparisons with API-based models are available in a separate CWE-bench ranking.