Artificial Analysis CWE-Bench-AA (CWE-Bench-AA)
We show this table for reference; we do not rank on it.
Artificial Analysis' implementation of Collinear AI's CWE-bench, testing whether agents can patch real-world software vulnerabilities. One of the three components of the AA Cyber Index.
Benchmark score on CWE-Bench-AA — October 29, 2026
We compile the CWE-Bench-AA rows from secondary reports. DeepSeek V4.1 Flash 1017 leads the table at 68.3%, followed by Grok 4.7 (68.3%) and GPT-6 Sol (64.2%). We do not use these results to rank models overall.
DeepSeek V4.1 Flash 1017
DeepSeek
Grok 4.7
xAI
GPT-6 Sol
OpenAI
19 modelsAgenticCurrentDisplay onlyUpdated October 29, 2026
Benchmark score table (19 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash 1017DeepSeek | 68.3% | Not reported | Open |
| 2 | Grok 4.7xAI | 68.3% | Not reported | Closed |
| 3 | GPT-6 SolOpenAI | 64.2% | Not reported | Closed |
| 4 | GPT-6 AstraOpenAI | 63.3% | Not reported | Closed |
| 5 | MiMo-V2.6-ProXiaomi | 63.3% | Not reported | Open |
| 6 | Kimi K3.1 ProMoonshot AI | 60.8% | Not reported | Pending |
| 7 | Claude Opus 5.5Anthropic | 58.3% | Not reported | Closed |
| 8 | GPT-6 LunaOpenAI | 56.7% | Not reported | Closed |
| 9 | GLM-5.3Z.AI | 55.8% | Not reported | Open |
| 10 | GLM-5.3-FlashZ.AI | 55.8% | Not reported | Open |
| 11 | Kimi K3Moonshot AI | 53.3% | Not reported | Open |
| 12 | Gemini 3.8 FlashGoogle | 52.5% | Not reported | Closed |
| 13 | Step 5 PreviewStepFun | 52.5% | Not reported | Open |
| 14 | Claude Fable 5.1Anthropic | 50.8% | Not reported | Closed |
| 15 | Mistral Large 4Mistral | 50.8% | Not reported | Open |
| 16 | Beam 1.0 501B-A23BReflection AI | 49.2% | Not reported | Open |
| 17 | MiniMax M3MiniMax | 41.7% | Not reported | Open |
| 18 | Nemotron 3 UltraNVIDIA | 27.5% | Not reported | Open |
| 19 | Qwen3.8-27BAlibaba | 26.7% | Not reported | Open |
Among the reported CWE-Bench-AA rows, DeepSeek V4.1 Flash 1017 is first at 68.3%. The third row is 4.1 points behind. The broader top-10 range is 12.5 points, so the table still separates the published systems.
19 models have been evaluated on CWE-Bench-AA. The benchmark falls in the Agentic category. CWE-Bench-AA is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About CWE-Bench-AA
Year
2026
Tasks
Real-world vulnerability patching tasks
Format
Pass@1
Difficulty
Automated vulnerability remediation
Separate from the provider-reported CWE-Bench rows. Independently run by Artificial Analysis under its published harness. Display-only; not yet admitted as independent-run evidence.
Freshness and provenance
Version
CWE-Bench-AA 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does CWE-Bench-AA measure?
Artificial Analysis' implementation of Collinear AI's CWE-bench, testing whether agents can patch real-world software vulnerabilities. One of the three components of the AA Cyber Index.
Which model scores highest on CWE-Bench-AA?
DeepSeek V4.1 Flash 1017 by DeepSeek currently leads with a score of 68.3% on CWE-Bench-AA.
How many models are evaluated on CWE-Bench-AA?
19 AI models have published results on CWE-Bench-AA in the BenchLM catalog.
Compare top models on CWE-Bench-AA
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.