Aikido Arena
We show this table for reference; we do not rank on it.
Aikido Security's live benchmark of how well AI models rediscover real vulnerabilities. Each model runs three times on 32 recently disclosed CVEs in open-source repositories, and the score is the share it finds in at least one run (pass@3 recall).
Pass@3 recall on Aikido Arena — Aikido Arena (live leaderboard)
We mirror the published pass@3 recall view for Aikido Arena. GPT-6 Astra leads the public snapshot at 90.6%, followed by DeepSeek V4.1 Flash 1017 (87.5%) and Gemini 4 Argon (87.5%). We do not use these results to rank models overall.
GPT-6 Astra
OpenAI
29/32 CVEs · 1 run 75.0% · precision 96.0% · $135.23 per CVE found
DeepSeek V4.1 Flash 1017
DeepSeek
28/32 CVEs · 1 run 70.8% · precision 86.4% · $2.40 per CVE found
Gemini 4 Argon
28/32 CVEs · 1 run 82.3% · precision 93.0% · $59.59 per CVE found
13 modelsAgenticCurrentDisplay onlyUpdated Aikido Arena (live leaderboard)
Pass@3 recall table (13 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAI29/32 CVEs · 1 run 75.0% · precision 96.0% · $135.23 per CVE found | 90.6% | Not reported | Closed |
| 2 | DeepSeek V4.1 Flash 1017DeepSeek28/32 CVEs · 1 run 70.8% · precision 86.4% · $2.40 per CVE found | 87.5% | Not reported | Open |
| 3 | Gemini 4 ArgonGoogle28/32 CVEs · 1 run 82.3% · precision 93.0% · $59.59 per CVE found | 87.5% | Not reported | Closed |
| 4 | Grok 4.7xAI27/32 CVEs · 1 run 70.8% · precision 93.2% · $58.74 per CVE found | 84.4% | Not reported | Closed |
| 5 | Claude Opus 5Anthropic26/32 CVEs · 1 run 70.8% · precision 87.3% · $64.96 per CVE found | 81.3% | Not reported | Closed |
| 6 | MiMo-V2.6-ProXiaomi25/32 CVEs · 1 run 64.6% · precision 88.0% · $11.79 per CVE found | 78.1% | Not reported | Open |
| 7 | GLM-5.3Z.AI25/32 CVEs · 1 run 65.6% · precision 88.7% · $19.65 per CVE found | 78.1% | Not reported | Open |
| 8 | Gemini 3.8 Flash CyberGoogle25/32 CVEs · 1 run 65.6% · precision 80.4% · $37.14 per CVE found | 78.1% | Not reported | Closed |
| 9 | GPT-5.6 SolOpenAI25/32 CVEs · 1 run 68.8% · precision 96.7% · $56.88 per CVE found | 78.1% | Not reported | Closed |
| 10 | MiniMax M3.1 FlashMiniMax24/32 CVEs · 1 run 62.5% · precision 80.5% · $4.20 per CVE found | 75.0% | Not reported | Closed |
| 11 | GPT-5.6 LunaOpenAI22/32 CVEs · 1 run 47.9% · precision 92.0% · $3.43 per CVE found | 68.8% | Not reported | Closed |
| 12 | GPT-6 SolOpenAI22/32 CVEs · 1 run 56.3% · precision 94.7% · $34.18 per CVE found | 68.8% | Not reported | Closed |
| 13 | GPT-6 LunaOpenAI17/32 CVEs · 1 run 40.6% · precision 95.1% · $2.01 per CVE found | 53.1% | Not reported | Closed |
Aikido Arena v1
BenchLM mirrors the pass@3 recall table from Aikido's July 2026 known-CVE benchmark. It is excluded from BenchLM overall and category rankings.
| Source rank | Source model or configuration | Pass@3 recall | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | GPT-5.6 TerraOpenAI | 88.5% | high reasoningOriginal result | Not reported | Closed |
| 2 | GPT-5.6 TerraOpenAI | 88.5% | max reasoningOriginal result | Not reported | Closed |
| 3 | GPT-5.6 SolOpenAI | 88.5% | high reasoningOriginal result | Not reported | Closed |
| 4 | GPT-5.6 SolOpenAI | 88.5% | max reasoningOriginal result | Not reported | Closed |
| 5 | Kimi K3Moonshot AI | 88.5% | max reasoningOriginal result | Not reported | Open |
| 6 | GPT-5.6 LunaOpenAI | 84.6% | high reasoningOriginal result | Not reported | Closed |
| 7 | GPT-5.5OpenAI | 84.6% | xhigh reasoningOriginal result | Not reported | Closed |
| 8 | GPT-5.6 LunaOpenAI | 80.8% | max reasoningOriginal result | Not reported | Closed |
| 9 | GPT-5.4 miniOpenAI | 80.8% | xhigh reasoningOriginal result | Not reported | Closed |
| 10 | GPT-5.4 miniOpenAI | 76.9% | high reasoningOriginal result | Not reported | Closed |
| 11 | Grok 4.5xAI | 76.9% | max reasoningOriginal result | Not reported | Closed |
| 12 | Grok 4.5xAI | 76.9% | high reasoningOriginal result | Not reported | Closed |
| 13 | GPT-5.5OpenAI | 73.1% | high reasoningOriginal result | Not reported | Closed |
| 14 | GPT-5.4 nanoOpenAI | 69.2% | high reasoningOriginal result | Not reported | Closed |
| 15 | Claude Opus 4.8Anthropic | 69.2% | xhigh reasoningOriginal result | Not reported | Closed |
| 16 | GPT-5.4 nanoOpenAI | 65.4% | xhigh reasoningOriginal result | Not reported | Closed |
| 17 | Gemini 3.1 ProGoogle | 65.4% | high reasoningOriginal result | Not reported | Closed |
| 18 | GLM-5.2Z.AI | 61.5% | max reasoningOriginal result | Not reported | Open |
| 19 | Claude Opus 4.8Anthropic | 61.5% | high reasoningOriginal result | Not reported | Closed |
| 20 | Claude Opus 4.7Anthropic | 57.7% | high reasoningOriginal result | Not reported | Closed |
| 21 | Claude Opus 4.7Anthropic | 57.7% | xhigh reasoningOriginal result | Not reported | Closed |
| 22 | Claude Haiku 4.5Anthropic | 53.8% | xhigh reasoningOriginal result | Not reported | Closed |
| 23 | Gemini 3.5 FlashGoogle | 53.8% | high reasoningOriginal result | Not reported | Closed |
| 24 | Claude Haiku 4.5Anthropic | 50.0% | high reasoningOriginal result | Not reported | Closed |
| 25 | GLM-5.2Z.AI | 50.0% | high reasoningOriginal result | Not reported | Open |
How to read this leaderboard
Higher is better. GPT-6 Astra leads with 29 of 32 CVEs (90.6%), ahead of DeepSeek V4.1 Flash 1017 (28, 87.5%) and Grok 4.7 (27, 84.4%). Read recall with cost and precision: DeepSeek V4.1 Flash 1017 costs $2.40 per CVE found against $135.23 for GPT-6 Astra, and Gemini 3.8 Flash Cyber has the lowest precision on the board (80.4%). Pooling runs matters: most models find 10 to 20 points more over three runs than in one.
Operator receipt: 13 sourced rows are currently displayable on this page; the leading published row is GPT-6 Astra at 90.6%.
Honest limit: The dataset is private and small (32 cases), so one CVE moves a score by about 3 points. Results depend on Aikido's harness and prompts; a model in another harness may do better or worse. Aikido sells the harness the benchmark runs in. Retired models leave the board, so the table is not a complete history. BenchLM shows Aikido Arena as display-only; it does not enter overall or category rankings.
How Aikido Arena is shown here
We mirror the live Aikido Arena leaderboard from Aikido Security. Each model runs 3 times on 32 recently disclosed CVEs in real open-source repositories, inside Aikido's code-analysis harness with a 30-turn limit and no internet access. The score is pass@3 recall: the share of CVEs found in at least one of the three runs.
Each row also gives the single-run recall, precision (the share of reported findings that were real) and the cost of the three runs per CVE found. Aikido retires older models from the live board, and BenchLM keeps only the models the board shows. Aikido Arena is display only on BenchLM. The archived first version, 26 known CVEs from July 2026, sits below on this page.
Snapshot
The published Aikido Arena snapshot places GPT-6 Astra first at 90.6%. The third row is 3.1 points behind. The broader top-10 range is 15.6 points, so the table still separates the published systems.
13 models have been evaluated on Aikido Arena. The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. Aikido Arena is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Aikido Arena
Year
2026
Tasks
32 recently disclosed CVEs in real open-source repositories
Format
Three runs per model in Aikido's code-analysis harness (30 turns, no internet)
Difficulty
Rediscovering real vulnerabilities from source, including missing checks and long call chains
Aikido swaps each model into the main exploration and reasoning stage of a bounded version of its AI code-analysis harness, with a 30-turn limit and no internet access. The 32 cases are recently disclosed CVEs, picked to lower the chance that a model saw the bug, write-up or patch in training; cases, prompts, tools and grading were frozen across runs. Every model runs the full set three times. The board reports recall for one run (the average), two runs and three runs pooled, precision (the share of reported findings that were real) and total cost, from which BenchLM shows the cost per CVE found. Aikido adds models as they ship and retires older ones from the live board; BenchLM mirrors the models the board shows. The first version, published July 16, 2026, used 26 known CVEs from the GitHub advisory database with investigators pointed at the vulnerable code; its table is kept below on this page.
Aikido Arena is one of the cyber benchmarks we track. The cybersecurity benchmark comparison sets it beside the others, one table per security job.
Freshness and provenance
Version
Aikido Arena 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Aikido Arena measure?
Whether an AI model, placed in Aikido's code-analysis harness, can rediscover real vulnerabilities from source code. Each model runs three times on 32 recently disclosed CVEs, and the main score is the share found in at least one run (pass@3 recall).
Which model leads Aikido Arena?
GPT-6 Astra finds 29 of 32 CVEs (90.6%), followed by DeepSeek V4.1 Flash 1017 with 28 and Grok 4.7 with 27. DeepSeek V4.1 Flash 1017 does it for $2.40 per CVE found, against $135.23 for GPT-6 Astra.
What was the first version of the benchmark?
Aikido's July 16, 2026 study ran 13 models (and Kimi K3 a few days later) on 26 known CVEs from the GitHub advisory database, at two reasoning levels each. GPT-5.6 Terra and Sol, and Kimi K3, reached 23 of 26. That table is archived below the live board on this page; its cases differ, so compare within one table.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.