# Aikido Arena

> Aikido Security's live benchmark of how well AI models rediscover real vulnerabilities. Each model runs three times on 32 recently disclosed CVEs in open-source repositories, and the score is the share it finds in at least one run (pass@3 recall).

Canonical page: https://benchlm.ai/benchmarks/aikido-arena

- Category: [Agentic](/agentic)
- Last updated: Aikido Arena (live leaderboard)

## About Aikido Arena

- Year: 2026
- Tasks: 32 recently disclosed CVEs in real open-source repositories
- Format: Three runs per model in Aikido's code-analysis harness (30 turns, no internet)
- Difficulty: Rediscovering real vulnerabilities from source, including missing checks and long call chains
- Paper: [Aikido Arena: AI model benchmarks for cybersecurity](https://www.aikido.dev/aikido-arena)

Aikido swaps each model into the main exploration and reasoning stage of a bounded version of its AI code-analysis harness, with a 30-turn limit and no internet access. The 32 cases are recently disclosed CVEs, picked to lower the chance that a model saw the bug, write-up or patch in training; cases, prompts, tools and grading were frozen across runs. Every model runs the full set three times. The board reports recall for one run (the average), two runs and three runs pooled, precision (the share of reported findings that were real) and total cost, from which BenchLM shows the cost per CVE found. Aikido adds models as they ship and retires older ones from the live board; BenchLM mirrors the models the board shows. The first version, published July 16, 2026, used 26 known CVEs from the GitHub advisory database with investigators pointed at the vulnerable code; its table is kept below on this page.

Aikido Arena is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (13 models)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [GPT-6 Astra](/models/gpt-6-astra) | 29/32 CVEs · 1 run 75.0% · precision 96.0% · $135.23 per CVE found | OpenAI | 90.6% |
| 2 | [DeepSeek V4.1 Flash 1017](/models/deepseek-v4-1-flash) | 28/32 CVEs · 1 run 70.8% · precision 86.4% · $2.40 per CVE found | DeepSeek | 87.5% |
| 3 | [Gemini 4 Argon](/models/gemini-4-argon) | 28/32 CVEs · 1 run 82.3% · precision 93.0% · $59.59 per CVE found | Google | 87.5% |
| 4 | [Grok 4.7](/models/grok-4-7) | 27/32 CVEs · 1 run 70.8% · precision 93.2% · $58.74 per CVE found | xAI | 84.4% |
| 5 | [Claude Opus 5](/models/claude-opus-5) | 26/32 CVEs · 1 run 70.8% · precision 87.3% · $64.96 per CVE found | Anthropic | 81.3% |
| 6 | [MiMo-V2.6-Pro](/models/mimo-v2-6-pro) | 25/32 CVEs · 1 run 64.6% · precision 88.0% · $11.79 per CVE found | Xiaomi | 78.1% |
| 7 | [GLM-5.3](/models/glm-5-3) | 25/32 CVEs · 1 run 65.6% · precision 88.7% · $19.65 per CVE found | Z.AI | 78.1% |
| 8 | [Gemini 3.8 Flash Cyber](/models/gemini-3-8-flash-cyber) | 25/32 CVEs · 1 run 65.6% · precision 80.4% · $37.14 per CVE found | Google | 78.1% |
| 9 | [GPT-5.6 Sol](/models/gpt-5-6-sol) | 25/32 CVEs · 1 run 68.8% · precision 96.7% · $56.88 per CVE found | OpenAI | 78.1% |
| 10 | [MiniMax M3.1 Flash](/models/minimax-m3-1-flash) | 24/32 CVEs · 1 run 62.5% · precision 80.5% · $4.20 per CVE found | MiniMax | 75.0% |
| 11 | [GPT-5.6 Luna](/models/gpt-5-6-luna) | 22/32 CVEs · 1 run 47.9% · precision 92.0% · $3.43 per CVE found | OpenAI | 68.8% |
| 12 | [GPT-6 Sol](/models/gpt-6-sol) | 22/32 CVEs · 1 run 56.3% · precision 94.7% · $34.18 per CVE found | OpenAI | 68.8% |
| 13 | [GPT-6 Luna](/models/gpt-6-luna) | 17/32 CVEs · 1 run 40.6% · precision 95.1% · $2.01 per CVE found | OpenAI | 53.1% |

## Aikido Arena v1

Source snapshot: Archived: published July 16, 2026 (Kimi K3 added July 19).

BenchLM mirrors the pass@3 recall table from Aikido's July 2026 known-CVE benchmark. It is excluded from BenchLM overall and category rankings.

- [Aikido known-CVE benchmark (July 2026)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves)

| Source model | Configuration | Pass@3 recall |
|---|---|---|
| [GPT-5.6 Terra (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 23/26 CVEs · 1 run 69.2% · $247 per run | 88.5% |
| [GPT-5.6 Terra (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 23/26 CVEs · 1 run 83.3% · $537 per run | 88.5% |
| [GPT-5.6 Sol (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 23/26 CVEs · 1 run 78.2% · $561 per run | 88.5% |
| [GPT-5.6 Sol (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 23/26 CVEs · 1 run 73.1% · $870 per run | 88.5% |
| [Kimi K3 (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | Added in Aikido's July 19 update to the post, which gives 23 of 26 CVEs (pass@3) but no single-run recall or cost. | 88.5% |
| [GPT-5.6 Luna (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 22/26 CVEs · 1 run 65.4% · $138 per run | 84.6% |
| [GPT-5.5 (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 22/26 CVEs · 1 run 73.1% · $746 per run | 84.6% |
| [GPT-5.6 Luna (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 21/26 CVEs · 1 run 74.4% · $213 per run | 80.8% |
| [GPT-5.4 mini (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 21/26 CVEs · 1 run 64.1% · $246 per run | 80.8% |
| [GPT-5.4 mini (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 20/26 CVEs · 1 run 64.1% · $150 per run | 76.9% |
| [Grok 4.5 (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 20/26 CVEs · 1 run 66.7% · $170 per run | 76.9% |
| [Grok 4.5 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 20/26 CVEs · 1 run 62.8% · $182 per run | 76.9% |
| [GPT-5.5 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 19/26 CVEs · 1 run 62.8% · $504 per run | 73.1% |
| [GPT-5.4 nano (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 18/26 CVEs · 1 run 50.0% · $55 per run | 69.2% |
| [Claude Opus 4.8 (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 18/26 CVEs · 1 run 55.1% · $334 per run | 69.2% |
| [GPT-5.4 nano (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 17/26 CVEs · 1 run 57.7% · $88 per run | 65.4% |
| [Gemini 3.1 Pro (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 17/26 CVEs · 1 run 47.4% · $274 per run | 65.4% |
| [GLM-5.2 (max)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 16/26 CVEs · 1 run 46.2% · $80 per run | 61.5% |
| [Claude Opus 4.8 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 16/26 CVEs · 1 run 47.4% · $165 per run | 61.5% |
| [Claude Opus 4.7 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 15/26 CVEs · 1 run 44.9% · $145 per run | 57.7% |
| [Claude Opus 4.7 (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 15/26 CVEs · 1 run 38.5% · $256 per run | 57.7% |
| [Claude Haiku 4.5 (xhigh)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 14/26 CVEs · 1 run 41.0% · $50 per run | 53.8% |
| [Gemini 3.5 Flash (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 14/26 CVEs · 1 run 44.9% · $169 per run | 53.8% |
| [Claude Haiku 4.5 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 13/26 CVEs · 1 run 38.5% · $47 per run | 50.0% |
| [GLM-5.2 (high)](https://www.aikido.dev/blog/benchmarking-ai-models-known-cves) | 13/26 CVEs · 1 run 37.2% · $60 per run | 50.0% |

## FAQ

### What does Aikido Arena measure?

Whether an AI model, placed in Aikido's code-analysis harness, can rediscover real vulnerabilities from source code. Each model runs three times on 32 recently disclosed CVEs, and the main score is the share found in at least one run (pass@3 recall).

### Which model leads Aikido Arena?

GPT-6 Astra finds 29 of 32 CVEs (90.6%), followed by DeepSeek V4.1 Flash 1017 with 28 and Grok 4.7 with 27. DeepSeek V4.1 Flash 1017 does it for $2.40 per CVE found, against $135.23 for GPT-6 Astra.

### What was the first version of the benchmark?

Aikido's July 16, 2026 study ran 13 models (and Kimi K3 a few days later) on 26 known CVEs from the GitHub advisory database, at two reasoning levels each. GPT-5.6 Terra and Sol, and Kimi K3, reached 23 of 26. That table is archived below the live board on this page; its cases differ, so compare within one table.