VulcanBench v3
An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.
Top models on VulcanBench v3 — November 4, 2026
As of November 4, 2026, Grok 4.5 leads the VulcanBench v3 leaderboard with 89.9% , followed by Claude Fable 5 (89.5%) and DeepSeek V4 Flash 0731 (88.4%).
Grok 4.5
xAI
Claude Fable 5
Anthropic
DeepSeek V4 Flash 0731
DeepSeek
15 modelsCoding3% of Coding reference weightCurrentUpdated November 4, 2026
Leaderboard (15 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Grok 4.5xAI | 89.9% | Not reported | Closed |
| 2 | Claude Fable 5Anthropic | 89.5% | Not reported | Closed |
| 3 | DeepSeek V4 Flash 0731DeepSeek | 88.4% | Not reported | Open |
| 4 | Claude Opus 5Anthropic | 87.0% | Not reported | Closed |
| 5 | GPT-5.6 SolOpenAI | 87.0% | Not reported | Closed |
| 6 | GPT-5.6 TerraOpenAI | 87.0% | Not reported | Closed |
| 7 | Muse Spark 1.2Meta | 87.0% | Not reported | Closed |
| 8 | Grok 4.6xAI | 87.0% | Not reported | Closed |
| 9 | DeepSeek V4 Pro (High)DeepSeek | 87.0% | Not reported | Open |
| 10 | GPT-5.6 LunaOpenAI | 85.5% | Not reported | Closed |
| 11 | Qwen3.8-27BAlibaba | 82.6% | Not reported | Open |
| 12 | Qwen3.8 MaxAlibaba | 81.2% | Not reported | Open |
| 13 | GLM-5.3Z.AI | 78.3% | Not reported | Open |
| 14 | Claude Haiku 4.5Anthropic | 76.2% | Not reported | Closed |
| 15 | Kimi K3Moonshot AI | 73.7% | Not reported | Open |
According to BenchLM.ai, Grok 4.5 leads the VulcanBench v3 benchmark with a score of 89.9%, followed by Claude Fable 5 (89.5%) and DeepSeek V4 Flash 0731 (88.4%). The top models are clustered within 1.5 points, suggesting this benchmark is nearing saturation for frontier models.
15 models have been evaluated on VulcanBench v3. The benchmark falls in the Coding category. BenchAlign v5.8 gives VulcanBench v3 3% of the Coding reference weight, so it moves the Coding leaderboard and the overall ranking. Reference weights are relative weights in the calibrated model, not fixed shares of a score.
About VulcanBench v3
Year
2026
Tasks
23 post-cutoff repository tasks in the v3 report
Format
Pass@1 with low, medium, and high effort
Difficulty
Professional multi-file software engineering
VulcanBench v3 evaluates repository-level patches with deterministic hidden tests inside network-isolated Docker sandboxes. The August 24 board covers 23 tasks, 14 models, 39 model-by-effort columns, and 1,582 runs. We mirror each benchmark-owner best-per-model row. Product-harness studies, extended-budget ablations, and other rows outside the raw uniform-harness board stay separate.
Freshness and provenance
Version
VulcanBench v3 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does VulcanBench v3 measure?
An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.
Which model scores highest on VulcanBench v3?
Grok 4.5 by xAI currently leads with a score of 89.9% on VulcanBench v3.
How many models are evaluated on VulcanBench v3?
15 AI models have published results on VulcanBench v3 in the BenchLM catalog.
Compare top models on VulcanBench v3
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.