VulcanBench v3
We mirror this table; we do not rank on it.
An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.
Benchmark score on VulcanBench v3 — September 18, 2026
We mirror the published score view for VulcanBench v3. Grok 4.5 leads the public snapshot at 89.9%, followed by Claude Fable 5 (89.5%) and DeepSeek V4 Flash 0731 (88.4%). We do not use these results to rank models overall.
Grok 4.5
xAI
Claude Fable 5
Anthropic
DeepSeek V4 Flash 0731
DeepSeek
15 modelsCodingCurrentDisplay onlyUpdated September 18, 2026
Benchmark score table (15 models)
ScoreThe published VulcanBench v3 snapshot places Grok 4.5 first at 89.9%. The third row is 1.5 points behind. The broader top-10 range is 4.4 points, so many of the published results sit in a relatively narrow band.
15 models have been evaluated on VulcanBench v3. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. VulcanBench v3 is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About VulcanBench v3
Year
2026
Tasks
23 post-cutoff repository tasks in the v3 report
Format
Pass@1 with low, medium, and high effort
Difficulty
Professional multi-file software engineering
VulcanBench v3 evaluates repository-level patches with deterministic hidden tests inside network-isolated Docker sandboxes. The August 24 board covers 23 tasks, 14 models, 39 model-by-effort columns, and 1,582 runs. We mirror each benchmark-owner best-per-model row as display-only evidence. Product-harness studies, extended-budget ablations, and other rows outside the raw uniform-harness board stay separate.
Freshness and provenance
Version
VulcanBench v3 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does VulcanBench v3 measure?
An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.
Which model scores highest on VulcanBench v3?
Grok 4.5 by xAI currently leads with a score of 89.9% on VulcanBench v3.
How many models are evaluated on VulcanBench v3?
15 AI models have been evaluated on VulcanBench v3 on BenchLM.
Compare top models on VulcanBench v3
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 2,000+ readers.
One email each week. Unsubscribe anytime.