SWE-bench Pro
A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.
Data verified 37 confirmed releases in the last 30 daysSee provider release alertsClaude Fable 5.1 leads the SWE-bench Pro leaderboard on BenchLM's September 2026 update with 81.2%, ahead of Claude Mythos 5 (80.3%) and Claude Fable 5 (80%), across 72 models.
How to read this leaderboard
Editorial review by Glevd · 2026-07-15
Use SWE-bench Pro as evidence for long-horizon repository work only after matching the split, scaffold, tool budget, retry policy, and run count. The table preserves exact published rows from different providers, so a small score gap is directional unless the setups match.
Operator receipt: 72 sourced rows are currently displayable on this page; the leading published row is Claude Fable 5.1 at 81.2%.
Honest limit: OpenAI's July 2026 audit estimated that about 30% of the 731-task public split is broken and retracted its earlier recommendation to adopt the benchmark. The current rows remain useful published receipts, but this page should not decide a coding-agent purchase by itself.
Top models on SWE-bench Pro — September 10, 2026
As of September 10, 2026, Claude Fable 5.1 leads the SWE-bench Pro leaderboard with 81.2% , followed by Claude Mythos 5 (80.3%) and Claude Fable 5 (80%).
Claude Fable 5.1
Anthropic
Claude Mythos 5
Anthropic
Claude Fable 5
Anthropic
72 modelsCoding25% of category scoreCurrentUpdated September 10, 2026
Leaderboard (72 models)
ScoreAccording to BenchLM.ai, Claude Fable 5.1 leads the SWE-bench Pro benchmark with a score of 81.2%, followed by Claude Mythos 5 (80.3%) and Claude Fable 5 (80%). The top models are clustered within 1.2 points, suggesting this benchmark is nearing saturation for frontier models.
72 models have been evaluated on SWE-bench Pro. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. Within that category, SWE-bench Pro contributes 25% of the category score, so strong performance here directly affects a model's overall ranking.
About SWE-bench Pro
Year
2025
Tasks
1,865 repository problems
Format
Repository task completion
Difficulty
Long-horizon professional engineering
The authors assembled 1,865 problems from 41 repositories across public, held-out, and commercial splits. Agents receive a repository and issue, then produce a patch that must pass the evaluation tests without breaking existing behavior.
BenchLM freshness & provenance
Version
SWE-bench Pro 2025
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does SWE-bench Pro measure?
SWE-bench Pro gives an agent a repository and issue description, then checks whether its patch passes new tests without breaking existing behavior. The full benchmark contains 1,865 problems from 41 repositories across public, held-out, and commercial splits, with work that can span multiple files and long execution horizons.
Are SWE-bench Pro scores directly comparable?
Only when the split and evaluation setup match. Public, held-out, and commercial tasks are different pools, while scaffold, tool budget, retry policy, and token budget can change pass rates. This page preserves exact published rows, but it does not pretend every provider ran the same harness.
Should SWE-bench Pro decide which coding agent to use?
No. The benchmark covers realistic repository work, but OpenAI's July 2026 audit estimated that about 30% of the public tasks are broken and retracted its earlier adoption recommendation. Use SWE-bench Pro alongside LiveCodeBench, other repository evaluations, and a workload-specific trial instead of treating one score as a procurement decision.
Compare Top Models on SWE-bench Pro
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 2,000+ readers.
One email each week. Unsubscribe anytime.