SWE-Marathon
We show this table for reference; we do not rank on it.
A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.
Pass@1 on SWE-Marathon — SWE-Marathon v1.0 archive
We mirror the published pass@1 view for SWE-Marathon. Grok 4.5 leads the public snapshot at 29.0%, followed by Claude Opus 4.8 (26.0%) and Claude Fable 5 (24.0%). We do not use these results to rank models overall.
Grok 4.5
xAI
Claude Opus 4.8
Anthropic
Claude Fable 5
Anthropic
16 modelsCodingCurrentDisplay onlyUpdated SWE-Marathon v1.0 archive
Pass@1 table (16 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Grok 4.5xAI | 29.0% | Not reported | Closed |
| 2 | Claude Opus 4.8Anthropic | 26.0% | Not reported | Closed |
| 3 | Claude Fable 5Anthropic | 24.0% | Not reported | Closed |
| 4 | Claude Opus 4.7Anthropic | 16.0% | Not reported | Closed |
| 5 | GLM 5.2Z.AI | 13.0% | Not reported | Unknown |
| 6 | GPT-5.5OpenAI | 12.0% | Not reported | Closed |
| 7 | Claude Opus 4.7Anthropic | 11.0% | Not reported | Closed |
| 8 | Gemini 3.5 FlashGoogle | 7.0% | Not reported | Closed |
| 9 | GPT-5.5OpenAI | 6.0% | Not reported | Closed |
| 10 | Gemini 3.1 Pro PreviewGoogle | 4.0% | Not reported | Unknown |
| 11 | DeepSeek V4 ProDeepSeek | 4.0% | Not reported | Unknown |
| 12 | Gemini 3.1 Pro PreviewGoogle | 2.0% | Not reported | Unknown |
| 13 | GLM 5.1Z.AI | 1.0% | Not reported | Unknown |
| 14 | MiniMax M2.7MiniMax | 0.0% | Not reported | Open |
| 15 | Kimi K2.6Moonshot AI | 0.0% | Not reported | Open |
| 16 | Kimi K2.6Moonshot AI | 0.0% | Not reported | Open |
How SWE-Marathon is shown here
We mirror the archived SWE-Marathon v1.0 aggregate table for 20 multi-hour software engineering tasks. Each row identifies a model and agent harness; rows with incomplete trials remain labeled in the source snapshot.
We mirror the archived v1.0 aggregate rows with their model and agent harness. Incomplete trials stay labeled. These results remain separate from the changed v1.1 suite and do not enter model rankings.
Snapshot
The published SWE-Marathon snapshot places Grok 4.5 first at 29.0%. The third row is 5.0 points behind. The broader top-10 range is 25.0 points, so the table still separates the published systems.
16 models have been evaluated on SWE-Marathon. The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWE-Marathon is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SWE-Marathon
Year
2026
Tasks
20 multi-hour software engineering tasks
Format
Task resolution and trajectory review
Difficulty
Ultra-long-horizon software engineering
BenchLM tracks SWE-Marathon as a display-only external benchmark. The official v1.0 site reports 20 multi-hour tasks, 1,300 logged trials, task-level leaderboards, and replayable trajectory artifacts; BenchLM keeps it source-metadata-only until there is a stable public aggregate feed.
Freshness and provenance
Version
SWE-Marathon 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SWE-Marathon measure?
A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.
Which model leads the published SWE-Marathon snapshot?
Grok 4.5 currently leads the published SWE-Marathon snapshot with 29.0% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on SWE-Marathon?
The SWE-Marathon v1.0 archive snapshot contains 16 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.