DeepSWE
A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.
Data verified 37 confirmed releases in the last 30 daysSee provider release alertsHow we show DeepSWE
We mirror the public DeepSWE leaderboard JSON from Datacurve. The snapshot shows the best available mini-swe-agent configuration per model, while preserving 70 underlying effort-level rows in the source metadata.
DeepSWE evaluates coding agents on 113 original, long-horizon software engineering tasks across 91 repositories and 5 languages, using isolated task environments and program-based verifiers.
We keep DeepSWE display only. Each public row combines a model, mini-swe-agent harness, and reasoning-effort setting, so the scores do not enter the weighted model-only rankings.
Snapshot
Pass@1 on DeepSWE — September 1, 2026
We mirror the published pass@1 view for DeepSWE. Gemini 3.8 Flash leads the public snapshot at 73.8%, followed by Claude Opus 5 (73.6%) and GPT-6 Astra (73.2%). We do not use these results to rank models overall.
Gemini 3.8 Flash
mini-swe-agent · high reasoning
Claude Opus 5
Anthropic
mini-swe-agent · max reasoning
GPT-6 Astra
OpenAI
mini-swe-agent · max reasoning
28 modelsCodingCurrentDisplay onlyUpdated September 1, 2026
Pass@1 table (28 models)
ScoreThe published DeepSWE snapshot places Gemini 3.8 Flash first at 73.8%. The third row is 0.6 points behind. The broader top-10 range is 6.8 points, so many of the published results sit in a relatively narrow band.
28 models have been evaluated on DeepSWE. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. DeepSWE is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About DeepSWE
Year
2026
Tasks
113 software engineering tasks across 91 repositories and 5 languages
Format
Pass@1 with confidence interval, cost, time, and token metadata
Difficulty
Long-horizon software engineering
DeepSWE includes original tasks with isolated environments and program-based verifiers. BenchLM mirrors the public DeepSWE leaderboard JSON as display-only, using the best available mini-swe-agent configuration per model and preserving cost, time, token, and effort-level source metadata. Each row combines a model, agent harness, and reasoning-effort setting rather than a pure model-only benchmark score.
BenchLM freshness & provenance
Version
DeepSWE 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does DeepSWE measure?
A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.
Which model leads the published DeepSWE snapshot?
Gemini 3.8 Flash currently leads the published DeepSWE snapshot with 73.8% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on DeepSWE?
The September 1, 2026 snapshot contains 28 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 2,000+ readers.
One email each week. Unsubscribe anytime.