Senior SWE-Bench
We show this table for reference; we do not rank on it.
A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.
Tasteful solve rate (pass@1) on Senior SWE-Bench — v2026.06 release
We mirror the published tasteful solve rate (pass@1) view for Senior SWE-Bench. Claude Opus 5.5 leads the public snapshot at 39.3%, followed by GPT-6 Astra (36.8%) and Claude Fable 5.1 (34.7%). We do not use these results to rank models overall.
Claude Opus 5.5
Anthropic
high reasoning
GPT-6 Astra
OpenAI
high reasoning
Claude Fable 5.1
Anthropic
medium reasoning
10 modelsCodingCurrentDisplay onlyUpdated v2026.06 release
Tasteful solve rate (pass@1) table (10 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Opus 5.5Anthropichigh reasoning | 39.3% | Not reported | Closed |
| 2 | GPT-6 AstraOpenAIhigh reasoning | 36.8% | Not reported | Closed |
| 3 | Claude Fable 5.1Anthropicmedium reasoning | 34.7% | Not reported | Closed |
| 4 | Claude Fable 5Anthropichigh reasoning | 34.7% | Not reported | Closed |
| 5 | Claude Opus 5Anthropichigh reasoning | 34.7% | Not reported | Closed |
| 6 | GPT-5.6 SolOpenAIxhigh reasoning | 34.7% | Not reported | Closed |
| 7 | Claude Opus 4.8Anthropicmax reasoning | 30.5% | Not reported | Closed |
| 8 | GPT-5.5OpenAIxhigh reasoning | 29.5% | Not reported | Closed |
| 9 | Claude Sonnet 5Anthropicmax reasoning | 28% | Not reported | Closed |
| 10 | Gemini 4 ArgonGooglemedium reasoning | 27.7% | Not reported | Closed |
How Senior SWE-Bench is shown here
BenchLM mirrors the official Senior SWE-Bench leaderboard published by Snorkel AI for the v2026.06 release. The benchmark contains 100 long-horizon tasks sourced from real pull requests across 12 production repositories, with 50 tasks public and 50 held private to mitigate contamination.
This page ranks models by the primary published metric: tasteful solve rate (pass@1), scored by a taste judge and a validation agent judge that Snorkel calibrated against reviews from its senior software engineering expert network. Snorkel removes runs with detected reward hacking, such as agents searching GitHub for the original pull request, from published scores.
Senior SWE-Bench is display only on BenchLM. The published rows are long-horizon agent-harness results with judge-based scoring rather than normalized model-only comparisons, and half the task suite is private, so BenchLM does not use these scores as weighted ranking inputs.
Snapshot
The published Senior SWE-Bench snapshot places Claude Opus 5.5 first at 39.3%. The third row is 4.6 points behind. The broader top-10 range is 11.6 points, so the table still separates the published systems.
10 models have been evaluated on Senior SWE-Bench. The benchmark falls in the Coding category. Senior SWE-Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Senior SWE-Bench
Year
2026
Tasks
Senior-level repository tasks
Format
Agentic software-engineering evaluation
Difficulty
Professional senior engineering
BenchLM tracks Senior SWE-Bench as source metadata for now. The public page describes 50 public and 50 private tasks and renders a model comparison chart, but the crawler output does not expose exact aggregate model scores for each row.
Freshness and provenance
Version
Senior SWE-Bench v2026.06
Refresh cadence
Quarterly
Staleness state
Current
Question availability
50 of 100 tasks public as a Harbor dataset
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Senior SWE-Bench measure?
A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.
Which model leads the published Senior SWE-Bench snapshot?
Claude Opus 5.5 currently leads the published Senior SWE-Bench snapshot with 39.3% tasteful solve rate (pass@1). BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on Senior SWE-Bench?
The v2026.06 release snapshot contains 10 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.