IDE-Bench
We show this table for reference; we do not rank on it.
An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.
No longer maintained · AfterQuery took this leaderboard down in October 2026, so it no longer updates. The table below is the last version we captured (October 12, 2026 snapshot) and stays here for reference.
Pass@1 accuracy on IDE-Bench — October 12, 2026 snapshot
We mirror the published pass@1 accuracy view for IDE-Bench. Claude Sonnet 4.5 leads the public snapshot at 87.50%, followed by GPT-5.2 (85.00%) and Claude Opus 4.5 (83.75%). We do not use these results to rank models overall.
Claude Sonnet 4.5
Anthropic
GPT-5.2
OpenAI
Claude Opus 4.5
Anthropic
15 modelsCodingCurrentDisplay onlyUpdated October 12, 2026 snapshot
Pass@1 accuracy table (15 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Sonnet 4.5Anthropic | 87.50% | Not reported | Closed |
| 2 | GPT-5.2OpenAI | 85.00% | Not reported | Closed |
| 3 | Claude Opus 4.5Anthropic | 83.75% | Not reported | Closed |
| 4 | Claude Haiku 4.5Anthropic | 78.75% | Not reported | Closed |
| 5 | GPT-5.1-Codex-MaxOpenAI | 73.75% | Not reported | Closed |
| 6 | Qwen3 MaxAlibaba | 65.00% | Not reported | Closed |
| 7 | Qwen3 CoderAlibaba | 57.50% | Not reported | Unknown |
| 8 | Gemini 3 ProGoogle | 55.00% | Not reported | Closed |
| 9 | Grok 4.1 FastxAI | 35.00% | Not reported | Closed |
| 10 | DeepSeek V3.2DeepSeek | 31.25% | Not reported | Open |
| 11 | DeepSeek-R1DeepSeek | 20.00% | Not reported | Open |
| 12 | Grok Code Fast 1xAI | 11.25% | Not reported | Closed |
| 13 | Llama 4 MaverickMeta | 2.50% | Not reported | Open |
| 14 | Llama 4 ScoutMeta | 2.50% | Not reported | Open |
| 15 | Command R+Cohere | 0.00% | Not reported | Unknown |
How we show IDE-Bench
We mirror AfterQuery's IDE-Bench pass@1 table captured on October 12, 2026 snapshot. The benchmark contains 80 software-engineering tasks across eight repositories, and the snapshot preserves the published score margins for all 15 rows.
IDE-Bench measures model performance inside an autonomous IDE-agent harness. We keep the table display-only because the result belongs to the complete model, tools, container, and grading setup rather than to a normalized base-model run.
Snapshot
The published IDE-Bench snapshot places Claude Sonnet 4.5 first at 87.50%. The third row is 3.75 points behind. The broader top-10 range is 56.25 points, so the table still separates the published systems.
15 models have been evaluated on IDE-Bench. The benchmark falls in the Coding category. IDE-Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About IDE-Bench
Year
2026
Tasks
80 tasks across 8 repositories
Format
Autonomous IDE-agent task completion (pass@1)
Difficulty
End-to-end software engineering
The published AfterQuery table reports pass@1 accuracy with a ± margin for 15 models. We preserve the source rows as display-only agent results because the model operates through an IDE tool harness inside a container rather than through a normalized model-only prompt.
Freshness and provenance
Version
IDE-Bench 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does IDE-Bench measure?
An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.
Which model leads the published IDE-Bench snapshot?
Claude Sonnet 4.5 currently leads the published IDE-Bench snapshot with 87.50% pass@1 accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on IDE-Bench?
The October 12, 2026 snapshot snapshot contains 15 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.