Benchmark profile
IDE-Bench
An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.
How we show IDE-Bench
We mirror AfterQuery's IDE-Bench pass@1 table captured on July 18, 2026 snapshot. The benchmark contains 80 software-engineering tasks across eight repositories, and the snapshot preserves the published score margins for all 15 rows.
IDE-Bench measures model performance inside an autonomous IDE-agent harness. We keep the table display-only because the result belongs to the complete model, tools, container, and grading setup rather than to a normalized base-model run.
Pass@1 accuracy on IDE-Bench — July 18, 2026 snapshot
BenchLM mirrors the published pass@1 accuracy view for IDE-Bench. Claude Sonnet 4.5 leads the public snapshot at 87.50% , followed by GPT-5.2 (85.00%) and Claude Opus 4.5 (83.75%). BenchLM does not use these results to rank models overall.
Claude Sonnet 4.5
Anthropic
afterquery/ide-bench/claude-sonnet-4-5
GPT-5.2
OpenAI
afterquery/ide-bench/gpt-5-2
Claude Opus 4.5
Anthropic
afterquery/ide-bench/claude-opus-4-5
Pass@1 accuracy table (15 models)
ScoreThe published IDE-Bench snapshot places Claude Sonnet 4.5 first at 87.50%. The third row is 3.75 points behind. The broader top-10 range is 56.25 points, so the table still separates the published systems.
15 models have been evaluated on IDE-Bench. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. IDE-Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About IDE-Bench
Year
2026
Tasks
80 tasks across 8 repositories
Format
Autonomous IDE-agent task completion (pass@1)
Difficulty
End-to-end software engineering
The published AfterQuery table reports pass@1 accuracy with a ± margin for 15 models. We preserve the source rows as display-only agent results because the model operates through an IDE tool harness inside a container rather than through a normalized model-only prompt.
BenchLM freshness & provenance
Version
IDE-Bench 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does IDE-Bench measure?
An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.
Which model leads the published IDE-Bench snapshot?
Claude Sonnet 4.5 currently leads the published IDE-Bench snapshot with 87.50% pass@1 accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on IDE-Bench?
15 AI models are included in BenchLM's mirrored IDE-Bench snapshot, based on the public leaderboard captured on July 18, 2026 snapshot.
The AI models change fast. We track them for you.
A weekly brief for engineers and researchers covering new models, ranking shifts, and pricing changes.
Free. No spam. Unsubscribe anytime.