# IDE-Bench

> An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.

Canonical page: https://benchlm.ai/benchmarks/idebench

- Category: [Coding](/coding)
- Last updated: September 23, 2026 snapshot

## About IDE-Bench

- Year: 2026
- Tasks: 80 tasks across 8 repositories
- Format: Autonomous IDE-agent task completion (pass@1)
- Difficulty: End-to-end software engineering
- Paper: [IDE-Bench](https://arxiv.org/abs/2601.20886)

The published AfterQuery table reports pass@1 accuracy with a ± margin for 15 models. We preserve the source rows as display-only agent results because the model operates through an IDE tool harness inside a container rather than through a normalized model-only prompt.

IDE-Bench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (15 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [Claude Sonnet 4.5](/models/claude-sonnet-4-5) | Anthropic | 87.50% |
| 2 | [GPT-5.2](/models/gpt-5-2) | OpenAI | 85.00% |
| 3 | [Claude Opus 4.5](/models/claude-opus-4-5) | Anthropic | 83.75% |
| 4 | [Claude Haiku 4.5](/models/claude-haiku-4-5) | Anthropic | 78.75% |
| 5 | [GPT-5.1-Codex-Max](/models/gpt-5-1-codex-max) | OpenAI | 73.75% |
| 6 | [Qwen3 Max](/models/qwen3-max) | Alibaba | 65.00% |
| 7 | [Qwen3 Coder](https://www.afterquery.com/leaderboard/ide-bench) | Alibaba | 57.50% |
| 8 | [Gemini 3 Pro](/models/gemini-3-pro) | Google | 55.00% |
| 9 | [Grok 4.1 Fast](/models/grok-4-1-fast) | xAI | 35.00% |
| 10 | [DeepSeek V3.2](/models/deepseek-v3-2) | DeepSeek | 31.25% |
| 11 | [DeepSeek-R1](/models/deepseek-r1) | DeepSeek | 20.00% |
| 12 | [Grok Code Fast 1](/models/grok-code-fast-1) | xAI | 11.25% |
| 13 | [Llama 4 Maverick](/models/llama-4-maverick) | Meta | 2.50% |
| 14 | [Llama 4 Scout](/models/llama-4-scout) | Meta | 2.50% |
| 15 | [Command R+](https://www.afterquery.com/leaderboard/ide-bench) | Cohere | 0.00% |

## FAQ

### What does IDE-Bench measure?

An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.

### Which model leads the published IDE-Bench snapshot?

Claude Sonnet 4.5 currently leads the published IDE-Bench snapshot with a score of 87.50%.

### How many models are evaluated on IDE-Bench?

The September 23, 2026 snapshot contains 15 AI models.

### Does IDE-Bench affect BenchLM's overall score?

Not directly. IDE-Bench is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.