# DeepSWE v1.1 — Mercor evaluation (DeepSWE v1.1 (Mercor))

> Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Canonical page: https://benchlm.ai/benchmarks/mercordeepswe11

- Category: [Coding](/coding)
- Last updated: October 15, 2026 capture

Evaluation results published by [Mercor](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/). Published evaluation aggregates only; task contents and dataset license grants are not included.

We captured Mercor's DeepSWE v1.1 results on October 15, 2026. The table preserves 53 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists programmatic unit tests (all must pass) and a judge of None. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

- [Mercor DeepSWE v1.1 results](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)
- [Mercor APEX benchmark index](https://www.mercor.com/apex/)
- [Mercor methodology](https://www.mercor.com/apex/methodology/)
- [Source paper](https://arxiv.org/abs/2607.07946)
- [Upstream evaluation code](https://github.com/datacurve-ai/deep-swe)
- [Upstream dataset](https://huggingface.co/datasets/datacurve/deep-swe)

## About DeepSWE v1.1 (Mercor)

- Tasks: 113 source-reported tasks
- Format: Pass@1
- Difficulty: Source-specific evaluation
- Paper: [DeepSWE v1.1 evaluation results](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

DeepSWE v1.1 (Mercor) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (53 configurations)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Opus 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.3 · 336 reported samples | Anthropic | 72.30% |
| 2 | [GPT-6.1 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.4 · 339 reported samples | OpenAI | 72.30% |
| 3 | [GPT-6 Astra](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.5 · 339 reported samples | OpenAI | 72.00% |
| 4 | [DeepSeek-V4.1-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.3 · 339 reported samples | DeepSeek | 71.70% |
| 5 | [Opus 5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.8 · 339 reported samples | Anthropic | 71.40% |
| 6 | [Gemini 3.8 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±6.8 · 339 reported samples | Google | 71.40% |
| 7 | [GPT-5.6 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.9 · 339 reported samples | OpenAI | 70.50% |
| 8 | [GPT-6 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 338 reported samples | OpenAI | 70.50% |
| 9 | [Sonnet 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.7 · 339 reported samples | Anthropic | 70.50% |
| 10 | [GLM-5.3](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.3 · 339 reported samples | Zhipu | 70.50% |
| 11 | [GPT-5.6 Terra](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 339 reported samples | OpenAI | 70.20% |
| 12 | [Opus 5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6.9 · 339 reported samples | Anthropic | 70.20% |
| 13 | [Fable 5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 339 reported samples | Anthropic | 67.30% |
| 14 | [Fable 5.1](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±6.9 · 339 reported samples | Anthropic | 67.30% |
| 15 | [GPT-6 Luna](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.8 · 339 reported samples | OpenAI | 67.00% |
| 16 | [Kimi K3](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±8.8 · 113 reported samples | Kimi | 66.40% |
| 17 | [GPT-5.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6.9 · 339 reported samples | OpenAI | 66.10% |
| 18 | [Gemini 3.7 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±7.1 · 339 reported samples | Google | 65.50% |
| 19 | [Fable 5.1](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 339 reported samples | Anthropic | 64.90% |
| 20 | [GPT-5.6 Luna](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.4 · 339 reported samples | OpenAI | 64.30% |
| 21 | [Grok 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±7.4 · 339 reported samples | xAI | 63.70% |
| 22 | [Haiku 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.7 · 339 reported samples | Anthropic | 59.90% |
| 23 | [Opus 4.8](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.1 · 339 reported samples | Anthropic | 59.60% |
| 24 | [DeepSeek-V4-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±8.8 · 113 reported samples | DeepSeek | 56.60% |
| 25 | [DeepSeek-V4-Pro-0813](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 339 reported samples | DeepSeek | 56.30% |
| 26 | [Qwen3.8-Max](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±8.8 · 113 reported samples | Alibaba | 53.10% |
| 27 | [Sonnet 5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±7.2 · 339 reported samples | Anthropic | 49.60% |
| 28 | [Grok 4.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±7.4 · 339 reported samples | xAI | 49.30% |
| 29 | [Opus 4.7](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.9 · 339 reported samples | Anthropic | 49.30% |
| 30 | [Gemini 3.6 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±7.1 · 339 reported samples | Google | 49.30% |
| 31 | [GPT-5.4](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6.8 · 339 reported samples | OpenAI | 49.00% |
| 32 | [GLM-5.3-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.5 · 339 reported samples | Zhipu | 42.50% |
| 33 | [Muse Spark 1.1](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6.5 · 339 reported samples | Meta | 40.40% |
| 34 | [GLM-5.2](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.5 · 339 reported samples | Zhipu | 38.90% |
| 35 | [Gemini 3.5 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±6.9 · 339 reported samples | Google | 36.60% |
| 36 | [Sonnet 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±6.3 · 339 reported samples | Anthropic | 34.20% |
| 37 | [Grok 4.7](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6 · 337 reported samples | xAI | 33.30% |
| 38 | [Opus 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±6.3 · 339 reported samples | Anthropic | 31.90% |
| 39 | [Kimi K2.7 Code](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±8.4 · 113 reported samples | Kimi | 30.10% |
| 40 | [Grok 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoning · Published margin ±6.8 · 339 reported samples | xAI | 24.80% |
| 41 | [GLM-5.1](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · Published margin ±5.5 · 339 reported samples | Zhipu | 18.90% |
| 42 | [MiniMax-M3](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±3.1 · 339 reported samples | MiniMax | 9.10% |
| 43 | [DeepSeek-V4-Pro](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning · Published margin ±3.8 · 337 reported samples | DeepSeek | 9.10% |
| 44 | [Gemini 3.1 Pro](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±3.1 · 339 reported samples | Google | 7.70% |
| 45 | [Inkling](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±2.8 · 339 reported samples | Thinking Machines | 5.00% |
| 46 | [GLM-5 (Thinking)](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · Published margin ±1.9 · 339 reported samples | Zhipu | 2.10% |
| 47 | [DeepSeek-V3.2](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · Published margin ±0.9 · 339 reported samples | DeepSeek | 0.60% |
| 48 | [Nemotron 3 Ultra](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±0.7 · 339 reported samples | NVIDIA | 0.60% |
| 49 | [GPT-OSS-120B](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±0 · 339 reported samples | OpenAI | 0.00% |
| 50 | [Kimi K2 (Thinking)](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±0 · 339 reported samples | Kimi | 0.00% |
| 51 | [MiniMax-M2.7](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · high reasoning · Published margin ±0 · 337 reported samples | MiniMax | 0.00% |
| 52 | [Qwen3.5](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · Published margin ±0 · 113 reported samples | Alibaba | 0.00% |
| 53 | [Gemma 4 31B](https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/) | mini-swe-agent (500 max steps, 2 hour time limit) · Published margin ±0 · 339 reported samples | Google | 0.00% |

## FAQ

### What does DeepSWE v1.1 (Mercor) measure?

Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

### Which model leads the published DeepSWE v1.1 (Mercor) snapshot?

Opus 5.5 currently leads the published DeepSWE v1.1 (Mercor) snapshot with a score of 72.30%.

### How many models are evaluated on DeepSWE v1.1 (Mercor)?

The October 15, 2026 capture contains 53 source configurations.

### Does DeepSWE v1.1 (Mercor) affect BenchLM's overall score?

Not directly. DeepSWE v1.1 (Mercor) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.