DeepSWE v1.1 — Mercor evaluation (DeepSWE v1.1 (Mercor))

We show this table for reference; we do not rank on it.

Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on DeepSWE v1.1 (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for DeepSWE v1.1 (Mercor). Opus 5.5 leads the public snapshot at 72.30%, followed by GPT-6.1 Sol (72.30%) and GPT-6 Astra (72.00%). We do not use these results to rank models overall.

Opus 5.5

Anthropic

mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning

Published margin ±7.3 · 336 reported samples

72.30%

GPT-6.1 Sol

OpenAI

mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning

Published margin ±7.4 · 339 reported samples

72.30%

GPT-6 Astra

OpenAI

mini-swe-agent (500 max steps, 2 hour time limit) · max reasoning

Published margin ±7.5 · 339 reported samples

72.00%

53 configurationsCodingDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture

Pass@1 table (53 configurations)

Pass@1 results for DeepSWE v1.1 (Mercor)
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Opus 5.5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.3 · 336 reported samples	72.30%	Not reported	Unknown
2	GPT-6.1 SolOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.4 · 339 reported samples	72.30%	Not reported	Unknown
3	GPT-6 AstraOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.5 · 339 reported samples	72.00%	Not reported	Unknown
4	DeepSeek-V4.1-FlashDeepSeekmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.3 · 339 reported samples	71.70%	Not reported	Unknown
5	Opus 5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.8 · 339 reported samples	71.40%	Not reported	Unknown
6	Gemini 3.8 FlashGooglemini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±6.8 · 339 reported samples	71.40%	Not reported	Unknown
7	GPT-5.6 SolOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.9 · 339 reported samples	70.50%	Not reported	Unknown
8	GPT-6 SolOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 338 reported samples	70.50%	Not reported	Unknown
9	Sonnet 5.5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.7 · 339 reported samples	70.50%	Not reported	Unknown
10	GLM-5.3Zhipumini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.3 · 339 reported samples	70.50%	Not reported	Unknown
11	GPT-5.6 TerraOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 339 reported samples	70.20%	Not reported	Unknown
12	Opus 5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6.9 · 339 reported samples	70.20%	Not reported	Unknown
13	Fable 5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 339 reported samples	67.30%	Not reported	Unknown
14	Fable 5.1Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±6.9 · 339 reported samples	67.30%	Not reported	Unknown
15	GPT-6 LunaOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.8 · 339 reported samples	67.00%	Not reported	Unknown
16	Kimi K3Kimimini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±8.8 · 113 reported samples	66.40%	Not reported	Unknown
17	GPT-5.5OpenAImini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6.9 · 339 reported samples	66.10%	Not reported	Unknown
18	Gemini 3.7 FlashGooglemini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±7.1 · 339 reported samples	65.50%	Not reported	Unknown
19	Fable 5.1Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 339 reported samples	64.90%	Not reported	Unknown
20	GPT-5.6 LunaOpenAImini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.4 · 339 reported samples	64.30%	Not reported	Unknown
21	Grok 4.6xAImini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±7.4 · 339 reported samples	63.70%	Not reported	Unknown
22	Haiku 5.5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.7 · 339 reported samples	59.90%	Not reported	Unknown
23	Opus 4.8Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.1 · 339 reported samples	59.60%	Not reported	Unknown
24	DeepSeek-V4-FlashDeepSeekmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±8.8 · 113 reported samples	56.60%	Not reported	Unknown
25	DeepSeek-V4-Pro-0813DeepSeekmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 339 reported samples	56.30%	Not reported	Unknown
26	Qwen3.8-MaxAlibabamini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±8.8 · 113 reported samples	53.10%	Not reported	Unknown
27	Sonnet 5Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±7.2 · 339 reported samples	49.60%	Not reported	Unknown
28	Grok 4.5xAImini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±7.4 · 339 reported samples	49.30%	Not reported	Unknown
29	Opus 4.7Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.9 · 339 reported samples	49.30%	Not reported	Unknown
30	Gemini 3.6 FlashGooglemini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±7.1 · 339 reported samples	49.30%	Not reported	Unknown
31	GPT-5.4OpenAImini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6.8 · 339 reported samples	49.00%	Not reported	Unknown
32	GLM-5.3-FlashZhipumini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.5 · 339 reported samples	42.50%	Not reported	Unknown
33	Muse Spark 1.1Metamini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6.5 · 339 reported samples	40.40%	Not reported	Unknown
34	GLM-5.2Zhipumini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.5 · 339 reported samples	38.90%	Not reported	Unknown
35	Gemini 3.5 FlashGooglemini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±6.9 · 339 reported samples	36.60%	Not reported	Unknown
36	Sonnet 4.6Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±6.3 · 339 reported samples	34.20%	Not reported	Unknown
37	Grok 4.7xAImini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6 · 337 reported samples	33.30%	Not reported	Unknown
38	Opus 4.6Anthropicmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±6.3 · 339 reported samples	31.90%	Not reported	Unknown
39	Kimi K2.7 CodeKimimini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±8.4 · 113 reported samples	30.10%	Not reported	Unknown
40	Grok 4.6xAImini-swe-agent (500 max steps, 2 hour time limit) · xhigh reasoningPublished margin ±6.8 · 339 reported samples	24.80%	Not reported	Unknown
41	GLM-5.1Zhipumini-swe-agent (500 max steps, 2 hour time limit)Published margin ±5.5 · 339 reported samples	18.90%	Not reported	Unknown
42	MiniMax-M3MiniMaxmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±3.1 · 339 reported samples	9.10%	Not reported	Unknown
43	DeepSeek-V4-ProDeepSeekmini-swe-agent (500 max steps, 2 hour time limit) · max reasoningPublished margin ±3.8 · 337 reported samples	9.10%	Not reported	Unknown
44	Gemini 3.1 ProGooglemini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±3.1 · 339 reported samples	7.70%	Not reported	Unknown
45	InklingThinking Machinesmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±2.8 · 339 reported samples	5.00%	Not reported	Unknown
46	GLM-5 (Thinking)Zhipumini-swe-agent (500 max steps, 2 hour time limit)Published margin ±1.9 · 339 reported samples	2.10%	Not reported	Unknown
47	DeepSeek-V3.2DeepSeekmini-swe-agent (500 max steps, 2 hour time limit)Published margin ±0.9 · 339 reported samples	0.60%	Not reported	Unknown
48	Nemotron 3 UltraNVIDIAmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±0.7 · 339 reported samples	0.60%	Not reported	Unknown
49	GPT-OSS-120BOpenAImini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±0 · 339 reported samples	0.00%	Not reported	Unknown
50	Kimi K2 (Thinking)Kimimini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±0 · 339 reported samples	0.00%	Not reported	Unknown
51	MiniMax-M2.7MiniMaxmini-swe-agent (500 max steps, 2 hour time limit) · high reasoningPublished margin ±0 · 337 reported samples	0.00%	Not reported	Unknown
52	Qwen3.5Alibabamini-swe-agent (500 max steps, 2 hour time limit)Published margin ±0 · 113 reported samples	0.00%	Not reported	Unknown
53	Gemma 4 31BGooglemini-swe-agent (500 max steps, 2 hour time limit)Published margin ±0 · 339 reported samples	0.00%	Not reported	Unknown

How to read this Mercor evaluation

We captured Mercor's DeepSWE v1.1 results on October 15, 2026. The table preserves 53 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists programmatic unit tests (all must pass) and a judge of None. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

Snapshot

Pass@153 configurationsMercor evaluationDisplay only

Mercor DeepSWE v1.1 results Mercor APEX benchmark index Mercor methodology Source paper Upstream evaluation code Upstream dataset

The published DeepSWE v1.1 (Mercor) snapshot places Opus 5.5 first at 72.30%. The third row is 0.30 points behind. The broader top-10 range is 1.80 points, so many of the published results sit in a relatively narrow band.

53 configurations are shown for DeepSWE v1.1 (Mercor). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. DeepSWE v1.1 (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About DeepSWE v1.1 (Mercor)

Tasks

113 source-reported tasks

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

DeepSWE v1.1 evaluation results Public benchmark source

Freshness and provenance

Version

DeepSWE v1.1; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does DeepSWE v1.1 (Mercor) measure?

Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published DeepSWE v1.1 (Mercor) snapshot?

Opus 5.5 currently leads the published DeepSWE v1.1 (Mercor) snapshot with 72.30% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on DeepSWE v1.1 (Mercor)?

The October 15, 2026 capture snapshot contains 53 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.