SWE Atlas - Codebase QnA Extended — Mercor evaluation (SWE Atlas - Codebase QnA Extended (Mercor))

We show this table for reference; we do not rank on it.

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on SWE Atlas - Codebase QnA Extended (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for SWE Atlas - Codebase QnA Extended (Mercor). Sonnet 5.5 leads the public snapshot at 44.70%, followed by Opus 5.5 (43.30%) and Opus 5 (40.00%). We do not use these results to rank models overall.

Sonnet 5.5

Anthropic

Mercor extended-set run · max reasoning

Published margin ±12 · Original set 50.3% ±7.8 · 150 reported samples

44.70%

Opus 5.5

Anthropic

Mercor extended-set run · max reasoning

Published margin ±12.7 · Original set 48.9% ±7.9 · 150 reported samples

43.30%

Opus 5

Anthropic

Mercor extended-set run · max reasoning

Published margin ±11 · Original set 50.3% ±7.7 · 150 reported samples

40.00%

37 configurationsCodingDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture

Pass@1 table (37 configurations)

Pass@1 results for SWE Atlas - Codebase QnA Extended (Mercor)
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Sonnet 5.5AnthropicMercor extended-set run · max reasoningPublished margin ±12 · Original set 50.3% ±7.8 · 150 reported samples	44.70%	Not reported	Unknown
2	Opus 5.5AnthropicMercor extended-set run · max reasoningPublished margin ±12.7 · Original set 48.9% ±7.9 · 150 reported samples	43.30%	Not reported	Unknown
3	Opus 5AnthropicMercor extended-set run · max reasoningPublished margin ±11 · Original set 50.3% ±7.7 · 150 reported samples	40.00%	Not reported	Unknown
4	Fable 5.1AnthropicMercor extended-set run · max reasoningPublished margin ±11.7 · Original set 43.3% ±7.9 · 150 reported samples	38.00%	Not reported	Unknown
5	DeepSeek-V4.1-FlashDeepSeekMercor extended-set run · max reasoningPublished margin ±10.7 · Original set 53.5% ±7.4 · 150 reported samples	36.70%	Not reported	Unknown
6	Qwen3.8-MaxAlibabaMercor extended-set run · xhigh reasoningPublished margin ±11 · Original set 50.5% ±7.7 · 150 reported samples	33.30%	Not reported	Unknown
7	Sonnet 5AnthropicMercor extended-set run · max reasoningPublished margin ±11 · Original set 39.5% ±7.3 · 150 reported samples	32.70%	Not reported	Unknown
8	GPT-6 SolOpenAIMercor extended-set run · max reasoningPublished margin ±11 · Original set 44.1% ±7.7 · 150 reported samples	31.30%	Not reported	Unknown
9	GPT-6.1 SolOpenAIMercor extended-set run · max reasoningPublished margin ±10.3 · Original set 48.4% ±7.7 · 150 reported samples	31.30%	Not reported	Unknown
10	GPT-5.6 Sol (Pro)OpenAIMercor extended-set run · max reasoningPublished margin ±9.7 · Original set 30.9% ±6.8 · 150 reported samples	24.70%	Not reported	Unknown
11	Kimi K3KimiMercor extended-set run · max reasoningPublished margin ±9 · Original set 52.7% ±7.1 · 150 reported samples	24.00%	Not reported	Unknown
12	Opus 4.8AnthropicMercor extended-set run · max reasoningPublished margin ±10 · Original set 44.4% ±7.5 · 150 reported samples	24.00%	Not reported	Unknown
13	GPT-5.6 LunaOpenAIMercor extended-set run · max reasoningPublished margin ±8.3 · Original set 46% ±7.7 · 150 reported samples	19.30%	Not reported	Unknown
14	Grok 4.6xAIMercor extended-set run · xhigh reasoningPublished margin ±7.7 · Original set 50.3% ±7.4 · 149 reported samples	19.30%	Not reported	Unknown
15	DeepSeek-V4-FlashDeepSeekMercor extended-set run · max reasoningPublished margin ±9 · Original set 44.9% ±7.3 · 150 reported samples	18.00%	Not reported	Unknown
16	GPT-5.5OpenAIMercor extended-set run · xhigh reasoningPublished margin ±8.3 · Original set 45.2% ±7.5 · 150 reported samples	17.30%	Not reported	Unknown
17	GLM-5.2ZhipuMercor extended-set run · max reasoningPublished margin ±8 · Original set 46% ±7.5 · 150 reported samples	16.00%	Not reported	Unknown
18	GPT-5.6 TerraOpenAIMercor extended-set run · max reasoningPublished margin ±8 · Original set 45.2% ±7.5 · 150 reported samples	16.00%	Not reported	Unknown
19	GPT-6 LunaOpenAIMercor extended-set run · max reasoningPublished margin ±8 · Original set 33.9% ±7.7 · 150 reported samples	15.30%	Not reported	Unknown
20	Grok 4.7xAIMercor extended-set run · xhigh reasoningPublished margin ±7.3 · Original set 47% ±7 · 150 reported samples	15.30%	Not reported	Unknown
21	Grok 4.5xAIMercor extended-set run · high reasoningPublished margin ±6 · Original set 41.9% ±7.1 · 150 reported samples	12.00%	Not reported	Unknown
22	Opus 4.7AnthropicMercor extended-set run · max reasoningPublished margin ±6.3 · Original set 47% ±7.4 · 150 reported samples	11.30%	Not reported	Unknown
23	GPT-5.4OpenAIMercor extended-set run · xhigh reasoningPublished margin ±5.7 · Original set 37.1% ±7 · 150 reported samples	10.70%	Not reported	Unknown
24	MiniMax-M3MiniMaxMercor extended-set run · high reasoningPublished margin ±5.7 · Original set 37.9% ±7 · 150 reported samples	10.00%	Not reported	Unknown
25	Sonnet 4.6AnthropicMercor extended-set run · high reasoningPublished margin ±5.7 · Original set 35.5% ±7.3 · 150 reported samples	6.70%	Not reported	Unknown
26	DeepSeek-V4-ProDeepSeekMercor extended-set run · max reasoningPublished margin ±4.7 · Original set 25% ±6.6 · 150 reported samples	5.30%	Not reported	Unknown
27	Kimi K2.7 CodeKimiMercor extended-set run · high reasoningPublished margin ±4.7 · Original set 33.1% ±7.1 · 150 reported samples	4.70%	Not reported	Unknown
28	Gemini 3.6 FlashGoogleMercor extended-set run · high reasoningPublished margin ±4.7 · Original set 32% ±7.1 · 150 reported samples	4.70%	Not reported	Unknown
29	Gemini 3.1 ProGoogleMercor extended-set run · high reasoningPublished margin ±3.3 · Original set 18% ±6 · 150 reported samples	2.70%	Not reported	Unknown
30	Gemini 3.5 FlashGoogleMercor extended-set run · high reasoningPublished margin ±3.7 · Original set 19.6% ±5.8 · 150 reported samples	2.70%	Not reported	Unknown
31	InklingThinking MachinesMercor extended-set run · high reasoningPublished margin ±1.7 · Original set 21.2% ±6 · 150 reported samples	1.30%	Not reported	Unknown
32	MiniMax-M2.7MiniMaxMercor extended-set run · high reasoningPublished margin ±1 · Original set not reported · 150 reported samples	0.70%	Not reported	Unknown
33	Kimi K2 (Thinking)KimiMercor extended-set run · high reasoningPublished margin ±0 · Original set not reported · 150 reported samples	0.00%	Not reported	Unknown
34	Fable 5AnthropicMercor extended-set run · max reasoningPublished margin ±0 · Original set 33.1% ±7.3 · 150 reported samples	0.00%	Not reported	Unknown
35	DeepSeek-V3.2DeepSeekMercor extended-set runPublished margin ±0 · Original set 5.9% ±3.4 · 150 reported samples	0.00%	Not reported	Unknown
36	Gemma 4 31BGoogleMercor extended-set runPublished margin ±0 · Original set not reported · 150 reported samples	0.00%	Not reported	Unknown
37	GPT-OSS-120BOpenAIMercor extended-set run · high reasoningPublished margin ±0 · Original set not reported · 150 reported samples	0.00%	Not reported	Unknown

How to read this Mercor evaluation

We captured Mercor's SWE Atlas - Codebase QnA Extended results on October 15, 2026. The table preserves 37 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

The headline uses the 50 Mercor-built tasks. The paired original-set values refer to the separately listed 124-task public benchmark. Missing baselines remain unreported. We keep the extension and original set separate from their upstream benchmark-owner and provider-run tables.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) over software quality rubrics and a judge of Claude Opus 4.5. Its run-count label is k = 3. Those notes describe the original public benchmark; extension-specific changes are not established by that reference.

Snapshot

Pass@137 configurationsMercor task extensionDisplay only

Mercor SWE Atlas - Codebase QnA Extended results Mercor APEX benchmark index Mercor methodology Paired Mercor original-set results Source paper Upstream evaluation code Upstream dataset

The published SWE Atlas - Codebase QnA Extended (Mercor) snapshot places Sonnet 5.5 first at 44.70%. The third row is 4.70 points behind. The broader top-10 range is 20.00 points, so the table still separates the published systems.

37 configurations are shown for SWE Atlas - Codebase QnA Extended (Mercor). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWE Atlas - Codebase QnA Extended (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About SWE Atlas - Codebase QnA Extended (Mercor)

Tasks

50 Mercor tasks; 124 tasks in the separately reported original set

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

SWE Atlas - Codebase QnA Extended evaluation results Public benchmark source

Freshness and provenance

Version

SWE Atlas - Codebase QnA Extended; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does SWE Atlas - Codebase QnA Extended (Mercor) measure?

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published SWE Atlas - Codebase QnA Extended (Mercor) snapshot?

Sonnet 5.5 currently leads the published SWE Atlas - Codebase QnA Extended (Mercor) snapshot with 44.70% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on SWE Atlas - Codebase QnA Extended (Mercor)?

The October 15, 2026 capture snapshot contains 37 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.