SWE Atlas - Codebase QnA — Mercor evaluation (SWE Atlas - Codebase QnA (Mercor))

We show this table for reference; we do not rank on it.

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on SWE Atlas - Codebase QnA (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for SWE Atlas - Codebase QnA (Mercor). DeepSeek-V4.1-Flash leads the public snapshot at 53.50%, followed by Kimi K3 (52.70%) and Qwen3.8-Max (50.50%). We do not use these results to rank models overall.

DeepSeek-V4.1-Flash

DeepSeek

mini-swe-agent (250 max steps, 1 hour time limit) · max reasoning

Published margin ±7.4 · 372 reported samples

53.50%

Kimi K3

Kimi

mini-swe-agent (250 max steps, 1 hour time limit) · max reasoning

Published margin ±7.1 · 372 reported samples

52.70%

Qwen3.8-Max

Alibaba

mini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoning

Published margin ±7.7 · 372 reported samples

50.50%

41 configurationsCodingDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture

Pass@1 table (41 configurations)

Pass@1 results for SWE Atlas - Codebase QnA (Mercor)
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	DeepSeek-V4.1-FlashDeepSeekmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.4 · 372 reported samples	53.50%	Not reported	Unknown
2	Kimi K3Kimimini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.1 · 372 reported samples	52.70%	Not reported	Unknown
3	Qwen3.8-MaxAlibabamini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7.7 · 372 reported samples	50.50%	Not reported	Unknown
4	Opus 5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.7 · 372 reported samples	50.30%	Not reported	Unknown
5	Sonnet 5.5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.8 · 372 reported samples	50.30%	Not reported	Unknown
6	Grok 4.6xAImini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7.4 · 372 reported samples	50.30%	Not reported	Unknown
7	Opus 5.5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.9 · 372 reported samples	48.90%	Not reported	Unknown
8	GPT-6 AstraOpenAImini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7.9 · 372 reported samples	48.90%	Not reported	Unknown
9	Haiku 5.5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.4 · 372 reported samples	48.70%	Not reported	Unknown
10	GPT-6.1 SolOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.7 · 372 reported samples	48.40%	Not reported	Unknown
11	GPT-6 AstraOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.5 · 372 reported samples	47.80%	Not reported	Unknown
12	Opus 4.7Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.4 · 372 reported samples	47.00%	Not reported	Unknown
13	Grok 4.7xAImini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7 · 372 reported samples	47.00%	Not reported	Unknown
14	GLM-5.2Zhipumini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.5 · 372 reported samples	46.00%	Not reported	Unknown
15	GPT-5.6 LunaOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.7 · 372 reported samples	46.00%	Not reported	Unknown
16	GPT-5.5OpenAImini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7.5 · 372 reported samples	45.20%	Not reported	Unknown
17	GPT-5.6 TerraOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.5 · 372 reported samples	45.20%	Not reported	Unknown
18	DeepSeek-V4-FlashDeepSeekmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.3 · 372 reported samples	44.90%	Not reported	Unknown
19	Opus 4.8Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.5 · 372 reported samples	44.40%	Not reported	Unknown
20	GPT-6 SolOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.7 · 372 reported samples	44.10%	Not reported	Unknown
21	Fable 5.1Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.9 · 372 reported samples	43.30%	Not reported	Unknown
22	Fable 5.1Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.9 · 372 reported samples	42.70%	Not reported	Unknown
23	MiMo V2.6 ProXiaomimini-swe-agent (250 max steps, 1 hour time limit) · auto reasoningPublished margin ±7.3 · 372 reported samples	42.70%	Not reported	Unknown
24	GLM-5.3Zhipumini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7 · 371 reported samples	42.50%	Not reported	Unknown
25	Grok 4.5xAImini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.1 · 372 reported samples	41.90%	Not reported	Unknown
26	Gemini 3.8 FlashGooglemini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.5 · 372 reported samples	40.60%	Not reported	Unknown
27	Sonnet 5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.3 · 372 reported samples	39.50%	Not reported	Unknown
28	MiniMax-M3MiniMaxmini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7 · 372 reported samples	37.90%	Not reported	Unknown
29	GPT-5.4OpenAImini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±7 · 372 reported samples	37.10%	Not reported	Unknown
30	Sonnet 4.6Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.3 · 372 reported samples	35.50%	Not reported	Unknown
31	GPT-6 LunaOpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.7 · 372 reported samples	33.90%	Not reported	Unknown
32	Kimi K2.7 CodeKimimini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.1 · 372 reported samples	33.10%	Not reported	Unknown
33	Fable 5Anthropicmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±7.3 · 372 reported samples	33.10%	Not reported	Unknown
34	Gemini 3.6 FlashGooglemini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±7.1 · 372 reported samples	32.00%	Not reported	Unknown
35	GPT-5.6 Sol (Pro)OpenAImini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±6.8 · 365 reported samples	30.90%	Not reported	Unknown
36	DeepSeek-V4-ProDeepSeekmini-swe-agent (250 max steps, 1 hour time limit) · max reasoningPublished margin ±6.6 · 372 reported samples	25.00%	Not reported	Unknown
37	InklingThinking Machinesmini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±6 · 372 reported samples	21.20%	Not reported	Unknown
38	Gemini 3.5 FlashGooglemini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±5.8 · 372 reported samples	19.60%	Not reported	Unknown
39	Gemini 3.1 ProGooglemini-swe-agent (250 max steps, 1 hour time limit) · high reasoningPublished margin ±6 · 372 reported samples	18.00%	Not reported	Unknown
40	DeepSeek-V3.2DeepSeekmini-swe-agent (250 max steps, 1 hour time limit)Published margin ±3.4 · 372 reported samples	5.90%	Not reported	Unknown
41	Muse Spark 1.1Metamini-swe-agent (250 max steps, 1 hour time limit) · xhigh reasoningPublished margin ±2.6 · 370 reported samples	3.90%	Not reported	Unknown

How to read this Mercor evaluation

We captured Mercor's SWE Atlas - Codebase QnA results on October 15, 2026. The table preserves 41 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) over software quality rubrics and a judge of Claude Opus 4.5. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

Snapshot

Pass@141 configurationsMercor evaluationDisplay only

Mercor SWE Atlas - Codebase QnA results Mercor APEX benchmark index Mercor methodology Source paper Upstream evaluation code Upstream dataset

The published SWE Atlas - Codebase QnA (Mercor) snapshot places DeepSeek-V4.1-Flash first at 53.50%. The third row is 3.00 points behind. The broader top-10 range is 5.10 points, so many of the published results sit in a relatively narrow band.

41 configurations are shown for SWE Atlas - Codebase QnA (Mercor). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWE Atlas - Codebase QnA (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About SWE Atlas - Codebase QnA (Mercor)

Tasks

124 source-reported tasks

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

SWE Atlas - Codebase QnA evaluation results Public benchmark source

Freshness and provenance

Version

SWE Atlas - Codebase QnA; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does SWE Atlas - Codebase QnA (Mercor) measure?

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published SWE Atlas - Codebase QnA (Mercor) snapshot?

DeepSeek-V4.1-Flash currently leads the published SWE Atlas - Codebase QnA (Mercor) snapshot with 53.50% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on SWE Atlas - Codebase QnA (Mercor)?

The October 15, 2026 capture snapshot contains 41 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.