# SWE Atlas - Codebase QnA Extended — Mercor evaluation (SWE Atlas - Codebase QnA Extended (Mercor))

> Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Canonical page: https://benchlm.ai/benchmarks/mercorsweatlasqnaextended

- Category: [Coding](/coding)
- Last updated: October 15, 2026 capture

Evaluation results published by [Mercor](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/). Published evaluation aggregates only; task contents and dataset license grants are not included.

We captured Mercor's SWE Atlas - Codebase QnA Extended results on October 15, 2026. The table preserves 37 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

The headline uses the 50 Mercor-built tasks. The paired original-set values refer to the separately listed 124-task public benchmark. Missing baselines remain unreported. We keep the extension and original set separate from their upstream benchmark-owner and provider-run tables.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) over software quality rubrics and a judge of Claude Opus 4.5. Its run-count label is k = 3. Those notes describe the original public benchmark; extension-specific changes are not established by that reference.

- [Mercor SWE Atlas - Codebase QnA Extended results](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/)
- [Mercor APEX benchmark index](https://www.mercor.com/apex/)
- [Mercor methodology](https://www.mercor.com/apex/methodology/)
- [Paired Mercor original-set results](https://www.mercor.com/apex/oss-benchmarks/oss-swe-atlas-leaderboard/)
- [Source paper](https://arxiv.org/abs/2605.08366)
- [Upstream evaluation code](https://github.com/scaleapi/SWE-Atlas)
- [Upstream dataset](https://github.com/scaleapi/SWE-Atlas)

## About SWE Atlas - Codebase QnA Extended (Mercor)

- Tasks: 50 Mercor tasks; 124 tasks in the separately reported original set
- Format: Pass@1
- Difficulty: Source-specific evaluation
- Paper: [SWE Atlas - Codebase QnA Extended evaluation results](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/)

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

SWE Atlas - Codebase QnA Extended (Mercor) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (37 configurations)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Sonnet 5.5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±12 · Original set 50.3% ±7.8 · 150 reported samples | Anthropic | 44.70% |
| 2 | [Opus 5.5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±12.7 · Original set 48.9% ±7.9 · 150 reported samples | Anthropic | 43.30% |
| 3 | [Opus 5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±11 · Original set 50.3% ±7.7 · 150 reported samples | Anthropic | 40.00% |
| 4 | [Fable 5.1](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±11.7 · Original set 43.3% ±7.9 · 150 reported samples | Anthropic | 38.00% |
| 5 | [DeepSeek-V4.1-Flash](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±10.7 · Original set 53.5% ±7.4 · 150 reported samples | DeepSeek | 36.70% |
| 6 | [Qwen3.8-Max](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · xhigh reasoning · Published margin ±11 · Original set 50.5% ±7.7 · 150 reported samples | Alibaba | 33.30% |
| 7 | [Sonnet 5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±11 · Original set 39.5% ±7.3 · 150 reported samples | Anthropic | 32.70% |
| 8 | [GPT-6 Sol](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±11 · Original set 44.1% ±7.7 · 150 reported samples | OpenAI | 31.30% |
| 9 | [GPT-6.1 Sol](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±10.3 · Original set 48.4% ±7.7 · 150 reported samples | OpenAI | 31.30% |
| 10 | [GPT-5.6 Sol (Pro)](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±9.7 · Original set 30.9% ±6.8 · 150 reported samples | OpenAI | 24.70% |
| 11 | [Kimi K3](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±9 · Original set 52.7% ±7.1 · 150 reported samples | Kimi | 24.00% |
| 12 | [Opus 4.8](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±10 · Original set 44.4% ±7.5 · 150 reported samples | Anthropic | 24.00% |
| 13 | [GPT-5.6 Luna](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±8.3 · Original set 46% ±7.7 · 150 reported samples | OpenAI | 19.30% |
| 14 | [Grok 4.6](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · xhigh reasoning · Published margin ±7.7 · Original set 50.3% ±7.4 · 149 reported samples | xAI | 19.30% |
| 15 | [DeepSeek-V4-Flash](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±9 · Original set 44.9% ±7.3 · 150 reported samples | DeepSeek | 18.00% |
| 16 | [GPT-5.5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · xhigh reasoning · Published margin ±8.3 · Original set 45.2% ±7.5 · 150 reported samples | OpenAI | 17.30% |
| 17 | [GLM-5.2](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±8 · Original set 46% ±7.5 · 150 reported samples | Zhipu | 16.00% |
| 18 | [GPT-5.6 Terra](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±8 · Original set 45.2% ±7.5 · 150 reported samples | OpenAI | 16.00% |
| 19 | [GPT-6 Luna](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±8 · Original set 33.9% ±7.7 · 150 reported samples | OpenAI | 15.30% |
| 20 | [Grok 4.7](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · xhigh reasoning · Published margin ±7.3 · Original set 47% ±7 · 150 reported samples | xAI | 15.30% |
| 21 | [Grok 4.5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±6 · Original set 41.9% ±7.1 · 150 reported samples | xAI | 12.00% |
| 22 | [Opus 4.7](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±6.3 · Original set 47% ±7.4 · 150 reported samples | Anthropic | 11.30% |
| 23 | [GPT-5.4](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · xhigh reasoning · Published margin ±5.7 · Original set 37.1% ±7 · 150 reported samples | OpenAI | 10.70% |
| 24 | [MiniMax-M3](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±5.7 · Original set 37.9% ±7 · 150 reported samples | MiniMax | 10.00% |
| 25 | [Sonnet 4.6](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±5.7 · Original set 35.5% ±7.3 · 150 reported samples | Anthropic | 6.70% |
| 26 | [DeepSeek-V4-Pro](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±4.7 · Original set 25% ±6.6 · 150 reported samples | DeepSeek | 5.30% |
| 27 | [Kimi K2.7 Code](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±4.7 · Original set 33.1% ±7.1 · 150 reported samples | Kimi | 4.70% |
| 28 | [Gemini 3.6 Flash](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±4.7 · Original set 32% ±7.1 · 150 reported samples | Google | 4.70% |
| 29 | [Gemini 3.1 Pro](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±3.3 · Original set 18% ±6 · 150 reported samples | Google | 2.70% |
| 30 | [Gemini 3.5 Flash](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±3.7 · Original set 19.6% ±5.8 · 150 reported samples | Google | 2.70% |
| 31 | [Inkling](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±1.7 · Original set 21.2% ±6 · 150 reported samples | Thinking Machines | 1.30% |
| 32 | [MiniMax-M2.7](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±1 · Original set not reported · 150 reported samples | MiniMax | 0.70% |
| 33 | [Kimi K2 (Thinking)](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±0 · Original set not reported · 150 reported samples | Kimi | 0.00% |
| 34 | [Fable 5](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · max reasoning · Published margin ±0 · Original set 33.1% ±7.3 · 150 reported samples | Anthropic | 0.00% |
| 35 | [DeepSeek-V3.2](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · Published margin ±0 · Original set 5.9% ±3.4 · 150 reported samples | DeepSeek | 0.00% |
| 36 | [Gemma 4 31B](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · Published margin ±0 · Original set not reported · 150 reported samples | Google | 0.00% |
| 37 | [GPT-OSS-120B](https://www.mercor.com/apex/mercor-extended/ots-held-out-swe-atlas-leaderboard/) | Mercor extended-set run · high reasoning · Published margin ±0 · Original set not reported · 150 reported samples | OpenAI | 0.00% |

## FAQ

### What does SWE Atlas - Codebase QnA Extended (Mercor) measure?

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

### Which model leads the published SWE Atlas - Codebase QnA Extended (Mercor) snapshot?

Sonnet 5.5 currently leads the published SWE Atlas - Codebase QnA Extended (Mercor) snapshot with a score of 44.70%.

### How many models are evaluated on SWE Atlas - Codebase QnA Extended (Mercor)?

The October 15, 2026 capture contains 37 source configurations.

### Does SWE Atlas - Codebase QnA Extended (Mercor) affect BenchLM's overall score?

Not directly. SWE Atlas - Codebase QnA Extended (Mercor) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.