Skip to main content
BenchLM
Data

CharXiv Extended — Mercor evaluation (CharXiv Extended (Mercor))

We show this table for reference; we do not rank on it.

Answering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on CharXiv Extended (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for CharXiv Extended (Mercor). Muse Spark 1.2 leads the public snapshot at 87.10%, followed by Muse Spark 1.1 (86.40%) and GPT-6 Astra (85.10%). We do not use these results to rank models overall.

1

Muse Spark 1.2

Meta

Mercor extended-set run · xhigh reasoning

Published margin ±4.6 · Original set 95.3% ±0.5 · 450 reported samples

87.10%
2

Muse Spark 1.1

Meta

Mercor extended-set run · xhigh reasoning

Published margin ±5 · Original set 95.2% ±0.5 · 450 reported samples

86.40%
3

GPT-6 Astra

OpenAI

Mercor extended-set run · xhigh reasoning

Published margin ±5.4 · Original set 95.8% ±0.5 · 450 reported samples

85.10%

39 configurationsMultimodal & GroundedDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture

Pass@1 table (39 configurations)

Pass@1 results for CharXiv Extended (Mercor)
RankModel / configurationScoreParameters (B)Open / closed
1Muse Spark 1.2MetaMercor extended-set run · xhigh reasoningPublished margin ±4.6 · Original set 95.3% ±0.5 · 450 reported samples
87.10%
Not reportedUnknown
2Muse Spark 1.1MetaMercor extended-set run · xhigh reasoningPublished margin ±5 · Original set 95.2% ±0.5 · 450 reported samples
86.40%
Not reportedUnknown
3GPT-6 AstraOpenAIMercor extended-set run · xhigh reasoningPublished margin ±5.4 · Original set 95.8% ±0.5 · 450 reported samples
85.10%
Not reportedUnknown
4GPT-6.1 SolOpenAIMercor extended-set run · max reasoningPublished margin ±5.4 · Original set 96% ±0.5 · 450 reported samples
85.10%
Not reportedUnknown
5Opus 5.5AnthropicMercor extended-set run · max reasoningPublished margin ±5.4 · Original set 96.4% ±0.5 · 450 reported samples
84.70%
Not reportedUnknown
6Sonnet 5.5AnthropicMercor extended-set run · max reasoningPublished margin ±5.7 · Original set 96.1% ±0.5 · 450 reported samples
84.20%
Not reportedUnknown
7GPT-6 AstraOpenAIMercor extended-set run · max reasoningPublished margin ±5.7 · Original set 96.1% ±0.5 · 450 reported samples
84.00%
Not reportedUnknown
8GPT-6 SolOpenAIMercor extended-set run · max reasoningPublished margin ±5.6 · Original set 95% ±0.6 · 450 reported samples
83.80%
Not reportedUnknown
9Qwen3.8-MaxAlibabaMercor extended-set run · xhigh reasoningPublished margin ±5.9 · Original set 94.1% ±0.6 · 450 reported samples
82.90%
Not reportedUnknown
10GPT-5.6 Sol (Pro)OpenAIMercor extended-set run · max reasoningPublished margin ±5.8 · Original set 92.1% ±0.6 · 450 reported samples
82.90%
Not reportedUnknown
11Haiku 5.5AnthropicMercor extended-set run · max reasoningPublished margin ±5.8 · Original set 94.1% ±0.6 · 450 reported samples
82.90%
Not reportedUnknown
12GPT-5.4OpenAIMercor extended-set run · xhigh reasoningPublished margin ±5.6 · Original set 93.4% ±0.6 · 450 reported samples
82.70%
Not reportedUnknown
13GPT-5.5OpenAIMercor extended-set run · xhigh reasoningPublished margin ±5.6 · Original set 94.6% ±0.6 · 450 reported samples
82.70%
Not reportedUnknown
14Fable 5.1AnthropicMercor extended-set run · max reasoningPublished margin ±5.9 · Original set 95.1% ±0.6 · 450 reported samples
82.20%
Not reportedUnknown
15Fable 5.1AnthropicMercor extended-set run · high reasoningPublished margin ±5.9 · Original set 94.1% ±0.6 · 450 reported samples
82.00%
Not reportedUnknown
16Muse Spark 1.3MetaMercor extended-set run · max reasoningPublished margin ±5.9 · Original set 93.9% ±0.6 · 450 reported samples
81.80%
Not reportedUnknown
17Gemini 3.5 FlashGoogleMercor extended-set run · high reasoningPublished margin ±5.7 · Original set 94.3% ±0.6 · 450 reported samples
81.30%
Not reportedUnknown
18GPT-5.6 TerraOpenAIMercor extended-set run · max reasoningPublished margin ±5.8 · Original set 94.1% ±0.6 · 450 reported samples
81.10%
Not reportedUnknown
19Fable 5AnthropicMercor extended-set run · max reasoningPublished margin ±6.1 · Original set 89.1% ±0.8 · 449 reported samples
80.70%
Not reportedUnknown
20Opus 5AnthropicMercor extended-set run · max reasoningPublished margin ±5.9 · Original set 94.6% ±0.6 · 450 reported samples
79.80%
Not reportedUnknown
21Gemini 3.7 FlashGoogleMercor extended-set run · high reasoningPublished margin ±6.1 · Original set 95.3% ±0.5 · 450 reported samples
79.10%
Not reportedUnknown
22Gemini 3.8 FlashGoogleMercor extended-set run · high reasoningPublished margin ±6.1 · Original set 95.4% ±0.5 · 450 reported samples
78.90%
Not reportedUnknown
23GPT-5.6 LunaOpenAIMercor extended-set run · max reasoningPublished margin ±6.1 · Original set 92.9% ±0.6 · 450 reported samples
78.70%
Not reportedUnknown
24Kimi K3KimiMercor extended-set run · max reasoningPublished margin ±5.9 · Original set 93.9% ±0.6 · 450 reported samples
78.70%
Not reportedUnknown
25Gemini 3.6 FlashGoogleMercor extended-set run · high reasoningPublished margin ±6 · Original set 95.1% ±0.6 · 450 reported samples
78.40%
Not reportedUnknown
26Grok 4.7xAIMercor extended-set run · xhigh reasoningPublished margin ±6 · Original set 92% ±0.7 · 447 reported samples
78.20%
Not reportedUnknown
27Grok 4.6xAIMercor extended-set run · xhigh reasoningPublished margin ±6.2 · Original set 92.9% ±0.6 · 450 reported samples
77.30%
Not reportedUnknown
28Gemini 3.1 ProGoogleMercor extended-set run · high reasoningPublished margin ±6.1 · Original set 94.6% ±0.6 · 450 reported samples
76.90%
Not reportedUnknown
29Opus 4.8AnthropicMercor extended-set run · max reasoningPublished margin ±6.4 · Original set 93.2% ±0.6 · 450 reported samples
76.70%
Not reportedUnknown
30GPT-6 LunaOpenAIMercor extended-set run · max reasoningPublished margin ±6.1 · Original set 90.9% ±0.7 · 450 reported samples
76.40%
Not reportedUnknown
31Qwen3.5AlibabaMercor extended-set runPublished margin ±6.1 · Original set 91.6% ±0.7 · 450 reported samples
76.20%
Not reportedUnknown
32GLM-5.3-FlashZhipuMercor extended-set run · max reasoningPublished margin ±6 · Original set 92.5% ±0.7 · 450 reported samples
76.20%
Not reportedUnknown
33Opus 4.7AnthropicMercor extended-set run · max reasoningPublished margin ±6.4 · Original set 92.6% ±0.7 · 450 reported samples
76.00%
Not reportedUnknown
34Grok 4.5xAIMercor extended-set run · high reasoningPublished margin ±6.6 · Original set 92.8% ±0.7 · 450 reported samples
72.70%
Not reportedUnknown
35Sonnet 5AnthropicMercor extended-set run · max reasoningPublished margin ±6.6 · Original set 88.8% ±0.7 · 450 reported samples
71.10%
Not reportedUnknown
36InklingThinking MachinesMercor extended-set run · high reasoningPublished margin ±6.8 · Original set 88.9% ±0.8 · 450 reported samples
69.80%
Not reportedUnknown
37DeepSeek-V4.1-FlashDeepSeekMercor extended-set run · max reasoningPublished margin ±6.4 · Original set 90.1% ±0.7 · 450 reported samples
68.90%
Not reportedUnknown
38Sonnet 4.6AnthropicMercor extended-set run · high reasoningPublished margin ±7.2 · Original set 90.5% ±0.8 · 450 reported samples
64.40%
Not reportedUnknown
39MiniMax-M3MiniMaxMercor extended-set run · high reasoningPublished margin ±6.7 · Original set 85.9% ±0.8 · 450 reported samples
58.40%
Not reportedUnknown

How to read this Mercor evaluation

We captured Mercor's CharXiv Extended results on October 15, 2026. The table preserves 39 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

The headline uses the 150 Mercor-built tasks. The paired original-set values refer to the separately listed 5000-task public benchmark. Missing baselines remain unreported. We keep the extension and original set separate from their upstream benchmark-owner and provider-run tables.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) and a judge of Gemini 3.5 Flash. Its run-count label is k = 3. Those notes describe the original public benchmark; extension-specific changes are not established by that reference.

Snapshot

Pass@139 configurationsMercor task extensionDisplay only
Mercor CharXiv Extended resultsMercor APEX benchmark indexMercor methodologyPaired Mercor original-set resultsSource paperUpstream evaluation codeUpstream dataset

The published CharXiv Extended (Mercor) snapshot places Muse Spark 1.2 first at 87.10%. The third row is 2.00 points behind. The broader top-10 range is 4.20 points, so many of the published results sit in a relatively narrow band.

39 configurations are shown for CharXiv Extended (Mercor). The benchmark falls in the Multimodal & Grounded category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. CharXiv Extended (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About CharXiv Extended (Mercor)

Tasks

150 Mercor tasks; 5000 tasks in the separately reported original set

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

CharXiv Extended evaluation resultsPublic benchmark source

Freshness and provenance

Version

CharXiv Extended; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does CharXiv Extended (Mercor) measure?

Answering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published CharXiv Extended (Mercor) snapshot?

Muse Spark 1.2 currently leads the published CharXiv Extended (Mercor) snapshot with 87.10% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on CharXiv Extended (Mercor)?

The October 15, 2026 capture snapshot contains 39 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.