HLE — Mercor evaluation (HLE (Mercor))

We show this table for reference; we do not rank on it.

Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on HLE (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for HLE (Mercor). Muse Spark 1.1 leads the public snapshot at 57.90%, followed by Opus 5.5 (57.40%) and GPT-6 Astra (55.60%). We do not use these results to rank models overall.

Muse Spark 1.1

Pass@1 table (55 configurations)

Pass@1 results for HLE (Mercor)
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Muse Spark 1.1MetaSingle-shot multimodal · xhigh reasoningPublished margin ±1.9 · 6,241 reported samples	57.90%	Not reported	Unknown
2	Opus 5.5AnthropicSingle-shot multimodal · max reasoningPublished margin ±2 · 6,474 reported samples	57.40%	Not reported	Unknown
3	GPT-6 AstraOpenAISingle-shot multimodal · xhigh reasoningPublished margin ±2 · 6,397 reported samples	55.60%	Not reported	Unknown
4	Fable 5.1AnthropicSingle-shot multimodal · max reasoningPublished margin ±2 · 6,474 reported samples	54.60%	Not reported	Unknown
5	GPT-6.1 SolOpenAISingle-shot multimodal · max reasoningPublished margin ±2.1 · 6,267 reported samples	54.50%	Not reported	Unknown
6	Opus 5AnthropicSingle-shot multimodal · max reasoningPublished margin ±2 · 6,422 reported samples	53.20%	Not reported	Unknown
7	Fable 5.1AnthropicSingle-shot multimodal · high reasoningPublished margin ±2 · 6,474 reported samples	53.10%	Not reported	Unknown
8	GPT-6 AstraOpenAISingle-shot multimodal · max reasoningPublished margin ±2 · 6,474 reported samples	52.20%	Not reported	Unknown
9	Opus 5AnthropicSingle-shot multimodal · high reasoningPublished margin ±2 · 6,474 reported samples	52.00%	Not reported	Unknown
10	Fable 5AnthropicSingle-shot multimodal · high reasoningPublished margin ±2 · 6,474 reported samples	51.50%	Not reported	Unknown
11	GPT-6 SolOpenAISingle-shot multimodal · max reasoningPublished margin ±2 · 6,471 reported samples	49.20%	Not reported	Unknown
12	GPT-5.6 SolOpenAISingle-shot multimodal · max reasoningPublished margin ±2 · 6,474 reported samples	47.00%	Not reported	Unknown
13	Gemini 3.8 FlashGoogleSingle-shot multimodal · high reasoningPublished margin ±2 · 6,474 reported samples	47.00%	Not reported	Unknown
14	Kimi K3KimiSingle-shot multimodal · max reasoningPublished margin ±2 · 6,294 reported samples	46.50%	Not reported	Unknown
15	Fable 5AnthropicSingle-shot multimodal · max reasoningPublished margin ±2 · 6,144 reported samples	46.50%	Not reported	Unknown
16	GPT-5.5OpenAISingle-shot multimodal · xhigh reasoningPublished margin ±2 · 6,442 reported samples	46.50%	Not reported	Unknown
17	Gemini 3.7 FlashGoogleSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,474 reported samples	46.00%	Not reported	Unknown
18	GPT-5.6 TerraOpenAISingle-shot multimodal · max reasoningPublished margin ±2 · 6,585 reported samples	44.90%	Not reported	Unknown
19	Opus 4.8AnthropicSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,474 reported samples	44.80%	Not reported	Unknown
20	Gemini 3.1 ProGoogleSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,474 reported samples	44.10%	Not reported	Unknown
21	GPT-5.4OpenAISingle-shot multimodal · xhigh reasoningPublished margin ±2 · 6,474 reported samples	44.10%	Not reported	Unknown
22	Qwen3.8-MaxAlibabaSingle-shot multimodal · xhigh reasoningPublished margin ±1.9 · 6,474 reported samples	42.20%	Not reported	Unknown
23	Opus 4.7AnthropicSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,474 reported samples	42.20%	Not reported	Unknown
24	Grok 4.5xAISingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,099 reported samples	41.90%	Not reported	Unknown
25	DeepSeek-V4-Pro-0813DeepSeekSingle-shot multimodal · max reasoningPublished margin ±1.9 · 6,306 reported samples	41.90%	Not reported	Unknown
26	Grok 4.6xAISingle-shot multimodal · xhigh reasoningPublished margin ±1.9 · 6,474 reported samples	41.30%	Not reported	Unknown
27	Gemini 3.6 FlashGoogleSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,471 reported samples	41.20%	Not reported	Unknown
28	Gemini 3.5 FlashGoogleSingle-shot multimodal · high reasoningPublished margin ±1.9 · 6,474 reported samples	40.60%	Not reported	Unknown
29	DeepSeek-V4.1-FlashDeepSeekSingle-shot multimodal · max reasoningPublished margin ±1.9 · 6,474 reported samples	40.50%	Not reported	Unknown
30	GLM-5.3ZhipuSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,474 reported samples	40.00%	Not reported	Unknown
31	GPT-5.6 LunaOpenAISingle-shot multimodal · max reasoningPublished margin ±1.9 · 6,528 reported samples	39.60%	Not reported	Unknown
32	DeepSeek-V4-FlashDeepSeekSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,428 reported samples	39.00%	Not reported	Unknown
33	GPT-6 LunaOpenAISingle-shot multimodal · max reasoningPublished margin ±1.9 · 6,474 reported samples	38.70%	Not reported	Unknown
34	MiniMax-M3MiniMaxSingle-shot multimodal · high reasoningPublished margin ±1.8 · 6,159 reported samples	37.80%	Not reported	Unknown
35	Opus 4.8AnthropicSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,473 reported samples	37.10%	Not reported	Unknown
36	Haiku 5.5AnthropicSingle-shot multimodal · max reasoningPublished margin ±1.9 · 6,474 reported samples	36.80%	Not reported	Unknown
37	Sonnet 5AnthropicSingle-shot multimodal · high reasoningPublished margin ±1.8 · 6,474 reported samples	36.30%	Not reported	Unknown
38	Opus 4.7AnthropicSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,474 reported samples	35.40%	Not reported	Unknown
39	Opus 4.6AnthropicSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,474 reported samples	35.00%	Not reported	Unknown
40	DeepSeek-V4-ProDeepSeekSingle-shot multimodal · max reasoningPublished margin ±1.8 · 5,267 reported samples	34.40%	Not reported	Unknown
41	Kimi K2.7 CodeKimiSingle-shot multimodal · high reasoningPublished margin ±1.8 · 6,378 reported samples	34.00%	Not reported	Unknown
42	GLM-5.3-FlashZhipuSingle-shot multimodal · max reasoningPublished margin ±1.8 · 6,474 reported samples	33.60%	Not reported	Unknown
43	GLM-5.2ZhipuSingle-shot multimodal · max reasoningPublished margin ±1.6 · 6,474 reported samples	31.70%	Not reported	Unknown
44	GLM-5.1ZhipuSingle-shot multimodalPublished margin ±1.6 · 6,474 reported samples	29.40%	Not reported	Unknown
45	Sonnet 4.6AnthropicSingle-shot multimodal · medium reasoningPublished margin ±1.7 · 6,474 reported samples	29.20%	Not reported	Unknown
46	Qwen3.5AlibabaSingle-shot multimodalPublished margin ±1.7 · 6,474 reported samples	29.20%	Not reported	Unknown
47	Nemotron 3 UltraNVIDIASingle-shot multimodal · high reasoningPublished margin ±1.9 · 4,475 reported samples	28.10%	Not reported	Unknown
48	Sonnet 5AnthropicSingle-shot multimodal · max reasoningPublished margin ±1.7 · 6,474 reported samples	27.30%	Not reported	Unknown
49	MiniMax-M2.7MiniMaxSingle-shot multimodal · high reasoningPublished margin ±1.6 · 6,474 reported samples	27.10%	Not reported	Unknown
50	InklingThinking MachinesSingle-shot multimodal · high reasoningPublished margin ±1.6 · 6,473 reported samples	25.50%	Not reported	Unknown
51	Sonnet 4.6AnthropicSingle-shot multimodal · high reasoningPublished margin ±1.6 · 6,453 reported samples	24.40%	Not reported	Unknown
52	Gemma 4 31BGoogleSingle-shot multimodalPublished margin ±1.6 · 6,474 reported samples	23.20%	Not reported	Unknown
53	DeepSeek-V3.2DeepSeekSingle-shot multimodalPublished margin ±1.5 · 6,474 reported samples	22.30%	Not reported	Unknown
54	GPT-OSS-120BOpenAISingle-shot multimodal · high reasoningPublished margin ±1.4 · 6,363 reported samples	18.50%	Not reported	Unknown
55	Kimi K2 (Thinking)KimiSingle-shot multimodal · high reasoningPublished margin ±1.2 · 6,306 reported samples	12.70%	Not reported	Unknown

How to read this Mercor evaluation

We captured Mercor's HLE results on October 15, 2026. The table preserves 55 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) and a judge of Gemini 3.5 Flash. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

Snapshot

Pass@155 configurationsMercor evaluationDisplay only

Mercor HLE results Mercor APEX benchmark index Mercor methodology Source paper Upstream evaluation code Upstream dataset

The published HLE (Mercor) snapshot places Muse Spark 1.1 first at 57.90%. The third row is 2.30 points behind. The broader top-10 range is 6.40 points, so many of the published results sit in a relatively narrow band.

55 configurations are shown for HLE (Mercor). The benchmark falls in the Knowledge category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. HLE (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About HLE (Mercor)

Tasks

2,158 source-reported tasks

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

HLE evaluation results Public benchmark source

Freshness and provenance

Version

HLE; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does HLE (Mercor) measure?

Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published HLE (Mercor) snapshot?

Muse Spark 1.1 currently leads the published HLE (Mercor) snapshot with 57.90% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on HLE (Mercor)?

The October 15, 2026 capture snapshot contains 55 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.