# HLE — Mercor evaluation (HLE (Mercor))

> Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Canonical page: https://benchlm.ai/benchmarks/mercorhle

- Category: [Knowledge](/knowledge)
- Last updated: October 15, 2026 capture

Evaluation results published by [Mercor](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/). Published evaluation aggregates only; task contents and dataset license grants are not included.

We captured Mercor's HLE results on October 15, 2026. The table preserves 55 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists llm-as-a-judge (lmaaj) and a judge of Gemini 3.5 Flash. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

- [Mercor HLE results](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/)
- [Mercor APEX benchmark index](https://www.mercor.com/apex/)
- [Mercor methodology](https://www.mercor.com/apex/methodology/)
- [Source paper](https://arxiv.org/abs/2501.14249)
- [Upstream evaluation code](https://github.com/centerforaisafety/hle)
- [Upstream dataset](https://huggingface.co/datasets/cais/hle)

## About HLE (Mercor)

- Tasks: 2,158 source-reported tasks
- Format: Pass@1
- Difficulty: Source-specific evaluation
- Paper: [HLE evaluation results](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/)

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

HLE (Mercor) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (55 configurations)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Muse Spark 1.1](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±1.9 · 6,241 reported samples | Meta | 57.90% |
| 2 | [Opus 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,474 reported samples | Anthropic | 57.40% |
| 3 | [GPT-6 Astra](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±2 · 6,397 reported samples | OpenAI | 55.60% |
| 4 | [Fable 5.1](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,474 reported samples | Anthropic | 54.60% |
| 5 | [GPT-6.1 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2.1 · 6,267 reported samples | OpenAI | 54.50% |
| 6 | [Opus 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,422 reported samples | Anthropic | 53.20% |
| 7 | [Fable 5.1](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±2 · 6,474 reported samples | Anthropic | 53.10% |
| 8 | [GPT-6 Astra](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,474 reported samples | OpenAI | 52.20% |
| 9 | [Opus 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±2 · 6,474 reported samples | Anthropic | 52.00% |
| 10 | [Fable 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±2 · 6,474 reported samples | Anthropic | 51.50% |
| 11 | [GPT-6 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,471 reported samples | OpenAI | 49.20% |
| 12 | [GPT-5.6 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,474 reported samples | OpenAI | 47.00% |
| 13 | [Gemini 3.8 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±2 · 6,474 reported samples | Google | 47.00% |
| 14 | [Kimi K3](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,294 reported samples | Kimi | 46.50% |
| 15 | [Fable 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,144 reported samples | Anthropic | 46.50% |
| 16 | [GPT-5.5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±2 · 6,442 reported samples | OpenAI | 46.50% |
| 17 | [Gemini 3.7 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,474 reported samples | Google | 46.00% |
| 18 | [GPT-5.6 Terra](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±2 · 6,585 reported samples | OpenAI | 44.90% |
| 19 | [Opus 4.8](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,474 reported samples | Anthropic | 44.80% |
| 20 | [Gemini 3.1 Pro](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,474 reported samples | Google | 44.10% |
| 21 | [GPT-5.4](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±2 · 6,474 reported samples | OpenAI | 44.10% |
| 22 | [Qwen3.8-Max](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±1.9 · 6,474 reported samples | Alibaba | 42.20% |
| 23 | [Opus 4.7](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,474 reported samples | Anthropic | 42.20% |
| 24 | [Grok 4.5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,099 reported samples | xAI | 41.90% |
| 25 | [DeepSeek-V4-Pro-0813](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.9 · 6,306 reported samples | DeepSeek | 41.90% |
| 26 | [Grok 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · xhigh reasoning · Published margin ±1.9 · 6,474 reported samples | xAI | 41.30% |
| 27 | [Gemini 3.6 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,471 reported samples | Google | 41.20% |
| 28 | [Gemini 3.5 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 6,474 reported samples | Google | 40.60% |
| 29 | [DeepSeek-V4.1-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.9 · 6,474 reported samples | DeepSeek | 40.50% |
| 30 | [GLM-5.3](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,474 reported samples | Zhipu | 40.00% |
| 31 | [GPT-5.6 Luna](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.9 · 6,528 reported samples | OpenAI | 39.60% |
| 32 | [DeepSeek-V4-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,428 reported samples | DeepSeek | 39.00% |
| 33 | [GPT-6 Luna](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.9 · 6,474 reported samples | OpenAI | 38.70% |
| 34 | [MiniMax-M3](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.8 · 6,159 reported samples | MiniMax | 37.80% |
| 35 | [Opus 4.8](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,473 reported samples | Anthropic | 37.10% |
| 36 | [Haiku 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.9 · 6,474 reported samples | Anthropic | 36.80% |
| 37 | [Sonnet 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.8 · 6,474 reported samples | Anthropic | 36.30% |
| 38 | [Opus 4.7](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,474 reported samples | Anthropic | 35.40% |
| 39 | [Opus 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,474 reported samples | Anthropic | 35.00% |
| 40 | [DeepSeek-V4-Pro](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 5,267 reported samples | DeepSeek | 34.40% |
| 41 | [Kimi K2.7 Code](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.8 · 6,378 reported samples | Kimi | 34.00% |
| 42 | [GLM-5.3-Flash](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.8 · 6,474 reported samples | Zhipu | 33.60% |
| 43 | [GLM-5.2](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.6 · 6,474 reported samples | Zhipu | 31.70% |
| 44 | [GLM-5.1](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · Published margin ±1.6 · 6,474 reported samples | Zhipu | 29.40% |
| 45 | [Sonnet 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · medium reasoning · Published margin ±1.7 · 6,474 reported samples | Anthropic | 29.20% |
| 46 | [Qwen3.5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · Published margin ±1.7 · 6,474 reported samples | Alibaba | 29.20% |
| 47 | [Nemotron 3 Ultra](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.9 · 4,475 reported samples | NVIDIA | 28.10% |
| 48 | [Sonnet 5](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · max reasoning · Published margin ±1.7 · 6,474 reported samples | Anthropic | 27.30% |
| 49 | [MiniMax-M2.7](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.6 · 6,474 reported samples | MiniMax | 27.10% |
| 50 | [Inkling](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.6 · 6,473 reported samples | Thinking Machines | 25.50% |
| 51 | [Sonnet 4.6](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.6 · 6,453 reported samples | Anthropic | 24.40% |
| 52 | [Gemma 4 31B](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · Published margin ±1.6 · 6,474 reported samples | Google | 23.20% |
| 53 | [DeepSeek-V3.2](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · Published margin ±1.5 · 6,474 reported samples | DeepSeek | 22.30% |
| 54 | [GPT-OSS-120B](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.4 · 6,363 reported samples | OpenAI | 18.50% |
| 55 | [Kimi K2 (Thinking)](https://www.mercor.com/apex/oss-benchmarks/oss-hle-leaderboard/) | Single-shot multimodal · high reasoning · Published margin ±1.2 · 6,306 reported samples | Kimi | 12.70% |

## FAQ

### What does HLE (Mercor) measure?

Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings.

### Which model leads the published HLE (Mercor) snapshot?

Muse Spark 1.1 currently leads the published HLE (Mercor) snapshot with a score of 57.90%.

### How many models are evaluated on HLE (Mercor)?

The October 15, 2026 capture contains 55 source configurations.

### Does HLE (Mercor) affect BenchLM's overall score?

Not directly. HLE (Mercor) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.