# MultiChallenge: provider-reported runs (MultiChallenge (provider reports))

> Tests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.

Canonical page: https://benchlm.ai/benchmarks/multichallenge-provider-reports

- Category: [Instruction Following](/instruction-following)
- Last updated: October 29, 2026

## About MultiChallenge (provider reports)

- Year: 2025
- Paper: [MultiChallenge](https://labs.scale.com/leaderboard/multichallenge)

This lane stores provider-reported runs separately from the Scale-run MultiChallenge leaderboard. Inception reports Mercury 3 at 60.35 with medium reasoning. Its chart does not identify the task revision, judge, prompting, or trial count.

MultiChallenge (provider reports) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (3 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [Mercury 3](/models/mercury-3) | Inception | 60.35% |
| 2 | [Nemotron 3.5 Bolt E4B](/models/nemotron-3-5-bolt-e4b) | NVIDIA | 55.60% |
| 3 | [Nemotron 3.5 Bolt E4B BF16](/models/nemotron-3-5-bolt-e4b-bf16) | NVIDIA | 53.10% |

## FAQ

### What does MultiChallenge (provider reports) measure?

Tests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.

### Which model scores highest on MultiChallenge (provider reports)?

Mercury 3 by Inception currently leads with a score of 60.35% on MultiChallenge (provider reports).

### How many models are evaluated on MultiChallenge (provider reports)?

3 AI models have published results on MultiChallenge (provider reports) in the BenchLM catalog.

### Does MultiChallenge (provider reports) affect BenchLM's overall score?

Not directly. MultiChallenge (provider reports) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Compare Top Models on MultiChallenge (provider reports)

- [Mercury 3 vs Nemotron 3.5 Bolt E4B](/compare/mercury-3-vs-nemotron-3-5-bolt-e4b)
- [Nemotron 3.5 Bolt E4B vs Nemotron 3.5 Bolt E4B BF16](/compare/nemotron-3-5-bolt-e4b-vs-nemotron-3-5-bolt-e4b-bf16)