MultiChallenge: provider-reported runs (MultiChallenge (provider reports))
We show this table for reference; we do not rank on it.
Tests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.
Benchmark score on MultiChallenge (provider reports) — October 29, 2026
We compile the MultiChallenge (provider reports) rows from provider self-reports. The table contains 3 models. Mercury 3 has the highest published score at 60.35%, but the available coverage and evaluation setups do not establish a market leader. We do not use these results to rank models overall.
Mercury 3
Inception
Nemotron 3.5 Bolt E4B
NVIDIA
Nemotron 3.5 Bolt E4B BF16
NVIDIA
3 modelsInstruction FollowingCurrentDisplay onlyUpdated October 29, 2026
Benchmark score table (3 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Mercury 3Inception | 60.35% | Not reported | Closed |
| 2 | Nemotron 3.5 Bolt E4BNVIDIA | 55.60% | Not reported | Open |
| 3 | Nemotron 3.5 Bolt E4B BF16NVIDIA | 53.10% | Not reported | Open |
How to read this leaderboard
Operator receipt: 3 sourced rows are currently displayable on this page; the highest published score among these 3 models is Mercury 3 at 60.35%, which does not establish a market leader.
Honest limit: The benchmark owner has revised tasks and judging. Provider runs with undisclosed versions are not interchangeable with the current Scale leaderboard. Display-only and excluded from weighted ranking.
Among the reported MultiChallenge (provider reports) rows, Mercury 3 is first at 60.35%. The third row is 7.25 points behind. The broader top-10 range is 7.25 points, so many of the published results sit in a relatively narrow band.
3 models have been evaluated on MultiChallenge (provider reports). The benchmark falls in the Instruction Following category. MultiChallenge (provider reports) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About MultiChallenge (provider reports)
Year
2025
This lane stores provider-reported runs separately from the Scale-run MultiChallenge leaderboard. Inception reports Mercury 3 at 60.35 with medium reasoning. Its chart does not identify the task revision, judge, prompting, or trial count.
Freshness and provenance
Version
MultiChallenge (provider reports) 2025
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does MultiChallenge (provider reports) measure?
Tests instruction retention, memory of user information, versioned editing, and coherence over multi-turn conversations.
Which model scores highest on MultiChallenge (provider reports)?
Mercury 3 by Inception currently leads with a score of 60.35% on MultiChallenge (provider reports).
How many models are evaluated on MultiChallenge (provider reports)?
3 AI models have published results on MultiChallenge (provider reports) in the BenchLM catalog.
Compare top models on MultiChallenge (provider reports)
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.