NoLiMa-Hard selected reports (NoLiMa-Hard)
We show this table for reference; we do not rank on it.
Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.
Evaluation results published by NoLiMa authors. Selected reasoning-model accuracy curves on the ten hardest question-needle pairs.
The original paper is CC BY-NC-SA 4.0. Evaluation code and needle data carry noncommercial Adobe research terms. This independently authored selective numerical report excludes source-table layout, source prose, code, and task contents; it does not claim a new commercial license. Original article terms.
| Source model or configuration | Base | 4K | 8K | 16K | 32K | Setup and original source | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|---|---|
| Gemini 2.5 Pronolima-hard:Gemini 2.5 Pro | 99.1% | 73.9% | 63% | 58.6% | 58.6% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update June 9, 2025 | Not reported | Unknown |
| GPT-o3nolima-hard:GPT-o3 | 100% | 94.4% | 86.2% | 74.9% | 58.5% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update July 17, 2025 | Not reported | Unknown |
| GPT-o4 Mininolima-hard:GPT-o4 Mini | 99.6% | 57.4% | 30.8% | 20.2% | 11.7% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update July 17, 2025 | Not reported | Unknown |
Hard-subset accuracy percentages. Exact model snapshots, generation settings, and case counts are undisclosed for these later rows.
Source names remain external because the exact API edition and effort are unreported. Results stay outside overall and category rankings.
Keep the Hard subset separate
NoLiMa-Hard contains the ten question-needle pairs that were most difficult in the original evaluations. This selected report shows three later reasoning-model curves published in the author repository. Sorting uses the exact 32K accuracy; the values do not belong to the original NoLiMa task.
The rows preserve the source names GPT-o3, Gemini 2.5 Pro, and GPT-o4 Mini. Exact API snapshots, reasoning budgets, decoding settings, and evaluation dates are unreported. Settings from the older paper's o1 and o3-mini experiments cannot fill those gaps.
Each value is the reported accuracy for that context length. The base score remains the source's short-context reference, not a mean across the displayed lengths. The later Hard rows do not disclose their per-length case counts beside the results.
Snapshot
These selected reasoning-model reports use the ten-pair NoLiMa-Hard subset. Exact API snapshots and reasoning budgets are undisclosed; they cannot be joined to final model SKUs or the original-task table. The table retains each source configuration and its reported value.
3 selected Hard-subset reports are shown for NoLiMa-Hard. The benchmark falls in the Reasoning category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. NoLiMa-Hard is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About NoLiMa-Hard
Year
2025
Tasks
10 hardest question-needle pairs
Format
Hard-subset character-name answer accuracy
Selected later reasoning-model reports retain source labels. The exact API snapshots, reasoning budgets, decoding settings, evaluation dates, and per-length case counts are undisclosed. These values remain separate from the original task.
Freshness and provenance
Version
Selected NoLiMa-Hard curves; author repository July 17, 2025
Refresh cadence
Reviewed pinned author artifact
Staleness state
Historical reasoning-model source reports
Question availability
Upstream research terms; no task contents mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does NoLiMa-Hard measure?
Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.
Can these source results be compared directly?
These selected reasoning-model reports use the ten-pair NoLiMa-Hard subset. Exact API snapshots and reasoning budgets are undisclosed; they cannot be joined to final model SKUs or the original-task table. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many source reports are shown?
This snapshot shows 3 selected source reports. Each row retains its original source and disclosed setup.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.