NoLiMa original-task reports (NoLiMa)
We show this table for reference; we do not rank on it.
Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.
Evaluation results published by NoLiMa authors. Selected original NoLiMa accuracy curves, with short-context base scores and source configurations.
The original paper is CC BY-NC-SA 4.0. Evaluation code and needle data carry the Adobe Research License, which permits noncommercial research. This page independently reports selected factual scores with its own text and layout; it does not reproduce source tables, code, or task contents or claim a new commercial license. Original article terms.
| Source model or configuration | Base | 1K | 2K | 4K | 8K | 16K | 32K | 64K | 128K | Setup and original source | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4.1gpt-4.1-2025-04-14 | 97% | 95.6% | 95.2% | 91.7% | 87.5% | 84.9% | 79.8% | 69.7% | 64.7% | Azure OpenAI; source config: temperature 0, top-p 1, output cap 192. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. README update June 9, 2025 | Not reported | Unknown |
| GPT-4ogpt-4o-2024-11-20 | 99.3% | 98.1% | 98% | 95.7% | 89.2% | 81.6% | 69.7% | 62.4% | 56% | Azure OpenAI; source config: temperature 0, top-p 1, output cap 192. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. Original paper cohort; extended-length results added June 9, 2025 | Not reported | Unknown |
| Gemini 2.5 Flash (w/o T)gemini-2.5-flash-preview-05-20 | 94.4% | 90.1% | 86.1% | 79.4% | 68.2% | 57.9% | 48.4% | Not reported | Not reported | Gemini API; source config: thinking budget 0, temperature 0, top-p 1, output cap 192; preview May 20 edition. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. README update June 9, 2025 | Not reported | Unknown |
Accuracy percentages. Through 32K: 26 placements; at 64K/128K: 11 placements and token-limit adjustments. A blank source cell remains unreported.
Base is a short-context reference, not a length average. Extended-length test counts differ. Historical source reports stay outside overall and category rankings.
NoLiMa requires more than matching words
NoLiMa asks a model to connect a question to a relevant fact whose wording differs, then find the answer in a long passage. This selected report keeps the original task separate from NoLiMa-Hard. Sorting uses the published 32K accuracy, with the rest of the curve beside it.
Accuracy counts answers that contain the correct character name. The original setup uses 58 question-needle pairs and five haystacks, with 26 placements per length through 32K. The later 64K and 128K results use 11 placements and account for token limits, so the test counts differ.
The published base score uses the best result at 250, 500, or 1K for each question-needle example after averaging over haystacks, then averages those best results. Effective length uses 85% of that base score. The page shows published values; it does not recalculate either measure.
The repository supplies dated configurations for GPT-4.1 and GPT-4o and a May 20 preview for Gemini 2.5 Flash with thinking disabled. Each row links to its configuration. The files are source evidence rather than per-run manifests; evaluation dates remain undisclosed. The preview stays separate from the final Flash release.
Snapshot
The selected 32K values use the original NoLiMa task. Extended 64K/128K results use fewer placements, and the Gemini Flash row is a May 2025 preview with thinking disabled. These results do not establish current API performance. The table retains each source configuration and its reported value.
3 selected original-task reports are shown for NoLiMa. The benchmark falls in the Reasoning category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. NoLiMa is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About NoLiMa
Year
2025
Tasks
58 question-needle pairs; five haystacks; 26 placements through 32K
Format
Character-name answer accuracy by context length
The 64K/128K results use 11 placements and token-limit adjustments. Base uses each example's best short-context result after haystack averaging. Source configuration files are not per-run manifests, and the Gemini row is a May 2025 preview without thinking.
Freshness and provenance
Version
Selected original NoLiMa curves; author repository July 17, 2025
Refresh cadence
Reviewed pinned author artifact
Staleness state
Historical dated API configurations
Question availability
Upstream research terms; no task contents mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does NoLiMa measure?
Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.
Can these source results be compared directly?
The selected 32K values use the original NoLiMa task. Extended 64K/128K results use fewer placements, and the Gemini Flash row is a May 2025 preview with thinking disabled. These results do not establish current API performance. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many source reports are shown?
This snapshot shows 3 selected source reports. Each row retains its original source and disclosed setup.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.