# NoLiMa original-task reports (NoLiMa)

> Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.

Canonical page: https://benchlm.ai/benchmarks/nolima

- Category: [Reasoning](/reasoning)
- Last updated: Original results through June 2025; checked October 1, 2026

Evaluation results published by NoLiMa authors. Selected original NoLiMa accuracy curves, with short-context base scores and source configurations.

The original paper is CC BY-NC-SA 4.0. Evaluation code and needle data carry the Adobe Research License, which permits noncommercial research. This page independently reports selected factual scores with its own text and layout; it does not reproduce source tables, code, or task contents or claim a new commercial license. [Original article terms](https://creativecommons.org/licenses/by-nc-sa/4.0/).

NoLiMa asks a model to connect a question to a relevant fact whose wording differs, then find the answer in a long passage. This selected report keeps the original task separate from NoLiMa-Hard. Sorting uses the published 32K accuracy, with the rest of the curve beside it.

Accuracy counts answers that contain the correct character name. The original setup uses 58 question-needle pairs and five haystacks, with 26 placements per length through 32K. The later 64K and 128K results use 11 placements and account for token limits, so the test counts differ.

The published base score uses the best result at 250, 500, or 1K for each question-needle example after averaging over haystacks, then averages those best results. Effective length uses 85% of that base score. The page shows published values; it does not recalculate either measure.

The repository supplies dated configurations for GPT-4.1 and GPT-4o and a May 20 preview for Gemini 2.5 Flash with thinking disabled. Each row links to its configuration. The files are source evidence rather than per-run manifests; evaluation dates remain undisclosed. The preview stays separate from the final Flash release.

The selected 32K values use the original NoLiMa task. Extended 64K/128K results use fewer placements, and the Gemini Flash row is a May 2025 preview with thinking disabled. These results do not establish current API performance.

- [Pinned original author results](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md)
- [Original metric and evaluation setup](https://arxiv.org/html/2502.05167v2#S4.SS3)
- [Original paper and article terms](https://arxiv.org/abs/2502.05167v2)
- [Adobe terms for evaluation code and needle data](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/LICENSE)

## About NoLiMa

- Year: 2025
- Tasks: 58 question-needle pairs; five haystacks; 26 placements through 32K
- Format: Character-name answer accuracy by context length
- Paper: [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://arxiv.org/abs/2502.05167v2)

The 64K/128K results use 11 placements and token-limit adjustments. Base uses each example's best short-context result after haystack averaging. Source configuration files are not per-run manifests, and the Gemini row is a May 2025 preview without thinking.

NoLiMa is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (3 selected original-task reports)

### Original NoLiMa task

| Source model or configuration | Evaluated identity | Base | 1K | 2K | 4K | 8K | 16K | 32K | 64K | 128K | Setup and report label | Original source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| [GPT-4.1](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results) | gpt-4.1-2025-04-14 | 97% | 95.6% | 95.2% | 91.7% | 87.5% | 84.9% | 79.8% | 69.7% | 64.7% | Azure OpenAI; source config: temperature 0, top-p 1, output cap 192. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. README update June 9, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results); [Source setup](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/evaluation/model_configs/gpt_4.1_20250414.json) |
| [GPT-4o](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results) | gpt-4o-2024-11-20 | 99.3% | 98.1% | 98% | 95.7% | 89.2% | 81.6% | 69.7% | 62.4% | 56% | Azure OpenAI; source config: temperature 0, top-p 1, output cap 192. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. Original paper cohort; extended-length results added June 9, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results); [Source setup](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/evaluation/model_configs/gpt_4o_20241120.json) |
| [Gemini 2.5 Flash (w/o T)](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results) | gemini-2.5-flash-preview-05-20 | 94.4% | 90.1% | 86.1% | 79.4% | 68.2% | 57.9% | 48.4% | Not reported | Not reported | Gemini API; source config: thinking budget 0, temperature 0, top-p 1, output cap 192; preview May 20 edition. The adjacent configuration is source evidence, not a per-run manifest. 26 placements through 32K; 11 at 64K and 128K. README update June 9, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#results); [Source setup](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/evaluation/model_configs/gemini_2.5_flash.json) |

Accuracy percentages. Through 32K: 26 placements; at 64K/128K: 11 placements and token-limit adjustments. A blank source cell remains unreported.

Base is a short-context reference, not a length average. Extended-length test counts differ. Historical source reports stay outside overall and category rankings.

## FAQ

### What does NoLiMa measure?

Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.

### Can these source results be compared directly?

The selected 32K values use the original NoLiMa task. Extended 64K/128K results use fewer placements, and the Gemini Flash row is a May 2025 preview with thinking disabled. These results do not establish current API performance. Results stay outside weighted overall and category scoring.

### How many source reports are shown?

This snapshot shows 3 selected source reports. Each row retains its original source and disclosed setup.

### Does NoLiMa affect BenchLM's overall score?

Not directly. NoLiMa is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.