# NoLiMa-Hard selected reports (NoLiMa-Hard)

> Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.

Canonical page: https://benchlm.ai/benchmarks/nolima-hard

- Category: [Reasoning](/reasoning)
- Last updated: Hard results through July 2025; checked October 1, 2026

Evaluation results published by NoLiMa authors. Selected reasoning-model accuracy curves on the ten hardest question-needle pairs.

The original paper is CC BY-NC-SA 4.0. Evaluation code and needle data carry noncommercial Adobe research terms. This independently authored selective numerical report excludes source-table layout, source prose, code, and task contents; it does not claim a new commercial license. [Original article terms](https://creativecommons.org/licenses/by-nc-sa/4.0/).

NoLiMa-Hard contains the ten question-needle pairs that were most difficult in the original evaluations. This selected report shows three later reasoning-model curves published in the author repository. Sorting uses the exact 32K accuracy; the values do not belong to the original NoLiMa task.

The rows preserve the source names GPT-o3, Gemini 2.5 Pro, and GPT-o4 Mini. Exact API snapshots, reasoning budgets, decoding settings, and evaluation dates are unreported. Settings from the older paper's o1 and o3-mini experiments cannot fill those gaps.

Each value is the reported accuracy for that context length. The base score remains the source's short-context reference, not a mean across the displayed lengths. The later Hard rows do not disclose their per-length case counts beside the results.

These selected reasoning-model reports use the ten-pair NoLiMa-Hard subset. Exact API snapshots and reasoning budgets are undisclosed; they cannot be joined to final model SKUs or the original-task table.

- [Pinned original author results](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md)
- [Original metric and evaluation setup](https://arxiv.org/html/2502.05167v2#S4.SS3)
- [Original paper and article terms](https://arxiv.org/abs/2502.05167v2)
- [Adobe terms for evaluation code and needle data](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/LICENSE)

## About NoLiMa-Hard

- Year: 2025
- Tasks: 10 hardest question-needle pairs
- Format: Hard-subset character-name answer accuracy
- Paper: [NoLiMa: Long-Context Evaluation Beyond Literal Matching](https://arxiv.org/abs/2502.05167v2)

Selected later reasoning-model reports retain source labels. The exact API snapshots, reasoning budgets, decoding settings, evaluation dates, and per-length case counts are undisclosed. These values remain separate from the original task.

NoLiMa-Hard is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (3 selected Hard-subset reports)

### NoLiMa-Hard reasoning-model reports (June-July 2025)

| Source model or configuration | Evaluated identity | Base | 4K | 8K | 16K | 32K | Setup and report label | Original source |
|---|---|---|---|---|---|---|---|---|
| [Gemini 2.5 Pro](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results) | nolima-hard:Gemini 2.5 Pro | 99.1% | 73.9% | 63% | 58.6% | 58.6% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update June 9, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results); [Source setup](https://arxiv.org/html/2502.05167v2#S4.SS4.SSS3) |
| [GPT-o3](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results) | nolima-hard:GPT-o3 | 100% | 94.4% | 86.2% | 74.9% | 58.5% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update July 17, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results); [Source setup](https://arxiv.org/html/2502.05167v2#S4.SS4.SSS3) |
| [GPT-o4 Mini](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results) | nolima-hard:GPT-o4 Mini | 99.6% | 57.4% | 30.8% | 20.2% | 11.7% | NoLiMa-Hard: 10 hardest question-needle pairs. Reasoning-model row; exact API snapshot, reasoning budget, decoding settings, and run date are unreported. README update July 17, 2025 | [Original result](https://github.com/adobe-research/NoLiMa/blob/cb14780b249fecf2851127b2101a062c1b2c6430/README.md#nolima-hard-results); [Source setup](https://arxiv.org/html/2502.05167v2#S4.SS4.SSS3) |

Hard-subset accuracy percentages. Exact model snapshots, generation settings, and case counts are undisclosed for these later rows.

Source names remain external because the exact API edition and effort are unreported. Results stay outside overall and category rankings.

## FAQ

### What does NoLiMa-Hard measure?

Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.

### Can these source results be compared directly?

These selected reasoning-model reports use the ten-pair NoLiMa-Hard subset. Exact API snapshots and reasoning budgets are undisclosed; they cannot be joined to final model SKUs or the original-task table. Results stay outside weighted overall and category scoring.

### How many source reports are shown?

This snapshot shows 3 selected source reports. Each row retains its original source and disclosed setup.

### Does NoLiMa-Hard affect BenchLM's overall score?

Not directly. NoLiMa-Hard is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.