# RULER context-length reports (RULER)

> Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.

Canonical page: https://benchlm.ai/benchmarks/ruler

- Category: [Reasoning](/reasoning)
- Last updated: Source check October 1, 2026

Evaluation results published by RULER authors and IBM. Selected original author context sweeps and separate Granite 4.2 provider reports.

The original RULER article is CC BY 4.0 and the author repository declares Apache 2.0. These declarations do not grant rights to upstream task content. This page independently reports selected results and excludes task contents. [Original article terms](https://creativecommons.org/licenses/by/4.0/).

RULER checks whether a model can still retrieve, trace, aggregate, and answer questions as the input grows. This report keeps four historical author curves apart from three Granite 4.2 reports. Rows are ordered by the reported 128K value within each source cohort; it is one context-length result, not an average across lengths.

The original paper evaluates 13 tasks with 500 examples per task and length, using recall-based accuracy and greedy decoding. These four author rows match the paper's August 2024 main table. The newer repository also links separate RULERv1 and RULERv2 pipelines; that does not establish a newer run for the retained scores.

IBM reports bare 64K and 128K values from a framework based on NeMo Evaluator SDK. The card does not identify the score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, or evaluation date. General inference recommendations in the card do not establish those evaluation settings.

128K values come from two separate evaluation cohorts. The author paper and IBM card do not establish the same benchmark version, harness, or settings; the curve is a report of each source, not a pooled ranking.

- [Pinned original author results](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/README.md)
- [Original paper main results and method](https://arxiv.org/html/2404.06654v3#S4.T3)
- [Pinned IBM own-model report](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md)

## About RULER

- Year: 2024
- Tasks: 13 tasks in the original author cohort; IBM task count undisclosed
- Format: Reported score by context length
- Paper: [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://arxiv.org/abs/2404.06654v3)

Four historical author rows and three separate IBM provider reports retain their source and setup. IBM units, metric/scorer, benchmark version, and generation settings are unreported. These cohorts do not establish comparable runs.

RULER is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (7 selected source reports)

### Original author paper cohort (August 2024)

| Source model or configuration | Evaluated identity | 4K | 8K | 16K | 32K | 64K | 128K | Setup and report label | Original source |
|---|---|---|---|---|---|---|---|---|---|
| [Gemini-1.5-pro](https://ai.google.dev/gemini-api/docs/models/gemini#:~:text=Gemini-,Gemini%201.5%20Pro%20%28Preview%20only%29,-Text%20and%20images) | gemini-1.5-pro | 96.7% | 95.8% | 96% | 95.9% | 95.9% | 94.4% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | [Original result](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/README.md); [Source setup](https://arxiv.org/html/2404.06654v3#S4) |
| [GPT-4-1106-preview](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4#:~:text=gpt%2D4%2D1106%2Dpreview,Up%20to%20Apr%202023) | gpt-4-1106-preview | 96.6% | 96.3% | 95.2% | 93.2% | 87% | 81.2% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | [Original result](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/README.md); [Source setup](https://arxiv.org/html/2404.06654v3#S4) |
| [Llama3.1 (8B)](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) | meta-llama/Meta-Llama-3.1-8B-Instruct | 95.5% | 93.8% | 91.6% | 87.4% | 84.7% | 77% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | [Original result](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/README.md); [Source setup](https://arxiv.org/html/2404.06654v3#S4) |
| [Llama3.1 (70B)](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) | meta-llama/Meta-Llama-3.1-70B-Instruct | 96.5% | 95.8% | 95.4% | 94.8% | 88.4% | 66.6% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | [Original result](https://github.com/NVIDIA/RULER/blob/c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a/README.md); [Source setup](https://arxiv.org/html/2404.06654v3#S4) |

Original 13-task main table; the dated Gemini API snapshot and evaluation dates are undisclosed. Each length value uses the source's recall-based accuracy score.

### IBM Granite 4.2 own-model report (August 2026 release)

| Source model or configuration | Evaluated identity | 64K | 128K | Setup and report label | Original source |
|---|---|---|---|---|---|
| [Granite 4.2 30B](/models/granite-4-2-30b) | ibm-granite/granite-4.2-30b | 89.96 | 81.38 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | [Original result](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md); [Source setup](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md) |
| [Granite 4.2 8B](/models/granite-4-2-8b) | ibm-granite/granite-4.2-8b | 80.99 | 71.41 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | [Original result](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md); [Source setup](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md) |
| [Granite 4.2 3B](/models/granite-4-2-3b) | ibm-granite/granite-4.2-3b | 67.52 | 55.3 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | [Original result](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md); [Source setup](https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md) |

Provider-reported RULER values, printed without units. Metric/scorer, benchmark version, task composition, thinking mode, and generation settings are unreported; this section is a separate cohort.

Each section retains its source and setup. Matching context labels do not establish comparable runs. Scores stay outside overall and category rankings.

## FAQ

### What does RULER measure?

Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.

### Can these source results be compared directly?

128K values come from two separate evaluation cohorts. The author paper and IBM card do not establish the same benchmark version, harness, or settings; the curve is a report of each source, not a pooled ranking. Results stay outside weighted overall and category scoring.

### How many source reports are shown?

This snapshot shows 7 selected source reports. Each row retains its original source and disclosed setup.

### Does RULER affect BenchLM's overall score?

Not directly. RULER is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.