RULER context-length reports (RULER)
We show this table for reference; we do not rank on it.
Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.
Evaluation results published by RULER authors and IBM. Selected original author context sweeps and separate Granite 4.2 provider reports.
The original RULER article is CC BY 4.0 and the author repository declares Apache 2.0. These declarations do not grant rights to upstream task content. This page independently reports selected results and excludes task contents. Original article terms.
| Source model or configuration | 4K | 8K | 16K | 32K | 64K | 128K | Setup and original source | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|---|---|---|
| Gemini-1.5-progemini-1.5-pro | 96.7% | 95.8% | 96% | 95.9% | 95.9% | 94.4% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | Not reported | Unknown |
| GPT-4-1106-previewgpt-4-1106-preview | 96.6% | 96.3% | 95.2% | 93.2% | 87% | 81.2% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | Not reported | Unknown |
| Llama3.1 (8B)meta-llama/Meta-Llama-3.1-8B-Instruct | 95.5% | 93.8% | 91.6% | 87.4% | 84.7% | 77% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | Not reported | Unknown |
| Llama3.1 (70B)meta-llama/Meta-Llama-3.1-70B-Instruct | 96.5% | 95.8% | 95.4% | 94.8% | 88.4% | 66.6% | Original author paper cohort: 13 tasks; 500 examples per task and length; recall-based accuracy; greedy decoding. Exact evaluation dates and dated Gemini API snapshot are undisclosed. Paper v3 August 6, 2024; repository checked October 1, 2026 | Not reported | Unknown |
Original 13-task main table; the dated Gemini API snapshot and evaluation dates are undisclosed. Each length value uses the source's recall-based accuracy score.
| Source model or configuration | 64K | 128K | Setup and original source | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| Granite 4.2 30Bibm-granite/granite-4.2-30bNamed profile reference | 89.96 | 81.38 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | Not reported | Open |
| Granite 4.2 8Bibm-granite/granite-4.2-8bNamed profile reference | 80.99 | 71.41 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | Not reported | Open |
| Granite 4.2 3Bibm-granite/granite-4.2-3bNamed profile reference | 67.52 | 55.3 | IBM own-model report; evaluation framework based on NeMo Evaluator SDK. Score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, and run date are unreported. Granite 4.2 release August 25, 2026; retained card revision September 4, 2026 | Not reported | Open |
Provider-reported RULER values, printed without units. Metric/scorer, benchmark version, task composition, thinking mode, and generation settings are unreported; this section is a separate cohort.
Each section retains its source and setup. Matching context labels do not establish comparable runs. Scores stay outside overall and category rankings.
Read each RULER curve within its source
RULER checks whether a model can still retrieve, trace, aggregate, and answer questions as the input grows. This report keeps four historical author curves apart from three Granite 4.2 reports. Rows are ordered by the reported 128K value within each source cohort; it is one context-length result, not an average across lengths.
The original paper evaluates 13 tasks with 500 examples per task and length, using recall-based accuracy and greedy decoding. These four author rows match the paper's August 2024 main table. The newer repository also links separate RULERv1 and RULERv2 pipelines; that does not establish a newer run for the retained scores.
IBM reports bare 64K and 128K values from a framework based on NeMo Evaluator SDK. The card does not identify the score unit, metric/scorer, RULER version, task count, thinking mode, decoding settings, or evaluation date. General inference recommendations in the card do not establish those evaluation settings.
Snapshot
128K values come from two separate evaluation cohorts. The author paper and IBM card do not establish the same benchmark version, harness, or settings; the curve is a report of each source, not a pooled ranking. The table retains each source configuration and its reported value.
7 selected source reports are shown for RULER. The benchmark falls in the Reasoning category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. RULER is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About RULER
Year
2024
Tasks
13 tasks in the original author cohort; IBM task count undisclosed
Format
Reported score by context length
Four historical author rows and three separate IBM provider reports retain their source and setup. IBM units, metric/scorer, benchmark version, and generation settings are unreported. These cohorts do not establish comparable runs.
Freshness and provenance
Version
Selected RULER author curves and IBM Granite 4.2 reports
Refresh cadence
Reviewed pinned source artifacts
Staleness state
Historical author and separate provider cohorts
Question availability
Upstream tasks; no task contents mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does RULER measure?
Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.
Can these source results be compared directly?
128K values come from two separate evaluation cohorts. The author paper and IBM card do not establish the same benchmark version, harness, or settings; the curve is a report of each source, not a pooled ranking. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many source reports are shown?
This snapshot shows 7 selected source reports. Each row retains its original source and disclosed setup.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.