The desk
How we work
A ranking is a shortlist.
Rank orders the evidence; it does not make the decision. Every row here opens to its sources so you can disagree with us.
We do not invent scores to fill a table.
A model without a public, independent row stays unranked, and says so. Vendor-run numbers are shown, labeled, and kept out of the rank.
When we were wrong, the fix is on the page.
Scoring bugs, missing coverage, and re-weightings are dated and linked, because a ranking you cannot audit is a launch post with a table.
Affiliate partnerships never influence scores, rankings, or model coverage. See the affiliate disclosure for details.
Who published the results on model pages, and how each kind of source counts toward a rank, is counted on where our scores come from.
Evidence strength is separate from score
Supported means the row clears the independent-evidence rules. Estimated means it relies on thinner evidence. The labels describe the inputs; conditional score ranges do not establish rank confidence.
Freshness is explicit
Every benchmark now carries BenchLM freshness metadata: version, refresh cadence, staleness state, saturation state, and whether the benchmark is weighted or display-only.
Independent signals share one scale
BenchAlign calibrates benchmark protocols, human-preference signals, and external composite indices before combining them. Raw percentages are never averaged across tests with different difficulty. Runtime metrics remain separate and were updated 2026-11-10.
Six pillars and familiar evidence lenses
The product ontology has six pillars: Agentic Work, Software Engineering, Reasoning and Mathematics, Knowledge and Factuality, Multimodal and Documents, and Communication and Language. Existing category URLs remain indexable evidence lenses beneath those pillars.
Agentic
22 scored benchmarks and 123 display-only benchmarks.
Coding
10 scored benchmarks and 59 display-only benchmarks.
Reasoning
6 scored benchmarks and 33 display-only benchmarks.
Multimodal
7 scored benchmarks and 71 display-only benchmarks.
Knowledge
10 scored benchmarks and 57 display-only benchmarks.
Multilingual
1 scored benchmark and 13 display-only benchmarks.
Instruction Following
2 scored benchmarks and 15 display-only benchmarks.
Math
5 scored benchmarks and 29 display-only benchmarks.
Missing data increases uncertainty
An Estimated position uses a learned blend of the model's own signals, its calibrated benchmark estimate, a reviewed lineage prior, the previous policy estimate, and eligible peers. Missing tests remain blank; the estimator never adds benchmark values. Supported Overall scores keep their existing calculation, although their ranks can move when Estimated rows change.
Profile peers must share at least two benchmark protocols from two benchmark families. The peer set needs at least three independent model families across two providers, and excludes the target's own family and provider. After accounting for evidence strength, peers can carry at most 60% of the final blend. A fallback based only on source similarity needs five independent peer families and carries at most 25%. Both limits shrink when the estimate extends beyond the peer scores.
The existing checks still run after the blend: broad benchmark evidence can lower an optimistic estimate, lineage never forces a newer release above its predecessor, and higher-effort variants retain their advantage only where their own results support it. A position resting only on preference leaderboards, without a benchmark composite or connected benchmark evidence, stays withheld. Unrounded scores determine order.
Method bench-align-v5.8-2026-09-30 · current input c99bf4536c5f154a
Benchmarks by category
A scored benchmark moves at least one public ranking. On Agentic, Coding, Knowledge, Reasoning, and Multimodal, a benchmark's percentage is its share of that category's reference weight, a relative weight in the calibrated model rather than a fixed share of any score. A benchmark stored under two categories, such as Terminal-Bench 2, is scored where the model uses it. The other categories rank by a weighted category score, and their percentages are shares of that score.
Agentic
145 tracked benchmarks
Scored
Display only
Subfamilies
Coding
69 tracked benchmarks
Scored
Display only
Reasoning
39 tracked benchmarks
Display only
Subfamilies
Multimodal
78 tracked benchmarks
Display only
Subfamilies
Knowledge
67 tracked benchmarks
Scored
Display only
Subfamilies
Multilingual
14 tracked benchmarks
Instruction Following
17 tracked benchmarks
Math
34 tracked benchmarks
Display only
Future tracked families
BenchLM tracks a small number of important benchmark families that are intentionally not weighted yet because exact-source density and cross-model coverage are still too thin for defensible ranking use.
Calibration approach
External consensus signals
Overall capability starts from independent external leaderboard families, such as benchmark composite indices and human-preference leaderboards, each weighted by its uncertainty. When several families are available, none may carry more than half of the external part of the score, and preference leaderboards together carry at most a quarter of it. Direct benchmark evidence then adds a correction of at most 3 points, scaled by how independent that evidence is. An overall position is Supported when two external families from different evidence classes agree and no single source carries more than half, or when two families are confirmed by broad independent benchmark evidence that lands within 3 points of them. A preference leaderboard plus one index is not Supported unless benchmarks confirm it. A model with no external source is scored from its benchmark evidence; its overall position stays Estimated, and only a category position can be Supported by broad independent benchmarks alone. A category position is also Supported when its external category sources span two evidence classes, or when an external category source is backed by independent benchmark evidence from at least two benchmark families. Every other position is Estimated, and Supported and Estimated rows share one ordering. Agentic and Coding combine external category signals with benchmark evidence. Knowledge has no external category source, so its Supported rows come from broad independent benchmark evidence. Without an external category source, a category position needs two benchmark families and a published overall position; a single benchmark is not a category measurement. Where category evidence is thin, all three lenses pool toward the model’s general capability, estimated without preference leaderboards.
Runtime metrics
Runtime metrics (tokens/sec, time-to-first-token) stay separate from ranking and are shown as operational metadata only. They do not affect overall or category scores.
Source refresh: 2026-11-10
Estimated ranks have limits
The September 30, 2026 retrospective tests reduced average Overall reconstruction error. Three checks of 95th-percentile error still worsened: family holdouts for preference-only rows and lower-scoring models, and provider holdouts for one middle-score group. These tests reused earlier cohorts. Publishing the estimator does not make those failed checks pass or establish future accuracy.
Estimated Overall ranges describe reconstruction error under the tested assumptions. Preference-only and benchmark-only patterns have too few independent calibration families to establish their own coverage, so their ranges use a broader residual pool. These ranges do not establish confidence in a rank, a future score, or the ordering of the whole board.
Agentic, Coding, Knowledge, Reasoning, and Multimodal also use a preference-free general estimate when category evidence is thin. Changing that estimate can move both Supported and Estimated category scores. The category intervals have not been validated for the added uncertainty from those prior and mapping changes; a Supported category label still describes its evidence, not validated coverage for the new interval.
BenchLM defaults and caveats
BenchLM uses benchmark freshness as a product layer, not as a claim about an official benchmark maintainer. A benchmark marked Current means BenchLM still treats it as a strong differentiator. A benchmark marked Stale means it is still useful for context but is no longer relied on heavily to separate frontier models.
Public BenchLM benchmark tables default to exact-source rows only. Generated benchmark values remain excluded from BenchAlign evidence.
Overall, Agentic, Coding, Knowledge, Reasoning, and Multimodal use BenchAlign v5.8; Multimodal is a preliminary board with a thin roster. We rebuild the rankings from current inputs on every build rather than freezing them at launch. Math, Multilingual, and Instruction Following remain evidence lenses on the legacy category score until their evidence is deep enough to rank frontier models.
BenchLM does not estimate runtime metrics when no sourced runtime snapshot is available. The leaderboard and model pages show N/A instead. Pricing sort uses the average of input and output token price so models can be ranked on a single cost column while the full input/output pair remains visible.
Last benchmark dataset refresh: November 11, 2026. For raw benchmark exploration, use the benchmark directory. For current provider rollups, use provider pages.