Benchmark Confidence & Contamination Flags

Score confidence (1-4 dots) indicates how much sourced benchmark data supports a model's score. A 4-dot score has at least 20 sourced rows across seven categories. A 1-dot score has limited coverage. Some benchmark-sparse models can appear provisionally when several independent public evaluation families agree.

Scores marked with "Est." or "~" have limited source-backed benchmark coverage or qualify through corroborating external evaluation families. Unresolved manual values and generated rows cannot score. The verified leaderboard requires sourced benchmark support; external consensus alone never upgrades an estimated row to verified status.

BenchLM tracks two key signals: (1) benchmark provenance — whether each score is a hand-entered public row ("manual") or was generated/inferred from related data, and (2) benchmark freshness — older benchmarks that haven't been updated are more likely to have been contaminated through training data inclusion. Models with mostly generated data or stale benchmarks receive lower confidence ratings. Exact-source verification is tracked separately from this manual-vs-generated split.

Provenance records where each benchmark value came from and whether the exact model, benchmark variant, harness, and score can be supported. Source-backed rows can score. Unresolved manual values and generated estimates remain in the audit trail but do not affect either public ranking view.

Fresh, held-out benchmarks like SWE-Rebench (rolling window), Terminal-Bench 2.0, and HLE are the hardest to game. Older, saturated benchmarks like MMLU (where top models all score 97-99%) provide little signal. BenchLM weights newer, harder benchmarks more heavily and flags saturated ones as display-only.