# SimpleQA Verified

> SimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate.

Canonical page: https://benchlm.ai/benchmarks/simpleqa-verified

- Category: [Knowledge](/knowledge)
- Last updated: March 10, 2026 paper revision

Evaluation results published by [SimpleQA Verified authors](https://arxiv.org/html/2509.07968v2#S4.T7). Published paper results for the 1,000-prompt set. No task contents or model responses are included.

## About SimpleQA Verified

- Year: 2025
- Tasks: 1,000 prompts
- Format: Short-form factual answers
- Paper: [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://arxiv.org/abs/2509.07968v2)

The paper uses gpt-4.1-2025-04-14 as its revised autorater. March 10, 2026 is the paper revision date, not an evaluation date. Source API snapshots and reasoning efforts are undisclosed. Results stay display only and do not enter overall or category scores.

SimpleQA Verified is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (13 source model rows)

| Paper rank | Source model | F1 | Accuracy | Accuracy given attempted | Attempted | Hedged | F1 change from SimpleQA (pp) |
|---|---|---|---|---|---|---|---|
| 1 | [Gemini 2.5 Pro](/models/gemini-2-5-pro) | 55.6% | 55.3% | 55.9% | 98.9% | 1.1% | +0.5 |
| 2 | [GPT 5](https://arxiv.org/html/2509.07968v2#S4.T7) | 52.3% | 50.9% | 53.8% | 94.6% | 5.4% | +1.8 |
| 3 | [o3](/models/o3) | 51.9% | 51.6% | 52.0% | 99.3% | 0.7% | +1.9 |
| 4 | [GPT 4.1](/models/gpt-4-1) | 39.9% | 39.8% | 40.1% | 99.3% | 0.7% | -1.0 |
| 5 | [GPT 4o](/models/gpt-4o) | 34.9% | 34.4% | 35.5% | 97.0% | 3.0% | -3.5* |
| 6 | [DeepSeek R1 (0528)](https://arxiv.org/html/2509.07968v2#S4.T7) | 33.3% | 32.7% | 33.9% | 96.4% | 3.6% | +1.4 |
| 7 | [Claude Opus 4](https://arxiv.org/html/2509.07968v2#S4.T7) | 28.3% | 19.2% | 54.1% | 35.5% | 64.5% | -4.0* |
| 8 | [Gemini 2.5 Flash](/models/gemini-2-5-flash) | 28.2% | 27.8% | 28.7% | 96.9% | 3.1% | -1.4 |
| 9 | [GPT 5 Mini](/models/gpt-5-mini) | 24.6% | 17.3% | 42.8% | 40.4% | 59.6% | +1.1 |
| 10 | [o4-mini](https://arxiv.org/html/2509.07968v2#S4.T7) | 23.4% | 23.0% | 23.8% | 96.5% | 3.5% | +2.9* |
| 11 | [Claude Sonnet 4](/models/claude-4-sonnet) | 18.7% | 12.5% | 36.9% | 33.9% | 66.1% | -4.4* |
| 12 | [GPT 5 Nano](/models/gpt-5-nano) | 14.4% | 10.2% | 24.2% | 42.2% | 57.8% | +0.7 |
| 13 | [Gemini 2.5 Flash Lite](https://arxiv.org/html/2509.07968v2#S4.T7) | 11.1% | 10.2% | 12.1% | 84.0% | 16.0% | -0.4 |

All five metrics are percentages. F1 change is in percentage points; * marks the paper's p < 0.05 result. Both benchmarks use the same revised autorater. Profile links refer to the named generation; evaluated API snapshots and reasoning efforts are not reported.

## Paper result attribution

Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, and Dipanjan Das, SimpleQA Verified, arXiv:2509.07968v2, Table 7. Copyright 2026 Google. All rights reserved. [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). Table 7 transcribed into structured aggregate fields and reformatted. Scores are unchanged. The paper table retains its upstream license. Paper result table only. No task data, prompts, responses, or Kaggle leaderboard contents are included.

## FAQ

### What does SimpleQA Verified measure?

SimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate.

### Which model leads the published SimpleQA Verified snapshot?

Gemini 2.5 Pro currently leads the published SimpleQA Verified snapshot with a score of 55.6%.

### How many models are evaluated on SimpleQA Verified?

The March 10, 2026 paper revision contains 13 AI models.

### Does SimpleQA Verified affect BenchLM's overall score?

Not directly. SimpleQA Verified is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.