SimpleQA Verified
We show this table for reference; we do not rank on it.
SimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate.
Evaluation results published by SimpleQA Verified authors. Published paper results for the 1,000-prompt set. No task contents or model responses are included.
Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, and Dipanjan Das, SimpleQA Verified, arXiv:2509.07968v2, Table 7. Copyright 2026 Google. All rights reserved. CC BY 4.0. Table 7 transcribed into structured aggregate fields and reformatted. Scores are unchanged. The paper table retains its upstream license.
F1 on SimpleQA Verified — March 10, 2026 paper revision
We mirror the published f1 view for SimpleQA Verified. Gemini 2.5 Pro leads the public snapshot at 55.6%, followed by GPT 5 (52.3%) and o3 (51.9%). We do not use these results to rank models overall.
Gemini 2.5 Pro
Standard API parameters; no tools; API snapshot and effort not reported
GPT 5
OpenAI
Standard API parameters; no tools; API snapshot and effort not reported
o3
OpenAI
Standard API parameters; no tools; API snapshot and effort not reported
13 source model rowsKnowledgeDated paper cohortDisplay onlyUpdated March 10, 2026 paper revision
| Paper rank | Source model | F1 | Accuracy | Accuracy given attempted | Attempted | Hedged | F1 change from SimpleQA (pp) | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemini 2.5 ProGoogle | 55.6% | 55.3% | 55.9% | 98.9% | 1.1% | +0.5 | Not reported | Closed |
| 2 | GPT 5OpenAI | 52.3% | 50.9% | 53.8% | 94.6% | 5.4% | +1.8 | Not reported | Unknown |
| 3 | o3OpenAI | 51.9% | 51.6% | 52.0% | 99.3% | 0.7% | +1.9 | Not reported | Closed |
| 4 | GPT 4.1OpenAI | 39.9% | 39.8% | 40.1% | 99.3% | 0.7% | -1.0 | Not reported | Closed |
| 5 | GPT 4oOpenAI | 34.9% | 34.4% | 35.5% | 97.0% | 3.0% | -3.5* | Not reported | Closed |
| 6 | DeepSeek R1 (0528)DeepSeek | 33.3% | 32.7% | 33.9% | 96.4% | 3.6% | +1.4 | Not reported | Unknown |
| 7 | Claude Opus 4Anthropic | 28.3% | 19.2% | 54.1% | 35.5% | 64.5% | -4.0* | Not reported | Unknown |
| 8 | Gemini 2.5 FlashGoogle | 28.2% | 27.8% | 28.7% | 96.9% | 3.1% | -1.4 | Not reported | Closed |
| 9 | GPT 5 MiniOpenAI | 24.6% | 17.3% | 42.8% | 40.4% | 59.6% | +1.1 | Not reported | Closed |
| 10 | o4-miniOpenAI | 23.4% | 23.0% | 23.8% | 96.5% | 3.5% | +2.9* | Not reported | Unknown |
| 11 | Claude Sonnet 4Anthropic | 18.7% | 12.5% | 36.9% | 33.9% | 66.1% | -4.4* | Not reported | Closed |
| 12 | GPT 5 NanoOpenAI | 14.4% | 10.2% | 24.2% | 42.2% | 57.8% | +0.7 | Not reported | Closed |
| 13 | Gemini 2.5 Flash LiteGoogle | 11.1% | 10.2% | 12.1% | 84.0% | 16.0% | -0.4 | Not reported | Unknown |
All five metrics are percentages. The F1 change is in percentage points; * marks the paper's p < 0.05 result. Both benchmarks use the same revised autorater. Profile links identify the named model generation; the evaluated API snapshots and effort settings are not reported.
How we show SimpleQA Verified
We transcribed all 13 rows of Table 7 from the author paper, arXiv:2509.07968v2, revised March 10, 2026. That is the paper revision date; the evaluation date is not reported. The table uses standard API parameters with no tools and gpt-4.1-2025-04-14 as the autorater.
F1 is the harmonic mean of overall accuracy and accuracy given an attempted answer. We show those two metrics, attempted, and hedged separately: a model that declines many questions can have high accuracy on the answers it does attempt while getting fewer questions right overall.
The paper does not identify evaluated API snapshots, per-model reasoning effort, or generation parameters. Eight profile links refer to the named generation only. Five source rows retain their own identity because the catalog has no exact generation or would require guessing an effort or checkpoint. We keep this paper cohort display only and separate from the original SimpleQA and tool-enabled evaluations.
Snapshot
The published SimpleQA Verified snapshot places Gemini 2.5 Pro first at 55.6%. The third row is 3.7 points behind. The broader top-10 range is 32.2 points, so the table still separates the published systems.
13 source model rows have been evaluated on SimpleQA Verified. The benchmark falls in the Knowledge category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SimpleQA Verified is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SimpleQA Verified
Year
2025
Tasks
1,000 prompts
Format
Short-form factual answers
The paper uses gpt-4.1-2025-04-14 as its revised autorater. March 10, 2026 is the paper revision date, not an evaluation date. Source API snapshots and reasoning efforts are undisclosed. Results stay display only and do not enter overall or category scores.
Freshness and provenance
Version
arXiv:2509.07968v2, Table 7
Refresh cadence
Pinned paper revision
Staleness state
Dated paper cohort
Question availability
Public upstream; no task contents mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SimpleQA Verified measure?
SimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate.
Which model leads the published SimpleQA Verified snapshot?
Gemini 2.5 Pro currently leads the published SimpleQA Verified snapshot with 55.6% f1. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on SimpleQA Verified?
The March 10, 2026 paper revision snapshot contains 13 AI models.
Is F1 the same as accuracy?
No. F1 is the harmonic mean of accuracy over all prompts and accuracy on attempted answers. The table reports both accuracy columns separately, together with attempted and hedged percentages.
Are these scores comparable to the original SimpleQA or search-enabled results?
Keep them separate. SimpleQA Verified uses a curated 1,000-prompt set and a revised autorater. This paper cohort has no tools, and its generation settings and API snapshots are not fully disclosed.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.