Measuring Short-Form Factuality in Large Language Models (SimpleQA)
We show this table for reference; we do not rank on it.
A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.
Benchmark score on SimpleQA — October 29, 2026
We compile the SimpleQA rows from provider self-reports. GPT-4.5 leads the table at 62.5%, followed by Qwen3 235B 2507 (54.3%) and Grok 3 [Beta] (43.6%). We do not use these results to rank models overall.
GPT-4.5
OpenAI
Qwen3 235B 2507
Alibaba
Grok 3 [Beta]
xAI
14 modelsKnowledgeRefreshingDisplay onlyUpdated October 29, 2026
Benchmark score table (14 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | GPT-4.5OpenAI | 62.5% | Not reported | Closed |
| 2 | Qwen3 235B 2507Alibaba | 54.3% | Not reported | Open |
| 3 | Grok 3 [Beta]xAI | 43.6% | Not reported | Closed |
| 4 | Kimi K2Moonshot AI | 31% | Not reported | Closed |
| 5 | DeepSeek-R1DeepSeek | 30.1% | Not reported | Open |
| 6 | Gemini 2.0 FlashGoogle | 29.9% | Not reported | Closed |
| 7 | DeepSeek-R1-0528DeepSeek | 27.8% | Not reported | Open |
| 8 | DeepSeek V3DeepSeek | 24.9% | Not reported | Open |
| 9 | o4-miniOpenAI | 20.2% | Not reported | Closed |
| 10 | Gemini 2.0 Flash-LiteGoogle | 16.5% | Not reported | Closed |
| 11 | Gemini 2.5 Flash-LiteGoogle | 10.7% | Not reported | Closed |
| 12 | Mistral Small 3.1Mistral | 10.4% | Not reported | Open |
| 13 | Gemini 1.5 FlashGoogle | 8.6% | Not reported | Closed |
| 14 | o1-miniOpenAI | 7.6% | Not reported | Closed |
SimpleQA (Kaggle)
Native proportions are displayed as percentages. Source profiles, task versions, evaluation dates, and symmetric intervals stay attached to each row. Missing results are not zero. How Kaggle Benchmarks reruns tests.
| Source rank | Source model or configuration | Task score (%) | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Gemini 3.1 Pro Preview | 74.8% | Task version 7; Evaluated 2026-03-10Original result | Not reported | Unknown |
| 2 | GPT-6 Astra | 73.1% | Task version 7; Evaluated 2026-09-23Original result | Not reported | Unknown |
| 3 | Gemini 4 Argon | 71.2% | Task version 7; Evaluated 2026-10-09Original result | Not reported | Unknown |
| 4 | GPT-6.1 Sol | 71.1% | Task version 7; Evaluated 2026-10-05Original result | Not reported | Unknown |
| 5 | Gemini 3 Pro Preview | 70.5% | Task version 7; Evaluated 2025-11-13Original result | Not reported | Unknown |
| 6 | Gemini 3.8 Flash | 69.4% | Task version 7; Evaluated 2026-09-06Original result | Not reported | Unknown |
| 7 | Gemini 3.7 Flash | 69.0% | Task version 7; Evaluated 2026-08-14Original result | Not reported | Unknown |
| 8 | GPT-5.6 Sol | 67.7% | Task version 7; Evaluated 2026-07-14Original result | Not reported | Unknown |
| 9 | Gemini 3.5 Flash | 66.9% | Task version 7; Evaluated 2026-06-01Original result | Not reported | Unknown |
| 10 | Gemini 3.6 Flash | 65.6% | Task version 7; Evaluated 2026-07-22Original result | Not reported | Unknown |
| 11 | GPT-6 Sol | 63.2% | Task version 7; Evaluated 2026-09-25Original result | Not reported | Unknown |
| 12 | Claude Opus 5 | 62.2% | Task version 7; Evaluated 2026-07-28Original result | Not reported | Unknown |
| 13 | Grok 4.7 | 58.2% | Task version 7; Evaluated 2026-10-02Original result | Not reported | Unknown |
| 14 | Grok 4.5 | 51.7% | Task version 7; Evaluated 2026-07-23Original result | Not reported | Unknown |
| 15 | o3 | 50.5% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 16 | Qwen 3 235B A22B Thinking | 50.4% | Task version 7; Evaluated 2025-08-20Original result | Not reported | Unknown |
| 17 | Grok 4 | 50.3% | Task version 7; Evaluated 2025-09-03Original result | Not reported | Unknown |
| 18 | Grok 4.6 | 49.5% | Task version 7; Evaluated 2026-09-16Original result | Not reported | Unknown |
| 19 | Claude Opus 4.6 | 48.3% | Task version 7; Evaluated 2026-03-09Original result | Not reported | Unknown |
| 20 | Claude Opus 4.8 | 46.6% | Task version 7; Evaluated 2026-05-29Original result | Not reported | Unknown |
| 21 | Claude Sonnet 5.5 | 46.0% | Task version 7; Evaluated 2026-09-30Original result | Not reported | Unknown |
| 22 | Deepseek V4.1 Flash | 43.5% | Task version 7; Evaluated 2026-10-20Original result | Not reported | Unknown |
| 23 | Grok 3 | 41.3% | Task version 7; Evaluated 2025-05-30Original result | Not reported | Unknown |
| 24 | GPT-5.6 Terra | 39.7% | Task version 7; Evaluated 2026-07-15Original result | Not reported | Unknown |
| 25 | GPT-5.6 Luna | 37.9% | Task version 7; Evaluated 2026-07-15Original result | Not reported | Unknown |
| 26 | Gemini 3.5 Flash-Lite | 34.4% | Task version 7; Evaluated 2026-07-23Original result | Not reported | Unknown |
| 27 | Gemini 2.5 Flash | 33.7% | Task version 7; Evaluated 2026-03-10Original result | Not reported | Unknown |
| 28 | GPT-5.4 | 32.7% | Task version 7; Evaluated 2026-03-10Original result | Not reported | Unknown |
| 29 | Claude Sonnet 4.6 | 32.1% | Task version 7; Evaluated 2026-03-09Original result | Not reported | Unknown |
| 30 | DeepSeek-R1 | 31.9% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 31 | DeepSeek-V3 | 29.5% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 32 | deepseek-v3.1 | 26.2% | Task version 7; Evaluated 2025-08-27Original result | Not reported | Unknown |
| 33 | Mistral Large 2 | 24.2% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 34 | Grok 2 | 23.6% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 35 | Claude Haiku 5.5 | 21.7% | Task version 7; Evaluated 2026-10-09Original result | Not reported | Unknown |
| 36 | o4 mini | 20.5% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 37 | Grok 3 Mini | 20.4% | Task version 7; Evaluated 2025-05-30Original result | Not reported | Unknown |
| 38 | Mixtral 8x22B | 19.3% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 39 | gpt-oss-120b | 12.6% | Task version 7; Evaluated 2025-08-20Original result | Not reported | Unknown |
| 40 | Gemma 3 27B | 10.0% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 41 | Gemma 4 31B | 9.3% | Task version 7; Evaluated 2026-06-01Original result | Not reported | Unknown |
| 42 | Granite 4.0 Small | 8.6% | Task version 7; Evaluated 2025-09-30Original result | Not reported | Unknown |
| 43 | Gemma 4 26B A4B | 8.3% | Task version 7; Evaluated 2026-06-01Original result | Not reported | Unknown |
| 44 | Gemma 3 12B | 6.1% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 45 | Ministral 8B | 5.9% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 46 | gpt-oss-20b | 5.5% | Task version 7; Evaluated 2025-08-20Original result | Not reported | Unknown |
| 47 | Codestral | 5.3% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 48 | Ministral 3B | 4.8% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
| 49 | Gemma 3 4B | 3.8% | Task version 7; Evaluated 2025-05-29Original result | Not reported | Unknown |
Among the reported SimpleQA rows, GPT-4.5 is first at 62.5%. The third row is 18.9 points behind. The broader top-10 range is 46.0 points, so the table still separates the published systems.
14 models have been evaluated on SimpleQA. The benchmark falls in the Knowledge category. SimpleQA is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SimpleQA
Year
2024
Tasks
Factual questions
Format
Short-form Q&A
Difficulty
Factual accuracy focused
SimpleQA prioritizes two key properties: questions should have short, factual answers that can be easily verified, and questions should be diverse and challenging. It serves as a crucial test of factual knowledge and accuracy.
Freshness and provenance
Version
SimpleQA 2024
Refresh cadence
Annual
Staleness state
Refreshing
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SimpleQA measure?
A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.
Which model scores highest on SimpleQA?
GPT-4.5 by OpenAI currently leads with a score of 62.5% on SimpleQA.
How many models are evaluated on SimpleQA?
14 AI models have published results on SimpleQA in the BenchLM catalog.
Compare top models on SimpleQA
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.