# Measuring Short-Form Factuality in Large Language Models (SimpleQA)

> A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.

Canonical page: https://benchlm.ai/benchmarks/simpleqa

- Category: [Knowledge](/knowledge)
- Last updated: September 10, 2026

## About SimpleQA

- Year: 2024
- Tasks: Factual questions
- Format: Short-form Q&A
- Difficulty: Factual accuracy focused
- Paper: [Measuring short-form factuality in large language models](https://arxiv.org/abs/2411.04368)

SimpleQA prioritizes two key properties: questions should have short, factual answers that can be easily verified, and questions should be diverse and challenging. It serves as a crucial test of factual knowledge and accuracy.

SimpleQA is currently weighted in BenchLM's scoring formula. The Knowledge category carries 12% of the overall score, and SimpleQA contributes 5% of that category score.

## Leaderboard (3 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [DeepSeek V4 Pro 0813](/models/deepseek-v4-pro-0813) | DeepSeek | 57.9% |
| 2 | [DeepSeek V4 Flash 0731](/models/deepseek-v4-flash-0731) | DeepSeek | 34.1% |
| 3 | [MAI-Thinking-1](/models/mai-thinking-1) | Microsoft | 31% |

## FAQ

### What does SimpleQA measure?

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.

### Which model scores highest on SimpleQA?

DeepSeek V4 Pro 0813 by DeepSeek currently leads with a score of 57.9% on SimpleQA.

### How many models are evaluated on SimpleQA?

3 AI models have been evaluated on SimpleQA on BenchLM.

### Does SimpleQA affect BenchLM's overall score?

Yes. SimpleQA is a weighted benchmark inside the Knowledge category, which carries 12% of BenchLM's overall score. SimpleQA itself contributes 5% of that category score.

## Compare Top Models on SimpleQA

- [DeepSeek V4 Pro 0813 vs DeepSeek V4 Flash 0731](/compare/deepseek-v4-flash-0731-vs-deepseek-v4-pro-0813)
- [DeepSeek V4 Flash 0731 vs MAI-Thinking-1](/compare/deepseek-v4-flash-0731-vs-mai-thinking-1)