# Massive Multitask Language Understanding (MMLU) > A comprehensive multiple-choice question answering test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Tests knowledge across diverse academic subjects from high school to professional level. Canonical page: https://benchlm.ai/benchmarks/mmlu - Category: [Knowledge](/knowledge) - Last updated: October 17, 2026 ## About MMLU - Year: 2020 - Tasks: 57 subjects - Format: Multiple choice questions - Difficulty: Elementary to professional level - Paper: [Measuring Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300) MMLU evaluates models on 57 subjects spanning humanities, social sciences, STEM, and other areas. Questions range from elementary to advanced professional level, making it a comprehensive test of world knowledge and reasoning ability. ## Benchmark definition Original MMLU is a four-choice benchmark covering 57 academic and professional tasks. Its subject mix ranges from elementary material to advanced domains, but it is now a saturated historical reference rather than a strong separator of current frontier models. **Variant check:** MMLU-Pro is a separate benchmark with more than 12,000 questions across 14 domains and ten choices per question. Its results are not interchangeable with scores stored under the original MMLU key. ## How to read this benchmark Use this table as a source receipt for original MMLU results. It does not expose a complete prompt and scoring setup for every row, so consult the original provider report before comparing results. Do not transfer MMLU-Pro's design or scores into this lane. **Provenance:** At the September 22, 2026 editorial check, the displayed values came from provider reports in the source ledger. The original repository's legacy leaderboard did not supply that table. **Source freshness:** The original paper was submitted on September 7, 2020 and published at ICLR 2021. The MMLU-Pro paper was last revised on November 6, 2024. Those source dates are separate from this page's data refresh date. **Honest limit:** MMLU is display-only here because the lane is stale and saturated. Small differences do not establish a current model winner without a controlled, like-for-like evaluation on a benchmark that still separates the systems being considered. ### Primary sources - [Original MMLU paper](https://arxiv.org/abs/2009.03300) - [Official MMLU repository](https://github.com/hendrycks/test) - [MMLU-Pro paper](https://arxiv.org/abs/2406.01574) ### Related BenchLM pages - [MMLU-Pro benchmark](/benchmarks/mmlu-pro) - [Knowledge ranking](/knowledge) - [Benchmark confidence](/benchmark-confidence) MMLU is displayed for reference and does not enter the active public ranking. ## Leaderboard (34 models) | Rank | Model | Creator | Score | |------|-------|---------|-------| | 1 | [o1](/models/o1) | OpenAI | 91.8% | | 2 | [DeepSeek-R1](/models/deepseek-r1) | DeepSeek | 90.8% | | 3 | [GPT-4.5](/models/gpt-4-5) | OpenAI | 90.8% | | 4 | [Sarvam 105B](/models/sarvam-105b) | Sarvam | 90.6% | | 5 | [GPT-4.1](/models/gpt-4-1) | OpenAI | 90.2% | | 6 | DeepSeek V4 Pro Base | DeepSeek | 90.1% | | 7 | [o4-mini](/models/o4-mini) | OpenAI | 90% | | 8 | [GPT-OSS 120B](/models/gpt-oss-120b) | OpenAI | 90% | | 9 | [Kimi K2](/models/kimi-k2) | Moonshot AI | 89.5% | | 10 | DeepSeek V4 Flash Base | DeepSeek | 88.7% | | 11 | [Grok 2](/models/grok-2) | xAI | 87.5% | | 12 | [GPT-4.1 mini](/models/gpt-4-1-mini) | OpenAI | 87.5% | | 13 | [Trinity-Large-Preview](/models/trinity-large-preview) | Arcee AI | 87.2% | | 14 | [o3-mini](/models/o3-mini) | OpenAI | 86.9% | | 15 | [Grok 2 mini](/models/grok-2-mini) | xAI | 86.2% | | 16 | [Llama 3.3 70B](/models/llama-3-3-70b) | Meta | 86% | | 17 | [Llama 3.1 70B](/models/llama-3-1-70b) | Meta | 86% | | 18 | [LongCat-Flash-Lite-Sparse](/models/longcat-flash-lite-sparse) | Meituan | 85.3% | | 19 | [GPT-OSS 20B](/models/gpt-oss-20b) | OpenAI | 85.3% | | 20 | [o1-mini](/models/o1-mini) | OpenAI | 85.2% | | 21 | [Sarvam 30B](/models/sarvam-30b) | Sarvam | 85.1% | | 22 | [Phi-4](/models/phi-4) | Microsoft | 84.8% | | 23 | [GPT-4o mini](/models/gpt-4o-mini) | OpenAI | 82% | | 24 | [Mistral Small 3.1](/models/mistral-small-3-1) | Mistral | 80.6% | | 25 | [Claude 3.5 Haiku](/models/claude-3-5-haiku) | Anthropic | 80.3% | | 26 | [GPT-4.1 nano](/models/gpt-4-1-nano) | OpenAI | 80.1% | | 27 | [Mixtral 8x22B Instruct v0.1](/models/mixtral-8x22b-instruct-v0-1) | Mistral | 77.8% | | 28 | [Gemma 2 27B](/models/gemma-2-27b) | Google | 76.2% | | 29 | [Llama 3.1 8B](/models/llama-3-1-8b) | Meta | 73% | | 30 | [Phi-3.5-mini](/models/phi-3-5-mini) | Microsoft | 69% | | 31 | [Granite-4.0-H-1B](/models/granite-4-0-h-1b) | IBM | 59.7% | | 32 | [Granite-4.0-1B](/models/granite-4-0-1b) | IBM | 59.4% | | 33 | [Granite-4.0-H-350M](/models/granite-4-0-h-350m) | IBM | 36.2% | | 34 | [Granite-4.0-350M](/models/granite-4-0-350m) | IBM | 35.0% | ## FAQ ### What does MMLU measure? Original MMLU tests knowledge and reasoning across 57 academic and professional tasks. Each question has four answer choices. The subjects range from elementary material to advanced professional domains, but the public lane is now saturated enough that BenchLM keeps these sourced rows as a display-only historical reference. ### Is MMLU-Pro the same benchmark as MMLU? MMLU-Pro is a separate benchmark, not a directly interchangeable score update. It contains more than 12,000 questions across 14 domains, expands each question from four choices to ten, and removes some noisy or trivial original items. Compare models within one named benchmark and evaluation setup. ### Does MMLU affect the current public ranking? No. The MMLU table is display-only and does not enter the active public ranking. Its rows remain useful as source receipts for historically important evaluations, but small differences should not decide a current model choice without a controlled, like-for-like run on a more discriminating benchmark. ## Compare Top Models on MMLU - [o1 vs DeepSeek-R1](/compare/deepseek-r1-vs-o1) - [DeepSeek-R1 vs GPT-4.5](/compare/deepseek-r1-vs-gpt-4-5) - [GPT-4.5 vs Sarvam 105B](/compare/gpt-4-5-vs-sarvam-105b) - [Sarvam 105B vs GPT-4.1](/compare/gpt-4-1-vs-sarvam-105b) ## Related Reading - [MMLU benchmark explainer](/blog/posts/mmlu-vs-mmlu-pro)