Evaluating Large Language Models Trained on Code (HumanEval)
HumanEval measures Python function correctness from a docstring and function signature. In pass@1, one sampled solution must pass the task's unit tests. Use this table to inspect sourced reports; use repository benchmarks and your own failures to choose a coding model. Original HumanEval paper.
We show this table for reference; we do not rank on it.
HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.
What a HumanEval score measures
| Measurement | How to read it |
|---|---|
| Task set | 164 handwritten Python problems, each with a function signature, docstring, and unit tests. |
| pass@1 | The share of tasks solved by one sampled solution, estimated with the source's sampling procedure. |
| pass@k | The chance that at least one of k sampled solutions passes. Match k, sampling settings, and the test harness before comparing reports. |
| Coverage limit | Function correctness does not measure multi-file changes, repository navigation, or the tools around a coding model. |
Method source: HumanEval paper, dataset, and pass@k method.
Benchmark score on HumanEval — October 17, 2026
We compile the HumanEval rows from provider self-reports and secondary reports. The table contains 24 models. o4-mini has the highest published score at 97.3%, but the available coverage and evaluation setups do not establish a market leader. We do not use these results to rank models overall.
o4-mini
OpenAI
BTL-3
Bad Theory Labs
o1-mini
OpenAI
24 modelsCodingStaleSaturatedDisplay onlyUpdated October 17, 2026
Benchmark score table (24 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | o4-miniOpenAI | 97.3% | Not reported | Closed |
| 2 | BTL-3Bad Theory Labs | 95.1% | Not reported | Open |
| 3 | o1-miniOpenAI | 92.4% | Not reported | Closed |
| 4 | Sarvam 30BSarvam | 92.1% | Not reported | Open |
| 5 | Qwen2.5-VL-32BAlibaba | 91.5% | Not reported | Open |
| 6 | DeepSeek-V2.5DeepSeek | 89% | Not reported | Open |
| 7 | GPT-4.5OpenAI | 88.6% | Not reported | Closed |
| 8 | Mistral Small 3.1Mistral | 88.4% | Not reported | Open |
| 9 | Grok 2xAI | 88.4% | Not reported | Closed |
| 10 | Llama 3.3 70BMeta | 88.4% | Not reported | Open |
| 11 | Claude 3.5 HaikuAnthropic | 88.1% | Not reported | Closed |
| 12 | GPT-4o miniOpenAI | 87.2% | Not reported | Closed |
| 13 | Grok 2 minixAI | 85.7% | Not reported | Closed |
| 14 | Phi-4Microsoft | 82.6% | Not reported | Open |
| 15 | Llama 3.1 70BMeta | 80.5% | Not reported | Open |
| 16 | DeepSeek V4 Pro BaseDeepSeek | 76.8% | Not reported | Open |
| 17 | Granite-4.0-1BIBM | 74% | Not reported | Open |
| 18 | Soofi S 30B-A3BSoofi Project | 73.8% | Not reported | Open |
| 19 | Granite-4.0-H-1BIBM | 73% | Not reported | Open |
| 20 | Llama 3.1 8BMeta | 72.6% | Not reported | Open |
| 21 | DeepSeek V4 Flash BaseDeepSeek | 69.5% | Not reported | Open |
| 22 | Phi-3.5-miniMicrosoft | 62.8% | Not reported | Open |
| 23 | Granite-4.0-350MIBM | 39% | Not reported | Open |
| 24 | Granite-4.0-H-350MIBM | 38% | Not reported | Open |
How to read this leaderboard
Read this page as a source-coverage receipt for the exact rows BenchLM can currently display, not as a ranking of the best coding models. Use the calibrated coding leaderboard and repository benchmarks for a broader decision.
Operator receipt: 24 sourced rows are currently displayable on this page; the highest published score among these 24 models is o4-mini at 97.3%, which does not establish a market leader.
Honest limit: Only rows with displayable source records appear. Missing model rows are missing evidence, not zero scores. Different sampling settings or test harnesses can change the result, and this limited table does not establish an overall coding winner.
Among the reported HumanEval rows, o4-mini is first at 97.3%. The third row is 4.9 points behind. The broader top-10 range is 8.9 points, so many of the published results sit in a relatively narrow band.
24 models have been evaluated on HumanEval. The benchmark falls in the Coding category. HumanEval is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About HumanEval
Year
2021
Tasks
164 problems
Format
Python function generation
Difficulty
Introductory to intermediate programming
HumanEval measures functional correctness for programs synthesized from docstrings. The tasks range from string manipulation to intermediate algorithms, but they do not test repository navigation, multi-file changes, tool use, or long-running agent work.
Freshness and provenance
Version
HumanEval
Refresh cadence
Static
Staleness state
Stale
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
Can HumanEval choose a coding agent?
HumanEval tests Python functions from a supplied signature and docstring. A coding agent also needs to find the right files, use tools, make compatible changes, and recover from failures. Use HumanEval as one function-level check, then compare repository evidence and accepted changes on your own codebase.
Why is a model missing from this HumanEval table?
This table includes only results with displayable source records. A missing row means we cannot publish a sourced score here; it does not mean the model scored zero. Check the linked model page for its other coding evidence, and match the sampling settings and test harness before comparing reports.
Compare top models on HumanEval
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.