# Evaluating Large Language Models Trained on Code (HumanEval)

> HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.

HumanEval measures Python function correctness from a docstring and function signature. In pass@1, one sampled solution must pass the task's unit tests. Use this table to inspect sourced reports; use repository benchmarks and your own failures to choose a coding model. [Original HumanEval paper](https://arxiv.org/abs/2107.03374).

Canonical page: https://benchlm.ai/benchmarks/humaneval

- Category: [Coding](/coding)
- Last updated: October 17, 2026

## About HumanEval

- Year: 2021
- Tasks: 164 problems
- Format: Python function generation
- Difficulty: Introductory to intermediate programming
- Paper: [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374)

HumanEval measures functional correctness for programs synthesized from docstrings. The tasks range from string manipulation to intermediate algorithms, but they do not test repository navigation, multi-file changes, tool use, or long-running agent work.

## What a HumanEval score measures

| Measure | Meaning |
|---|---|
| Task set | 164 handwritten Python problems, each with a function signature, docstring, and unit tests. |
| pass@1 | The share of tasks solved by one sampled solution, estimated with the source's sampling procedure. |
| pass@k | The chance that at least one of k sampled solutions passes. Match k, sampling settings, and the test harness before comparing reports. |
| Coverage limit | Function correctness does not measure multi-file changes, repository navigation, or the tools around a coding model. |

Method source: [HumanEval paper, dataset, and pass@k method](https://arxiv.org/abs/2107.03374).

HumanEval is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (22 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [o4-mini](/models/o4-mini) | OpenAI | 97.3% |
| 2 | [BTL-3](/models/btl-3) | Bad Theory Labs | 95.1% |
| 3 | [o1-mini](/models/o1-mini) | OpenAI | 92.4% |
| 4 | [Sarvam 30B](/models/sarvam-30b) | Sarvam | 92.1% |
| 5 | [Qwen2.5-VL-32B](/models/qwen2-5-vl-32b) | Alibaba | 91.5% |
| 6 | [DeepSeek-V2.5](/models/deepseek-v2-5) | DeepSeek | 89% |
| 7 | [GPT-4.5](/models/gpt-4-5) | OpenAI | 88.6% |
| 8 | [Mistral Small 3.1](/models/mistral-small-3-1) | Mistral | 88.4% |
| 9 | [Grok 2](/models/grok-2) | xAI | 88.4% |
| 10 | [Llama 3.3 70B](/models/llama-3-3-70b) | Meta | 88.4% |
| 11 | [Claude 3.5 Haiku](/models/claude-3-5-haiku) | Anthropic | 88.1% |
| 12 | [GPT-4o mini](/models/gpt-4o-mini) | OpenAI | 87.2% |
| 13 | [Grok 2 mini](/models/grok-2-mini) | xAI | 85.7% |
| 14 | [Phi-4](/models/phi-4) | Microsoft | 82.6% |
| 15 | [Llama 3.1 70B](/models/llama-3-1-70b) | Meta | 80.5% |
| 16 | [Granite-4.0-1B](/models/granite-4-0-1b) | IBM | 74% |
| 17 | [Soofi S 30B-A3B](/models/soofi-s-30b-a3b) | Soofi Project | 73.8% |
| 18 | [Granite-4.0-H-1B](/models/granite-4-0-h-1b) | IBM | 73% |
| 19 | [Llama 3.1 8B](/models/llama-3-1-8b) | Meta | 72.6% |
| 20 | [Phi-3.5-mini](/models/phi-3-5-mini) | Microsoft | 62.8% |
| 21 | [Granite-4.0-350M](/models/granite-4-0-350m) | IBM | 39% |
| 22 | [Granite-4.0-H-350M](/models/granite-4-0-h-350m) | IBM | 38% |

## FAQ

### Can HumanEval choose a coding agent?

HumanEval tests Python functions from a supplied signature and docstring. A coding agent also needs to find the right files, use tools, make compatible changes, and recover from failures. Use HumanEval as one function-level check, then compare repository evidence and accepted changes on your own codebase.

### Why is a model missing from this HumanEval table?

This table includes only results with displayable source records. A missing row means we cannot publish a sourced score here; it does not mean the model scored zero. Check the linked model page for its other coding evidence, and match the sampling settings and test harness before comparing reports.

## Compare Top Models on HumanEval

- [o4-mini vs BTL-3](/compare/btl-3-vs-o4-mini)
- [BTL-3 vs o1-mini](/compare/btl-3-vs-o1-mini)
- [o1-mini vs Sarvam 30B](/compare/o1-mini-vs-sarvam-30b)
- [Sarvam 30B vs Qwen2.5-VL-32B](/compare/qwen2-5-vl-32b-vs-sarvam-30b)

## Related Reading

- [HumanEval benchmark explainer](/blog/posts/what-is-humaneval-coding-benchmark)