A set of 164 handwritten programming problems that test the ability to generate correct Python functions from natural language descriptions. Each problem includes function signature, docstring, body, and several unit tests.
BenchLM mirrors the published score view for HumanEval. DeepSeek V4 Pro Base leads the public snapshot at 76.8% , followed by DeepSeek V4 Flash Base (69.5%). BenchLM does not use these results to rank models overall.
DeepSeek V4 Pro Base
DeepSeek
DeepSeek V4 Flash Base
DeepSeek
Year
2021
Tasks
164 problems
Format
Python function generation
Difficulty
Introductory to intermediate programming
HumanEval measures functional correctness for synthesizing programs from docstrings. It focuses on whether generated code actually works correctly rather than just looking syntactically correct. Problems range from simple string manipulation to more complex algorithmic challenges.
Version
HumanEval
Refresh cadence
Static
Staleness state
Stale
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
A set of 164 handwritten programming problems that test the ability to generate correct Python functions from natural language descriptions. Each problem includes function signature, docstring, body, and several unit tests.
DeepSeek V4 Pro Base by DeepSeek currently leads with a score of 76.8% on HumanEval.
2 AI models have been evaluated on HumanEval on BenchLM.
For engineers, researchers, and the plain curious — a weekly brief on new models, ranking shifts, and pricing changes.
Free. No spam. Unsubscribe anytime.