PMPP-Hard GPU kernel optimization (AUC, 0-10 hours) (PMPP-Hard)
We show this table for reference; we do not rank on it.
An agentic GPU kernel-optimization benchmark of 69 hard tasks, adapted by Thinking Machines Lab for consistent cross-model evaluation. Each model gets four independent attempts with a 10-hour budget, and the score is the area under the mean tasks-solved curve, out of a maximum of 690 task-hours.
Benchmark score on PMPP-Hard — November 13, 2026
We compile the PMPP-Hard rows from provider self-reports. Inkling 2 Small leads the table at 494.27 task-hours. We do not use these results to rank models overall.
1 modelCodingCurrentDisplay onlyUpdated November 13, 2026
Benchmark score table (1 model)
The one result on this page was reported by the lab that built the model.
| Rank | Model / configuration | Score | Who ran it | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Inkling 2 SmallThinking Machines LabReported by Thinking Machines Lab | 494.27 task-hours | Reported by Thinking Machines Lab | Not reported | Closed |
- Lab-reported
- Benchmark-run
- Independent
How to read this leaderboard
Operator receipt: 1 sourced row is currently displayable on this page; the leading published row is Inkling 2 Small at 494.27 task-hours.
Honest limit: Thinking Machines-run adaptation of an external benchmark, with task and grading fixes that are not published task by task.
About PMPP-Hard
Year
2026
Tasks
69 hard GPU kernel-optimization tasks
Format
Area under the mean tasks-solved curve over 10 hours (task-hours, max 690)
Difficulty
Long-horizon GPU kernel optimization
Thinking Machines introduced the adapted PMPP-Hard runs with Inkling 2 Small on November 5, 2026. Each model uses its maximum reasoning setting, the PI harness and a shared compaction policy; each submitted solution gets two independent LLM audits, with a third review on disagreement. Reported AUC (task-hours): GPT-6 Astra 616.39, Kimi K3.1 Pro 583.86, Claude Opus 5.5 499.65, DeepSeek V4.1 Flash 497.36, Inkling 2 Small 494.27, Mistral Large 4 478.39, Kimi K3 453.58, GLM 5.3 430.96 and GLM 5.2 389.12. BenchLM stores the Inkling 2 Small row; the other figures are Thinking Machines runs and are listed here for context.
Freshness and provenance
Version
PMPP-Hard 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does PMPP-Hard measure?
An agentic GPU kernel-optimization benchmark of 69 hard tasks, adapted by Thinking Machines Lab for consistent cross-model evaluation. Each model gets four independent attempts with a 10-hour budget, and the score is the area under the mean tasks-solved curve, out of a maximum of 690 task-hours.
Which model scores highest on PMPP-Hard?
Inkling 2 Small by Thinking Machines Lab currently leads with a score of 494.27 task-hours on PMPP-Hard.
How many models are evaluated on PMPP-Hard?
1 AI models have published results on PMPP-Hard in the BenchLM catalog.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.