CompileBench (building real projects from source, pass@1) (CompileBench)
We show this table for reference; we do not rank on it.
Quesma's CompileBench hands an agent the unmodified source of an open-source project such as curl, jq or GNU coreutils and asks for a working binary. Tasks include static and musl builds, cross-compiling to Windows and ARM64, and reviving code from 2003.
Benchmark score on CompileBench — November 13, 2026
We compile the CompileBench rows from benchmark-owner or independent runs. GPT-5 (high) leads the table at 86.7%, followed by Claude Sonnet 4.5 (86.7%) and Claude Sonnet 4.5 Thinking (84.4%). We do not use these results to rank models overall.
GPT-5 (high)
OpenAI
Run by CompileBenchClaude Sonnet 4.5
Anthropic
Run by CompileBenchClaude Sonnet 4.5 Thinking
Anthropic
Run by CompileBench17 modelsCodingCurrentDisplay onlyUpdated November 13, 2026
Benchmark score table (17 models)
| Rank | Model / configuration | Score | Who ran it | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | GPT-5 (high)OpenAIRun by CompileBench | 86.7% | Run by CompileBench | Not reported | Closed |
| 2 | Claude Sonnet 4.5AnthropicRun by CompileBench | 86.7% | Run by CompileBench | Not reported | Closed |
| 3 | Claude Sonnet 4.5 ThinkingAnthropicRun by CompileBench | 84.4% | Run by CompileBench | Not reported | Closed |
| 4 | GPT-5 miniOpenAIRun by CompileBench | 75.6% | Run by CompileBench | Not reported | Closed |
| 5 | Claude Haiku 4.5 ThinkingAnthropicRun by CompileBench | 73.3% | Run by CompileBench | Not reported | Closed |
| 6 | Grok 4xAIRun by CompileBench | 71.1% | Run by CompileBench | Not reported | Closed |
| 7 | Claude Haiku 4.5AnthropicRun by CompileBench | 68.9% | Run by CompileBench | Not reported | Closed |
| 8 | Grok Code Fast 1xAIRun by CompileBench | 64.4% | Run by CompileBench | Not reported | Closed |
| 9 | DeepSeek V3.1DeepSeekRun by CompileBench | 64.4% | Run by CompileBench | Not reported | Open |
| 10 | Grok 4 Fast (Reasoning)xAIRun by CompileBench | 55.6% | Run by CompileBench | Not reported | Closed |
| 11 | Gemini 2.5 ProGoogleRun by CompileBench | 53.3% | Run by CompileBench | Not reported | Closed |
| 12 | GPT-4.1OpenAIRun by CompileBench | 53.3% | Run by CompileBench | Not reported | Closed |
| 13 | Qwen3 MaxAlibabaRun by CompileBench | 53.3% | Run by CompileBench | Not reported | Closed |
| 14 | GLM-4.5Z.AIRun by CompileBench | 48.9% | Run by CompileBench | Not reported | Closed |
| 15 | GPT-4.1 miniOpenAIRun by CompileBench | 46.7% | Run by CompileBench | Not reported | Closed |
| 16 | GPT-OSS 120BOpenAIRun by CompileBench | 42.2% | Run by CompileBench | Not reported | Open |
| 17 | Gemini 2.5 FlashGoogleRun by CompileBench | 40.0% | Run by CompileBench | Not reported | Closed |
- Lab-reported
- Benchmark-run
- Independent
How to read this leaderboard
Higher is better. Among the models we track, GPT-5 at high effort and Claude Sonnet 4.5 lead at 86.7% pass@1. pass@1 and pass@3 can order models differently: Claude Opus 4.1 with thinking is the only configuration that solved all 15 tasks within three attempts, yet its pass@1 is 80%. The hardest task, a static ARM64 cross-compile of curl, succeeded in 1% of first attempts across the field.
Operator receipt: 17 sourced rows are currently displayable on this page; the leading published row is GPT-5 (high) at 86.7%.
Honest limit: The leaderboard was last updated on October 15, 2025 and has no model released after that, so it is a record of that generation. Several of its rows are effort or thinking-budget variants; we map each to the matching model here and leave out the ones we don't list, including the two configurations with the highest pass@1. Fifteen tasks is a small set, and four projects supply all of them. We show it as display-only; it does not enter overall or category rankings.
Among the reported CompileBench rows, GPT-5 (high) is first at 86.7%. The third row is 2.3 points behind. The broader top-10 range is 31.1 points, so the table still separates the published systems.
17 models have been evaluated on CompileBench. The benchmark falls in the Coding category. We keep these results separate from the weighted global scoring system, so they remain source-specific evidence. CompileBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About CompileBench
Year
2025
Tasks
15 build tasks on cowsay, jq, GNU coreutils and curl, three attempts each (45 attempts)
Format
Agent in a Docker Linux terminal builds a project from source; the binary is checked
Difficulty
Dependency resolution, legacy toolchains, static linking and cross-compilation
The agent gets the source tree, an interactive Linux terminal in Docker and a build goal. It has to work out the build system, install or patch what is missing, pick compiler and linker flags, and produce the binary. A task-specific check then runs the result: that the version matches the source, that a static build has no dynamic libraries, that sha1sum returns the right hash. Those checks also catch shortcuts such as copying a system utility. There are 15 tasks built from cowsay, jq, GNU coreutils (version 9.7 and the 2003 release 5.0) and curl, and each is attempted three times. pass@1 is the share of the 45 attempts that succeed; pass@3 counts a task as solved if any of its three attempts does. We store pass@1 and give the pass@3 task count in the row note. Every attempt's transcript is published.
Freshness and provenance
Version
CompileBench 2025
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does CompileBench measure?
Whether an agent can turn real source code into a working program. It gets the unmodified source of projects such as curl, jq and GNU coreutils plus a Linux terminal, and must sort out dependencies, toolchains and flags. The result only counts if the produced binary passes a functional check.
Which model leads CompileBench?
On the October 15, 2025 leaderboard, Claude Sonnet 4 with a 16k thinking budget and GPT-5-Codex at high effort share the best pass@1, 91.1%. Of the models tracked here, GPT-5 at high effort and Claude Sonnet 4.5 are next at 86.7%, followed by Claude Sonnet 4.5 with thinking at 84.4%.
What is the difference between pass@1 and pass@3?
Each task is attempted three times. pass@1 is the share of all attempts that succeed, so it rewards a model that builds the project reliably. pass@3 counts a task as solved if any one attempt works. The stored score is pass@1; the pass@3 task count is in each row note.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.