# CompileBench (building real projects from source, pass@1) (CompileBench)

> Quesma's CompileBench hands an agent the unmodified source of an open-source project such as curl, jq or GNU coreutils and asks for a working binary. Tasks include static and musl builds, cross-compiling to Windows and ARM64, and reviving code from 2003.

Canonical page: https://benchlm.ai/benchmarks/compilebench

- Category: [Coding](/coding)
- Last updated: November 13, 2026

## About CompileBench

- Year: 2025
- Tasks: 15 build tasks on cowsay, jq, GNU coreutils and curl, three attempts each (45 attempts)
- Format: Agent in a Docker Linux terminal builds a project from source; the binary is checked
- Difficulty: Dependency resolution, legacy toolchains, static linking and cross-compilation
- Paper: [CompileBench: can AI compile 22-year-old code? (Quesma)](https://quesma.com/blog/introducing-compilebench/)

The agent gets the source tree, an interactive Linux terminal in Docker and a build goal. It has to work out the build system, install or patch what is missing, pick compiler and linker flags, and produce the binary. A task-specific check then runs the result: that the version matches the source, that a static build has no dynamic libraries, that sha1sum returns the right hash. Those checks also catch shortcuts such as copying a system utility. There are 15 tasks built from cowsay, jq, GNU coreutils (version 9.7 and the 2003 release 5.0) and curl, and each is attempted three times. pass@1 is the share of the 45 attempts that succeed; pass@3 counts a task as solved if any of its three attempts does. We store pass@1 and give the pass@3 task count in the row note. Every attempt's transcript is published.

CompileBench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (16 models)

| Rank | Model | Creator | Score | Who ran it |
|------|-------|---------|-------|-----------|
| 1 | [Claude Sonnet 4.5](/models/claude-sonnet-4-5) | Anthropic | 86.7% | Run by CompileBench |
| 2 | [GPT-5 (high)](/models/gpt-5-high) | OpenAI | 86.7% | Run by CompileBench |
| 3 | [Claude Sonnet 4.5 Thinking](/models/claude-sonnet-4-5-thinking) | Anthropic | 84.4% | Run by CompileBench |
| 4 | [Claude Haiku 4.5 Thinking](/models/claude-haiku-4-5-thinking) | Anthropic | 73.3% | Run by CompileBench |
| 5 | [Grok 4](/models/grok-4) | xAI | 71.1% | Run by CompileBench |
| 6 | [Claude Haiku 4.5](/models/claude-haiku-4-5) | Anthropic | 68.9% | Run by CompileBench |
| 7 | [DeepSeek V3.1](/models/deepseek-v3-1) | DeepSeek | 64.4% | Run by CompileBench |
| 8 | [Grok Code Fast 1](/models/grok-code-fast-1) | xAI | 64.4% | Run by CompileBench |
| 9 | [Grok 4 Fast (Reasoning)](/models/grok-4-fast-reasoning) | xAI | 55.6% | Run by CompileBench |
| 10 | [Gemini 2.5 Pro](/models/gemini-2-5-pro) | Google | 53.3% | Run by CompileBench |
| 11 | [Qwen3 Max](/models/qwen3-max) | Alibaba | 53.3% | Run by CompileBench |
| 12 | [GPT-4.1](/models/gpt-4-1) | OpenAI | 53.3% | Run by CompileBench |
| 13 | [GLM-4.5](/models/glm-4-5) | Z.AI | 48.9% | Run by CompileBench |
| 14 | [GPT-4.1 mini](/models/gpt-4-1-mini) | OpenAI | 46.7% | Run by CompileBench |
| 15 | [GPT-OSS 120B](/models/gpt-oss-120b) | OpenAI | 42.2% | Run by CompileBench |
| 16 | [Gemini 2.5 Flash](/models/gemini-2-5-flash) | Google | 40.0% | Run by CompileBench |

## FAQ

### What does CompileBench measure?

Whether an agent can turn real source code into a working program. It gets the unmodified source of projects such as curl, jq and GNU coreutils plus a Linux terminal, and must sort out dependencies, toolchains and flags. The result only counts if the produced binary passes a functional check.

### Which model leads CompileBench?

On the October 15, 2025 leaderboard, Claude Sonnet 4 with a 16k thinking budget and GPT-5-Codex at high effort share the best pass@1, 91.1%. Of the models tracked here, GPT-5 at high effort and Claude Sonnet 4.5 are next at 86.7%, followed by Claude Sonnet 4.5 with thinking at 84.4%.

### What is the difference between pass@1 and pass@3?

Each task is attempted three times. pass@1 is the share of all attempts that succeed, so it rewards a model that builds the project reliably. pass@3 counts a task as solved if any one attempt works. The stored score is pass@1; the pass@3 task count is in each row note.

## Compare Top Models on CompileBench

- [Claude Sonnet 4.5 vs GPT-5 (high)](/compare/claude-sonnet-4-5-vs-gpt-5-high)
- [GPT-5 (high) vs Claude Sonnet 4.5 Thinking](/compare/claude-sonnet-4-5-thinking-vs-gpt-5-high)
- [Claude Sonnet 4.5 Thinking vs Claude Haiku 4.5 Thinking](/compare/claude-haiku-4-5-thinking-vs-claude-sonnet-4-5-thinking)
- [Claude Haiku 4.5 Thinking vs Grok 4](/compare/claude-haiku-4-5-thinking-vs-grok-4)