Skip to main content
BenchLM
Data

ProgramBench: Can Language Models Rebuild Programs From Scratch? (ProgramBench)

We show this table for reference; we do not rank on it.

Data verified

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

Almost resolved rate on ProgramBench — Oct. 22, 2026

We mirror the published almost resolved rate view for ProgramBench. Claude Opus 5.5 leads the public snapshot at 70.5%, followed by Muse Wonder Full (60.0%) and Claude Fable 5.1 (55.5%). We do not use these results to rank models overall.

1Closed

Claude Opus 5.5

Anthropic

70.5%
Overall 86.4Context 1M
2Closed

Muse Wonder Full

Meta

60.0%
Overall 69.39Context 1M
3Closed

Claude Fable 5.1

Anthropic

55.5%
Overall 81.23Context 1M

30 modelsCodingCurrentDisplay onlyUpdated Oct. 22, 2026

Almost resolved rate table (30 models)

Almost resolved rate results for ProgramBench
RankModel / configurationScoreParameters (B)Open / closed
1Claude Opus 5.5Anthropic
70.5%
Not reportedClosed
2Muse Wonder FullMeta
60.0%
Not reportedClosed
3Claude Fable 5.1Anthropic
55.5%
Not reportedClosed
4GPT-6 AstraOpenAI
52.5%
Not reportedClosed
5Muse WonderMeta
50.5%
Not reportedClosed
6Claude Opus 5Anthropic
37.0%
Not reportedClosed
7Muse Spark 1.3Meta
25.0%
Not reportedClosed
8Muse Spark 1.3Meta
16.5%
Not reportedClosed
9Claude Opus 4.8Anthropic
16.5%
Not reportedClosed
10GPT-5.6 SolOpenAI
15.5%
Not reportedClosed
11GPT-5.5OpenAI
13.5%
Not reportedClosed
12GLM-5.2Z.AI
8.5%
Not reportedOpen
13Gemini 3.7 FlashGoogle
5.5%
Not reportedClosed
14GPT-5.5OpenAI
5.0%
Not reportedClosed
15Muse Spark 1.2Meta
4.5%
Not reportedClosed
16Claude Opus 4.7Anthropic
4.5%
Not reportedClosed
17Gemini 3.6 FlashGoogle
4.0%
Not reportedClosed
18Muse Spark 1.1Meta
4.0%
Not reportedClosed
19Gemini 3.5 FlashGoogle
3.0%
Not reportedClosed
20Claude Opus 4.7Anthropic
3.0%
Not reportedClosed
21GPT-5.6 SolOpenAI
2.5%
Not reportedClosed
22Claude Opus 4.6Anthropic
2.5%
Not reportedClosed
23GPT-5.5OpenAI
1.5%
Not reportedClosed
24Claude Sonnet 4.6Anthropic
1.0%
Not reportedClosed
25GPT-5.4OpenAI
0.0%
Not reportedClosed
26Gemini 3.1 ProGoogle
0.0%
Not reportedClosed
27Gemini 3 FlashGoogle
0.0%
Not reportedClosed
28Claude Haiku 4.5Anthropic
0.0%
Not reportedClosed
29GPT-5.4 miniOpenAI
0.0%
Not reportedClosed
30GPT-5 miniOpenAI
0.0%
Not reportedClosed

How ProgramBench is shown here

BenchLM mirrors the public ProgramBench leaderboard from Oct. 22, 2026. The official benchmark uses mini-SWE-agent on 200 cleanroom program-reconstruction tasks and reports both fully resolved and almost-resolved rates.

BenchLM displays the published almost-resolved rate as the visible score because the primary fully resolved metric is still very low and less differentiating. Almost resolved means the agent passed at least 95% of the hidden behavioral tests for a task. ProgramBench remains display only on BenchLM and is excluded from weighted coding and overall rankings until public model coverage broadens.

Snapshot

30 models200 tasks248,853 behavioral tests18.5% best fully resolvedDisplay only
Extended ProgramBench resultsProgramBench paperGitHub repositoryHugging Face datasetsLocalLLaMA launch discussion

The published ProgramBench snapshot places Claude Opus 5.5 first at 70.5%. The third row is 15.0 points behind. The broader top-10 range is 55.0 points, so the table still separates the published systems.

30 models have been evaluated on ProgramBench. The benchmark falls in the Coding category. ProgramBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About ProgramBench

Year

2026

Tasks

200 program reconstruction tasks

Format

Cleanroom executable reimplementation

Difficulty

Full-repository software architecture

ProgramBench turns open-source projects into cleanroom reconstruction tasks. Each task starts from an execute-only binary and usage documentation, with no source code, internet, decompilation, or prescribed skeleton. Evaluation uses hidden behavioral tests generated through agent-driven fuzzing. BenchLM shows ProgramBench as display-only because all current public rows are tied at 0% fully resolved and the visible score is the auxiliary almost-resolved metric.

ProgramBench: Can Language Models Rebuild Programs From Scratch?Public benchmark source

Freshness and provenance

Version

ProgramBench 2026

Refresh cadence

Quarterly

Staleness state

Current

Question availability

Public task metadata and hidden behavioral tests

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does ProgramBench measure?

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

Which model leads the published ProgramBench snapshot?

Claude Opus 5.5 currently leads the published ProgramBench snapshot with 70.5% almost resolved rate. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on ProgramBench?

The Oct. 22, 2026 snapshot contains 30 AI models.

Learn More

Read our explainer: ProgramBench benchmark deep dive
Last updated: Oct. 22, 2026 · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 6,100+ subscribers.

One email each week. Unsubscribe anytime.