Benchmark profile
ProgramBench: Can Language Models Rebuild Programs From Scratch? (ProgramBench)
A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.
Data verifiedHow BenchLM shows ProgramBench
BenchLM mirrors the public ProgramBench leaderboard from Jul. 24, 2026. The official benchmark uses mini-SWE-agent on 200 cleanroom program-reconstruction tasks and reports both fully resolved and almost-resolved rates.
BenchLM displays the published almost-resolved rate as the visible score because the primary fully resolved metric is still very low and less differentiating. Almost resolved means the agent passed at least 95% of the hidden behavioral tests for a task. ProgramBench remains display only on BenchLM and is excluded from weighted coding and overall rankings until public model coverage broadens.
Almost resolved rate on ProgramBench — Jul. 24, 2026
BenchLM mirrors the published almost resolved rate view for ProgramBench. Claude Opus 4.8 leads the public snapshot at 16.5% , followed by GPT-5.5 (13.5%) and GLM-5.2 (8.5%). BenchLM does not use these results to rank models overall.
Claude Opus 4.8
Anthropic
claude-opus-4-8-xhigh-programbench
GPT-5.5
OpenAI
gpt-5-5-xhigh-programbench
GLM-5.2
Z.AI
glm-5-2-programbench
Almost resolved rate table (15 models)
ScoreThe published ProgramBench snapshot places Claude Opus 4.8 first at 16.5%. The third row is 8.0 points behind. The broader top-10 range is 16.5 points, so the table still separates the published systems.
15 models have been evaluated on ProgramBench. The benchmark falls in the Coding category. This category carries a 20% weight in BenchLM.ai's overall scoring system. ProgramBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About ProgramBench
Year
2026
Tasks
200 program reconstruction tasks
Format
Cleanroom executable reimplementation
Difficulty
Full-repository software architecture
ProgramBench turns open-source projects into cleanroom reconstruction tasks. Each task starts from an execute-only binary and usage documentation, with no source code, internet, decompilation, or prescribed skeleton. Evaluation uses hidden behavioral tests generated through agent-driven fuzzing. BenchLM shows ProgramBench as display-only because all current public rows are tied at 0% fully resolved and the visible score is the auxiliary almost-resolved metric.
BenchLM freshness & provenance
Version
ProgramBench 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public task metadata and hidden behavioral tests
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
FAQ
What does ProgramBench measure?
A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.
Which model leads the published ProgramBench snapshot?
Claude Opus 4.8 currently leads the published ProgramBench snapshot with 16.5% almost resolved rate. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on ProgramBench?
15 AI models are included in BenchLM's mirrored ProgramBench snapshot, based on the public leaderboard captured on Jul. 24, 2026.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 2,000+ readers.
One email each week. Unsubscribe anytime.