# OpenWorldEngineer-Bench

> Mercor's benchmark of AI agents on 26 long, open-ended engineering tasks in science, software, ML and engineering, built by domain experts. Each model makes 8 attempts per task, of 4 to 18 hours each, and the score is the mean task score on a scale from the starting point (0) to the best result the task allows (100).

Canonical page: https://benchlm.ai/benchmarks/openworldengineer-bench

- Category: [Agentic](/agentic)
- Last updated: Release 1.0 preview, Mercor data of November 11, 2026

## About OpenWorldEngineer-Bench

- Year: 2026
- Tasks: 26 expert-built engineering tasks in 4 domains (science, software, ML, engineering)
- Format: 8 attempts per task in the Harbor agent harness, 4 to 18 hours each, web search over an October 1, 2026 copy of the web
- Difficulty: Long-horizon engineering work: kernels, data systems, scientific forecasting, CAD and structural and aircraft design
- Paper: [OpenWorldEngineer-Bench](https://openworldengineer.mercor.com/)

Each task gives an agent a container with its hardware (up to a B300 or H100 GPU), an instruction, starting code and development cases, and asks for a measurable engineering result: faster GPU kernels and interpreters, a SQL change replayer or Parquet lakehouse, earthquake relocation and storm-surge forecasts, CAD models rebuilt from drawings or scans, a reinforced-concrete frame, a small aircraft. Attempts run in the Harbor harness with web search over a copy of the web frozen on October 1, 2026, and use about 124 million tokens on average. Each task's result is placed on its own scale, from the untouched baseline (0) to the ceiling its metric allows (100). A model's task score is the mean of its graded attempts, and its benchmark score is the mean over tasks, with a 95% interval of 1.96 standard errors across tasks. Sixteen tasks ship a public grader with a public test split; ten keep a private grader because it cannot be separated from hidden test data or, for Design/Build/Fly Aircraft, a private flight simulator. Mercor labels the release 1.0 preview and re-ran the aircraft task on November 10 and 11, 2026 after patching it. BenchLM recomputes the scores from the attempt-level data on Mercor's site and keeps each task's mean and best attempt in its history.

OpenWorldEngineer-Bench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (14 models)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Claude Fable 5.2](/models/claude-fable-5-2) | max reasoning · 95% interval 46.1 to 63.0 · best attempt 68.3 · $105.08 and 8.7 h per attempt | Anthropic | 54.6% |
| 2 | [GPT-6 Astra](/models/gpt-6-astra) | max reasoning · 95% interval 36.5 to 51.9 · best attempt 58.3 · $234.63 and 9.3 h per attempt | OpenAI | 44.2% |
| 3 | [Claude Opus 5.5](/models/claude-opus-5-5) | max reasoning · 95% interval 36.0 to 47.1 · best attempt 58.4 · $75.70 and 7.9 h per attempt | Anthropic | 41.6% |
| 4 | [GPT-6.1 Sol](/models/gpt-6-1-sol) | max reasoning · 95% interval 34.3 to 48.0 · best attempt 57.1 · $52.12 and 9.4 h per attempt | OpenAI | 41.2% |
| 5 | [Claude Sonnet 5.5](/models/claude-sonnet-5-5) | max reasoning · 95% interval 30.8 to 47.2 · best attempt 52.7 · $38.65 and 8.4 h per attempt | Anthropic | 39.0% |
| 6 | [Gemini 4 Carbon](/models/gemini-4-carbon) | high reasoning · 95% interval 27.2 to 38.3 · best attempt 51.0 · $62.08 and 9.0 h per attempt | Google | 32.8% |
| 7 | [Claude Fable 5.1](/models/claude-fable-5-1) | max reasoning · 95% interval 24.1 to 34.1 · best attempt 45.6 · $87.70 and 6.6 h per attempt | Anthropic | 29.1% |
| 8 | [Gemini 4 Argon](/models/gemini-4-argon) | high reasoning · 95% interval 18.3 to 29.9 · best attempt 42.2 · $49.44 and 7.6 h per attempt | Google | 24.1% |
| 9 | [Kimi K3.1](https://openworldengineer.mercor.com/) | max reasoning · 95% interval 9.8 to 17.4 · best attempt 20.6 · $21.16 and 5.6 h per attempt | Moonshot AI | 13.6% |
| 10 | [DeepSeek V4.1 Flash 1017](/models/deepseek-v4-1-flash) | max reasoning · 95% interval 9.3 to 15.4 · best attempt 23.4 · $1.37 and 5.5 h per attempt | DeepSeek | 12.4% |
| 11 | [GLM-5.3](/models/glm-5-3) | max reasoning · 95% interval 8.4 to 15.3 · best attempt 21.0 · $24.83 and 7.2 h per attempt | Z.AI | 11.8% |
| 12 | [Grok 4.7](/models/grok-4-7) | max reasoning · 95% interval 4.4 to 9.6 · best attempt 13.8 · $146.28 and 6.4 h per attempt | xAI | 7.0% |
| 13 | [Beam 1.0 501B-A23B](/models/reflection-beam) | max reasoning · 95% interval 2.1 to 4.5 · best attempt 8.9 · $1.47 and 2.8 h per attempt | Reflection AI | 3.3% |
| 14 | [Inkling](/models/inkling) | max reasoning · 95% interval 0.5 to 1.8 · best attempt 3.8 · $3.26 and 0.6 h per attempt | Thinking Machines Lab | 1.1% |

## OpenWorldEngineer DBF Aircraft

Source snapshot: Release 1.0 preview, Mercor data of November 11, 2026.

BenchLM mirrors the mean of 8 attempts on the Design/Build/Fly Aircraft task from the data Mercor publishes. It is excluded from BenchLM overall and category rankings.

- [Design/Build/Fly Aircraft task (Mercor)](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design)

| Source model | Configuration | Mean of 8 attempts |
|---|---|---|
| [GPT-6 Astra](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 94.1 | 71.5% |
| [Claude Fable 5.2](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 100.0 | 71.4% |
| [GPT-6.1 Sol](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 78.4 | 57.1% |
| [Claude Opus 5.5](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 61.8 | 42.5% |
| [Gemini 4 Carbon](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 78.9 | 38.4% |
| [Claude Sonnet 5.5](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 56.8 | 34.5% |
| [Gemini 4 Argon](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 46.8 | 28.6% |
| [Claude Fable 5.1](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 67.5 | 22.3% |
| [GLM-5.3](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 30.0 | 5.6% |
| [DeepSeek V4.1 Flash](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 14.9 | 4.1% |
| [Kimi K3.1](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 11.9 | 3.3% |
| [Inkling](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 6.3 | 1.4% |
| [Beam 1.0](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 4.0 | 1.0% |
| [Grok 4.7](https://openworldengineer.mercor.com/tasks/dbf-aircraft-design) | best of 8 attempts 0.3 | 0.0% |

## FAQ

### What does OpenWorldEngineer-Bench measure?

Whether an AI agent can finish long, real engineering tasks that experts built from their own fields: speeding up GPU kernels and interpreters, building data systems, forecasting from scientific data, rebuilding CAD parts, sizing a concrete frame and designing an aircraft. Each model gets 8 attempts of 4 to 18 hours per task, and the score is the mean task score from the starting point (0) to the task's ceiling (100).

### Which model leads OpenWorldEngineer-Bench?

Claude Fable 5.2 leads with 54.6, followed by GPT-6 Astra at 44.2 and Claude Opus 5.5 at 41.6 (Mercor data updated 2026-11-11).

### What is the Design/Build/Fly Aircraft task?

One of the 26 tasks, modeled on the AIAA Design/Build/Fly competition: the agent redesigns a small electric aircraft for laps, cargo and precision drops, testing it only through a limited budget of simulated tests, and a private grader flies it through 34 hidden flights. GPT-6 Astra (71.5) and Claude Fable 5.2 (71.4) have the best mean of 8 attempts. Its table is below the main board on this page.