OpenWorldEngineer-Bench
We show this table for reference; we do not rank on it.
Mercor's benchmark of AI agents on 26 long, open-ended engineering tasks in science, software, ML and engineering, built by domain experts. Each model makes 8 attempts per task, of 4 to 18 hours each, and the score is the mean task score on a scale from the starting point (0) to the best result the task allows (100).
Mean task score on OpenWorldEngineer-Bench — Release 1.0 preview, Mercor data of November 11, 2026
We mirror the published mean task score view for OpenWorldEngineer-Bench. Claude Fable 5.2 leads the public snapshot at 54.6%, followed by GPT-6 Astra (44.2%) and Claude Opus 5.5 (41.6%). We do not use these results to rank models overall.
Claude Fable 5.2
Anthropic
max reasoning
95% interval 46.1 to 63.0 · best attempt 68.3 · $105.08 and 8.7 h per attempt
GPT-6 Astra
OpenAI
max reasoning
95% interval 36.5 to 51.9 · best attempt 58.3 · $234.63 and 9.3 h per attempt
Claude Opus 5.5
Anthropic
max reasoning
95% interval 36.0 to 47.1 · best attempt 58.4 · $75.70 and 7.9 h per attempt
14 modelsAgenticCurrentDisplay onlyUpdated Release 1.0 preview, Mercor data of November 11, 2026
Mean task score table (14 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Fable 5.2Anthropicmax reasoning95% interval 46.1 to 63.0 · best attempt 68.3 · $105.08 and 8.7 h per attempt | 54.6% | Not reported | Closed |
| 2 | GPT-6 AstraOpenAImax reasoning95% interval 36.5 to 51.9 · best attempt 58.3 · $234.63 and 9.3 h per attempt | 44.2% | Not reported | Closed |
| 3 | Claude Opus 5.5Anthropicmax reasoning95% interval 36.0 to 47.1 · best attempt 58.4 · $75.70 and 7.9 h per attempt | 41.6% | Not reported | Closed |
| 4 | GPT-6.1 SolOpenAImax reasoning95% interval 34.3 to 48.0 · best attempt 57.1 · $52.12 and 9.4 h per attempt | 41.2% | Not reported | Closed |
| 5 | Claude Sonnet 5.5Anthropicmax reasoning95% interval 30.8 to 47.2 · best attempt 52.7 · $38.65 and 8.4 h per attempt | 39.0% | Not reported | Closed |
| 6 | Gemini 4 CarbonGooglehigh reasoning95% interval 27.2 to 38.3 · best attempt 51.0 · $62.08 and 9.0 h per attempt | 32.8% | Not reported | Closed |
| 7 | Claude Fable 5.1Anthropicmax reasoning95% interval 24.1 to 34.1 · best attempt 45.6 · $87.70 and 6.6 h per attempt | 29.1% | Not reported | Closed |
| 8 | Gemini 4 ArgonGooglehigh reasoning95% interval 18.3 to 29.9 · best attempt 42.2 · $49.44 and 7.6 h per attempt | 24.1% | Not reported | Closed |
| 9 | Kimi K3.1Moonshot AImax reasoning95% interval 9.8 to 17.4 · best attempt 20.6 · $21.16 and 5.6 h per attempt | 13.6% | Not reported | Unknown |
| 10 | DeepSeek V4.1 Flash 1017DeepSeekmax reasoning95% interval 9.3 to 15.4 · best attempt 23.4 · $1.37 and 5.5 h per attempt | 12.4% | Not reported | Open |
| 11 | GLM-5.3Z.AImax reasoning95% interval 8.4 to 15.3 · best attempt 21.0 · $24.83 and 7.2 h per attempt | 11.8% | Not reported | Open |
| 12 | Grok 4.7xAImax reasoning95% interval 4.4 to 9.6 · best attempt 13.8 · $146.28 and 6.4 h per attempt | 7.0% | Not reported | Closed |
| 13 | Beam 1.0 501B-A23BReflection AImax reasoning95% interval 2.1 to 4.5 · best attempt 8.9 · $1.47 and 2.8 h per attempt | 3.3% | Not reported | Open |
| 14 | InklingThinking Machines Labmax reasoning95% interval 0.5 to 1.8 · best attempt 3.8 · $3.26 and 0.6 h per attempt | 1.1% | Not reported | Open |
OpenWorldEngineer DBF Aircraft
BenchLM mirrors the mean of 8 attempts on the Design/Build/Fly Aircraft task from the data Mercor publishes. It is excluded from BenchLM overall and category rankings.
| Source rank | Source model or configuration | Mean of 8 attempts | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAI | 71.5% | max reasoningOriginal result | Not reported | Closed |
| 2 | Claude Fable 5.2Anthropic | 71.4% | max reasoningOriginal result | Not reported | Closed |
| 3 | GPT-6.1 SolOpenAI | 57.1% | max reasoningOriginal result | Not reported | Closed |
| 4 | Claude Opus 5.5Anthropic | 42.5% | max reasoningOriginal result | Not reported | Closed |
| 5 | Gemini 4 CarbonGoogle | 38.4% | high reasoningOriginal result | Not reported | Closed |
| 6 | Claude Sonnet 5.5Anthropic | 34.5% | max reasoningOriginal result | Not reported | Closed |
| 7 | Gemini 4 ArgonGoogle | 28.6% | high reasoningOriginal result | Not reported | Closed |
| 8 | Claude Fable 5.1Anthropic | 22.3% | max reasoningOriginal result | Not reported | Closed |
| 9 | GLM-5.3Z.AI | 5.6% | max reasoningOriginal result | Not reported | Open |
| 10 | DeepSeek V4.1 Flash 1017DeepSeek | 4.1% | max reasoningOriginal result | Not reported | Open |
| 11 | Kimi K3.1Moonshot AI | 3.3% | max reasoningOriginal result | Not reported | Unknown |
| 12 | InklingThinking Machines Lab | 1.4% | max reasoningOriginal result | Not reported | Open |
| 13 | Beam 1.0 501B-A23BReflection AI | 1.0% | max reasoningOriginal result | Not reported | Open |
| 14 | Grok 4.7xAI | 0.0% | max reasoningOriginal result | Not reported | Closed |
How to read this leaderboard
Higher is better. Claude Fable 5.2 leads at 54.6 (95% interval 46.1 to 63.0), ahead of GPT-6 Astra (44.2) and Claude Opus 5.5 (41.6); the intervals of the second to fifth models overlap, so read those ranks as close. Cost per attempt varies widely: GPT-6 Astra spends about $235 per attempt against $105 for Claude Fable 5.2 and $76 for Claude Opus 5.5. Below the top eight, every model scores under 14.
Operator receipt: 14 sourced rows are currently displayable on this page; the leading published row is Claude Fable 5.2 at 54.6%.
Honest limit: 26 tasks is a small set, and task scores vary a lot, so the 95% intervals are wide. Every model runs in one harness (Harbor) at a fixed reasoning effort; another harness or effort may do better or worse. Ten tasks are graded privately, so their scores cannot be reproduced outside Mercor. The release is labelled a preview and tasks can be patched and re-run, as the aircraft task was in November 2026. BenchLM shows OpenWorldEngineer-Bench as display only; it does not enter overall or category rankings.
How OpenWorldEngineer-Bench is shown here
We mirror the OpenWorldEngineer-Bench leaderboard from Mercor. Each model makes 8 attempts at each of 26 long engineering tasks in 4 domains (science, software, ML and engineering), with 4 to 18 hours per attempt and web search over a copy of the web frozen on October 1, 2026. Every model runs in the same Harbor agent harness.
Each task is scored on its own scale, from the untouched starting point (0) to the best result its metric allows (100). A model's score is its mean of the attempts on each task, averaged over all tasks, as on Mercor's board. Each row also gives the 95% interval across tasks, the average best attempt, and the cost and agent time per attempt. Mercor labels this release 1.0 preview. OpenWorldEngineer-Bench is display only on BenchLM.
Snapshot
The published OpenWorldEngineer-Bench snapshot places Claude Fable 5.2 first at 54.6%. The third row is 13.0 points behind. The broader top-10 range is 42.2 points, so the table still separates the published systems.
14 models have been evaluated on OpenWorldEngineer-Bench. The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. OpenWorldEngineer-Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About OpenWorldEngineer-Bench
Year
2026
Tasks
26 expert-built engineering tasks in 4 domains (science, software, ML, engineering)
Format
8 attempts per task in the Harbor agent harness, 4 to 18 hours each, web search over an October 1, 2026 copy of the web
Difficulty
Long-horizon engineering work: kernels, data systems, scientific forecasting, CAD and structural and aircraft design
Each task gives an agent a container with its hardware (up to a B300 or H100 GPU), an instruction, starting code and development cases, and asks for a measurable engineering result: faster GPU kernels and interpreters, a SQL change replayer or Parquet lakehouse, earthquake relocation and storm-surge forecasts, CAD models rebuilt from drawings or scans, a reinforced-concrete frame, a small aircraft. Attempts run in the Harbor harness with web search over a copy of the web frozen on October 1, 2026, and use about 124 million tokens on average. Each task's result is placed on its own scale, from the untouched baseline (0) to the ceiling its metric allows (100). A model's task score is the mean of its graded attempts, and its benchmark score is the mean over tasks, with a 95% interval of 1.96 standard errors across tasks. Sixteen tasks ship a public grader with a public test split; ten keep a private grader because it cannot be separated from hidden test data or, for Design/Build/Fly Aircraft, a private flight simulator. Mercor labels the release 1.0 preview and re-ran the aircraft task on November 10 and 11, 2026 after patching it. BenchLM recomputes the scores from the attempt-level data on Mercor's site and keeps each task's mean and best attempt in its history.
Freshness and provenance
Version
OpenWorldEngineer-Bench 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does OpenWorldEngineer-Bench measure?
Whether an AI agent can finish long, real engineering tasks that experts built from their own fields: speeding up GPU kernels and interpreters, building data systems, forecasting from scientific data, rebuilding CAD parts, sizing a concrete frame and designing an aircraft. Each model gets 8 attempts of 4 to 18 hours per task, and the score is the mean task score from the starting point (0) to the task's ceiling (100).
Which model leads OpenWorldEngineer-Bench?
Claude Fable 5.2 leads with 54.6, followed by GPT-6 Astra at 44.2 and Claude Opus 5.5 at 41.6 (Mercor data updated 2026-11-11).
What is the Design/Build/Fly Aircraft task?
One of the 26 tasks, modeled on the AIAA Design/Build/Fly competition: the agent redesigns a small electric aircraft for laps, cargo and precision drops, testing it only through a limited budget of simulated tests, and a private grader flies it through 34 hidden flights. GPT-6 Astra (71.5) and Claude Fable 5.2 (71.4) have the best mean of 8 attempts. Its table is below the main board on this page.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.