ExplorationBench AlienCode Best@3 (ExplorationBench AlienCode)
We show this table for reference; we do not rank on it.
Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks.
Best@3 held-out accuracy on ExplorationBench AlienCode — October 15, 2026 snapshot
We mirror the published best@3 held-out accuracy view for ExplorationBench AlienCode. Claude Opus 5 leads the public snapshot at 89.0%, followed by GPT-5.6 Sol (88.1%) and Gemini 3.8 Flash (79.0%). We do not use these results to rank models overall.
Claude Opus 5
Anthropic
max reasoning
GPT-5.6 Sol
OpenAI
max reasoning
Gemini 3.8 Flash
high reasoning
10 systemsAgenticCurrentDisplay onlyUpdated October 15, 2026 snapshot
| Rank | System / reasoning | Before exploration | Best@3 | Mean@3 | Worst | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5Anthropic · max | 3.8% | 89.0% | 73.5% | 49.0% | Not reported | Closed |
| 2 | GPT-5.6 SolOpenAI · max | 11.0% | 88.1% | 79.5% | 73.8% | Not reported | Closed |
| 3 | Gemini 3.8 FlashGoogle · high | 1.4% | 79.0% | 39.7% | 6.2% | Not reported | Closed |
| 4 | Kimi K3Moonshot AI · max | 2.4% | 79.0% | 38.7% | 5.7% | Not reported | Unknown |
| 5 | Grok 4.6xAI · xhigh | 4.8% | 69.0% | 53.2% | 37.1% | Not reported | Closed |
| 6 | Qwen3.8-MaxAlibaba · max | 2.4% | 65.7% | 64.0% | 61.4% | Not reported | Open |
| 7 | Hy4 previewTencent · high | 0.5% | 63.3% | 30.2% | 1.4% | Not reported | Open |
| 8 | DeepSeek-V4.1-FlashDeepSeek · max | 1.4% | 59.5% | 46.3% | 23.3% | Not reported | Open |
| 9 | Seed2.1 ProByteDance · high | 2.4% | 40.0% | 24.8% | 4.8% | Not reported | Closed |
| 10 | DeepSeek-V4-ProDeepSeek · max | 1.9% | 12.9% | 7.3% | 2.4% | Not reported | Unknown |
Before exploration and Best@3 come from the same selected run. Mean@3 and Worst cover all three runs. Each checkpoint averages three answers per task.
How to read ExplorationBench AlienCode
AlienCode tests whether a system can discover hidden rules through experiments, then use them on 70 held-out tasks. Each system gets four exploration rounds and three independent runs. An interpreter or proof checker grades the answers.
Best@3 is the best complete run. Mean@3 and Worst show how repeatable that result was; Best@3 does not combine successful answers across runs. The code and tasks are private, so we keep these tables display only and outside overall and category rankings.
Scores follow the current web table. The September 24 paper describes the protocol but reports an earlier AlienCode maximum of 87.6% versus 89.0% on the website.
Snapshot
The published ExplorationBench AlienCode snapshot places Claude Opus 5 first at 89.0%. The third row is 10.0 points behind. The broader top-10 range is 76.1 points, so the table still separates the published systems.
10 systems have been evaluated on ExplorationBench AlienCode. The benchmark falls in the Agentic category. ExplorationBench AlienCode is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About ExplorationBench AlienCode
Year
2026
Tasks
70 held-out program-synthesis tasks; 31 discovery targets
Format
Held-out accuracy (Best@3)
Difficulty
Discovering and applying unfamiliar executable rules
Four exploration rounds precede tool-disabled evaluation on 70 tasks. The interpreter checks each program on five private inputs. The headline is the best complete run of three; the table also shows the mean and worst. Scores follow the benchmark-owned web table, with private code and tasks, and stay outside overall and category rankings.
Freshness and provenance
Version
September 2026 web protocol: AlienCode
Refresh cadence
On source changes
Staleness state
Current
Question availability
Code and held-out tasks private
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does ExplorationBench AlienCode measure?
Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks.
Which model leads the published ExplorationBench AlienCode snapshot?
Claude Opus 5 currently leads the published ExplorationBench AlienCode snapshot with 89.0% best@3 held-out accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on ExplorationBench AlienCode?
The October 15, 2026 snapshot snapshot contains 10 systems at their published reasoning settings.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.