STEMBench120: six-domain scientific research progress (STEMBench)
We show this table for reference; we do not rank on it.
Scores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved.
Progress score (0 to 4) on STEMBench — October 16, 2026 snapshot
We mirror the published progress score (0 to 4) view for STEMBench. GPT-6 Astra leads the public snapshot at 2.81, followed by GPT-6.1 Sol (2.76) and Muse Wonder (2.68). We do not use these results to rank models overall.
GPT-6 Astra
OpenAI
With tools (Codex) · 120 of 120 usable · 9 level-3
GPT-6.1 Sol
OpenAI
With tools (Codex) · 120 of 120 usable · 8 level-3
Muse Wonder
Meta
Closed-book · 120 of 120 usable · 4 level-3
10 audited runsReasoningCurrentDisplay onlyUpdated October 16, 2026 snapshot
Progress score (0 to 4) table (10 audited runs)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAIWith tools (Codex) · 120 of 120 usable · 9 level-3 | 2.81 | Not reported | Closed |
| 2 | GPT-6.1 SolOpenAIWith tools (Codex) · 120 of 120 usable · 8 level-3 | 2.76 | Not reported | Closed |
| 3 | Muse WonderMetaClosed-book · 120 of 120 usable · 4 level-3 | 2.68 | Not reported | Closed |
| 4 | Claude Opus 5.5AnthropicClosed-book · 113 of 120 usable · 5 level-3 | 2.62 | Not reported | Closed |
| 5 | Hy4 previewTencentClosed-book · 120 of 120 usable · 1 level-3 | 2.59 | Not reported | Open |
| 6 | Mistral Large 4MistralClosed-book · 118 of 120 usable · 0 level-3 | 2.50 | Not reported | Pending |
| 7 | DeepSeek V4.1 Flash 1017DeepSeekClosed-book · 117 of 120 usable · 0 level-3 | 2.49 | Not reported | Open |
| 8 | Gemini 4 ArgonGoogleClosed-book · 105 of 120 usable · 3 level-3 | 2.44 | Not reported | Closed |
| 9 | Qwen3.8-27B NVFP4 · Closed-bookAlibabaClosed-book · 109 of 120 usable · 0 level-3 | 2.31 | Not reported | Unknown |
| 10 | MiMo-V2.6-ProXiaomiClosed-book · 100 of 120 usable · 0 level-3 | 2.17 | Not reported | Open |
How to read this leaderboard
A mean near 2 means most responses reached meaningful partial progress; near 3 means substantial target-level progress is common. No run in the audit has a level-4 result, so the scale separates partial progress rather than solved tasks. Fractional differences within 0.05 are practical ties, and per-task review allowances of 0.05 to 0.20 apply.
Operator receipt: 10 sourced rows are currently displayable on this page; the leading published row is GPT-6 Astra at 2.81.
Honest limit: Runs differ in resources: the two Codex runs had web or source retrieval, code execution, file operations and multi-day agent workflows, while the closed-book API runs had no model-side tools, so the table compares complete agent configurations. Each model has one run per setting, all with up to three retries. The fractional index is experimental, and the task statements, per-task judgments and submissions of STEMBench120 are not public.
How we show STEMBench
Ulam audits 120 research tasks, 20 in each of six STEM domains, and scores how far each response advances the task on a 0 to 4 progress scale. We mirror the October 14, 2026 snapshot audit of STEMBench120 v1.0.0: 10 runs across 10 models, with the complete-run mean of the experimental fractional score as the shown value.
2 runs used Codex with retrieval, code execution, file operations and multi-day agent workflows; 8 closed-book runs called the API with no model-side tools. The settings measure different things, so every row names its setting, and each model's page shows its best run with the setting in its note.
No run in the audit reached a full-target success, so the scale separates partial progress. Fractional differences within 0.05 are practical ties. Each score belongs to one complete agent configuration under one resource setting, and the STEMBench120 tasks are private, so the table is display only and does not enter model rankings.
Snapshot
The published STEMBench snapshot places GPT-6 Astra first at 2.81. The third row is 0.13 score units behind. The broader top-10 range is 0.64 score units, so many of the published results sit in a relatively narrow band.
10 audited runs have been evaluated on STEMBench. The benchmark falls in the Reasoning category. STEMBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About STEMBench
Year
2026
Tasks
120 research tasks, 20 in each of six STEM domains
Format
Audited progress level 0 to 4 per task, refined to a fractional score; complete-run mean across all tasks
Difficulty
Open research targets; no run has reached a full-target success
STEMBench120 gives each model 120 research tasks, 20 per domain, and audits every response for progress toward the task's full target. An integer level records the outcome: 0 no usable progress, 1 framing or protocol, 2 meaningful partial progress with obligations still open, 3 substantial target-level progress, 4 full target met. An experimental quality index refines levels 1 to 3 into a fractional score (level plus 0.95 times the index) from supported achievement strength, obligation coverage, verification and reproducibility, scientific support, validity surviving decisive gaps, and scope honesty. The complete-run mean averages every task, zero-score attempts included; a usable-answer mean excludes assignments without usable content. Ulam publishes aggregates only for STEMBench120; the separate STEMBench-small set of 60 tasks is public under Apache 2.0 without baseline scores.
Freshness and provenance
Version
STEMBench120 v1.0.0
Refresh cadence
Audit snapshots
Staleness state
Current
Question availability
STEMBench120 tasks and judgments are private; the 60-task STEMBench-small set is public under Apache 2.0 without baseline scores
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does STEMBench measure?
STEMBench measures research progress, not answer correctness. Each of its 120 tasks sets a research target in one of six STEM domains, and an audit scores how far the response advances it on a 0 to 4 scale, crediting supported partial work and honest statements of what remains unresolved. BenchLM shows the complete-run mean of the fractional score.
Why do some rows say With tools and others Closed-book?
The two Codex runs used web or source retrieval, code execution, file operations and multi-day agent workflows. The closed-book runs called the model's API with no model-side tools. The settings measure different things, so every row names its setting and a model's page shows its best run with the setting in its note.
Why is STEMBench display only?
Each score belongs to one complete agent configuration under a specific resource setting, the fractional index is experimental, and the private STEMBench120 tasks and judgments cannot be re-run. The results compare those exact runs, so they stay out of BenchLM's rankings.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.