OSWorld 2.0
We show this table for reference; we do not rank on it.
OSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked.
Current release
OSWorld 2.1 is the current release. It fixes task evaluators, assets, and mocked websites, so its scores are not directly comparable with the OSWorld 2.0 releases on this page. OSWorld-Verified separately covers the repaired original task suite.
View OSWorld 2.1Results by comparable protocol (25 models)
Ranks appear only within groups that match the metric, task release, action mode, step budget, and reasoning effort. Rows with missing or unmatched protocol details remain visible without a cross-row rank.
Binary completion · batched actions · v2026.06.24 · 108 tasks · 500 steps · maximum reasoning
| Rank in group | Model | Metric | Score | Evidence |
|---|---|---|---|---|
| 1 | Claude Opus 4.8Batched actions | Binary completion | 20.6% | OSWorld 2.0 paper |
| 2 | Claude Opus 4.7 (Adaptive)Batched actions | Binary completion | 18.2% | OSWorld 2.0 paper |
| 3 | GPT-5.5Batched actions | Binary completion | 13.0% | OSWorld 2.0 paper |
Binary completion · single actions · v2026.06.24 · 108 tasks · 500 steps · maximum reasoning
| Rank in group | Model | Metric | Score | Evidence |
|---|---|---|---|---|
| 1 | Claude Opus 4.8Single actions | Binary completion | 18.5% | OSWorld 2.0 paper |
| 2 | Claude Opus 4.7Single actions | Binary completion | 13.9% | OSWorld 2.0 paper |
| 3 | Claude Sonnet 4.6Single actions | Binary completion | 8.3% | OSWorld 2.0 paper |
| 4 | Kimi K2.6Single actions | Binary completion | 4.6% | OSWorld 2.0 paper |
| 4 | MiniMax M3Single actions | Binary completion | 4.6% | OSWorld 2.0 paper |
| 6 | Qwen3.7 PlusSingle actions | Binary completion | 2.8% | OSWorld 2.0 paper |
Partial score · batched actions · v2026.06.24 · 108 tasks · 500 steps · maximum reasoning
| Rank in group | Model | Metric | Score | Evidence |
|---|---|---|---|---|
| 1 | Claude Opus 4.8Batched actions | Partial score | 54.8% | OSWorld 2.0 paper |
| 2 | GPT-5.5Batched actions | Partial score | 49.5% | OSWorld 2.0 paper |
| 3 | Claude Opus 4.7 (Adaptive)Batched actions | Partial score | 48.9% | OSWorld 2.0 paper |
Partial score · single actions · v2026.06.24 · 108 tasks · 500 steps · maximum reasoning
| Rank in group | Model | Metric | Score | Evidence |
|---|---|---|---|---|
| 1 | Claude Opus 4.8Single actions | Partial score | 49.3% | OSWorld 2.0 paper |
| 2 | Claude Opus 4.7Single actions | Partial score | 49.1% | OSWorld 2.0 paper |
| 3 | Claude Sonnet 4.6Single actions | Partial score | 41.5% | OSWorld 2.0 paper |
| 4 | MiniMax M3Single actions | Partial score | 22.3% | OSWorld 2.0 paper |
| 5 | Kimi K2.6Single actions | Partial score | 22.1% | OSWorld 2.0 paper |
| 6 | Qwen3.7 PlusSingle actions | Partial score | 21.5% | OSWorld 2.0 paper |
Other reported results · no cross-row rank
| Rank | Model | Metric | Score | Task set and run settings | Evidence |
|---|---|---|---|---|---|
| — | Claude Fable 5.1 | Partial score | 77.9% | OSWorld 2.0 task release · August 2026 (day not reported)Run settings not reportedThe source identifies the August 2026 release but not the exact release day or matching run settings. | Anthropic: Claude Fable 5.1 and Claude Mythos 5.1 system card |
| — | Claude Fable 5.1 | Strict binary completion | 41.7% | OSWorld 2.0 task release · August 2026 (day not reported)Run settings not reportedThe source identifies the August 2026 release but not the exact release day or matching run settings. | Anthropic: Claude Fable 5.1 and Claude Mythos 5.1 system card |
| — | Claude Opus 5 | Metric not reported | 70.6% | Task release not reportedRun settings not reportedThe source record reports a score but does not identify its metric or task release. | Anthropic: Claude Opus 5 system card |
| — | Claude Opus 5.5 | Partial score | 81.8% | Task release not reportedRun settings not reportedThe source reports a strict pass rate and partial score, but the stored source record does not identify the task release. | Anthropic: Claude Opus 5.5 system card |
| — | Claude Opus 5.5 | Strict binary completion | 48.7% | Task release not reportedRun settings not reportedThe source reports a strict pass rate and partial score, but the stored source record does not identify the task release. | Anthropic: Claude Opus 5.5 system card |
| — | Gemini 3.7 Flash | Metric not reported | 47.9% | Task release not reportedRun settings not reportedThe source record reports a score but does not identify its metric or task release. | Google DeepMind: Gemini 3.7 Flash model card |
| — | Gemini 3.8 FlashBatched actions | Partial score | 59.0% | Task release not reportedBatched actionsBatch tool use is reported, but the source record does not identify the task release or the remaining run settings. | Google DeepMind: Gemini 3.8 Flash model card |
| — | Gemini 4 ArgonBatched actions | Partial score | 69.2% | OSWorld 2.0 offline subset · v2026.08.08Batched actions · 500 steps · maximum reasoning · maximum of 3 runs · 1080pThe source reports the maximum of three single-attempt runs with a Gemini CUA harness; this setup has no matched source row. | Google: Gemini 4 Argon evaluation methodology |
| — | GPT-5.6 Luna | Metric not reported | 45.6% | Task release not reportedRun settings not reportedThe source record reports the score but does not identify the metric or task release. | OpenAI: GPT-5.6 |
| — | GPT-5.6 Sol | Metric not reported | 62.6% | Task release not reportedRun settings not reportedThe source record reports the score but does not identify the metric or task release. | OpenAI: GPT-5.6 |
| — | GPT-5.6 Terra | Metric not reported | 50.2% | Task release not reportedRun settings not reportedThe source record reports the score but does not identify the metric or task release. | OpenAI: GPT-5.6 |
| — | GPT-6 Astra | Partial score | 72.6% | OSWorld 2.0 offline set · v2026.08.08Run settings not reportedOpenAI identifies the offline set and partial score; the stored source record does not specify matching run settings. | OpenAI: GPT-6 Astra |
| — | GPT-6 Sol | Partial score | 60.5% | OSWorld 2.0 offline set · v2026.08.08xhigh reasoningOpenAI identifies the offline set, partial score, and xhigh effort; no matching row has the same disclosed setup. | OpenAI: Introducing GPT-6 Sol and Luna |
| — | Kimi K3.1 Pro | Partial score | 79.6% | OSWorld V2.1 · V2.1maximum reasoningMoonshot runs OSWorld V2.1 with its Kimi Agent harness in the official OSWorld v2 repository; its Claude Opus 5.5 comparison value matches that model's partial score. No stored row shares this release and harness. | Moonshot AI: Kimi K3.1 Pro launch blog |
| — | MiniMax M3.1 Flash | Metric not reported | 61.7% | OSWorld 2.0 0808 Full release · v2026.08.08Run settings not reportedMiniMax names the 0808 Full release but not the metric or run settings. | MiniMax: MiniMax M3.1 Flash |
| — | Muse Spark 1.1 | Metric not reported | 14.2% | Task release not reportedRun settings not reportedThe source record reports a score but does not identify its metric or task release. | Meta AI: Muse Spark 1.1 evaluation report |
| — | Muse Spark 1.3 | Mean partial score | 66.9% | Task release not reportedRun settings not reportedThe source record identifies mean partial score as primary but does not identify the task release. | Meta AI Research: Muse Spark 1.3 launch scorecard |
| — | Qwen3.8 Max | Binary completion | 19.4% | Task release not reportedRun settings not reportedThe source reports paired binary and partial scores but the source record does not identify the task release. | Qwen: Qwen3.8-Max release benchmarks |
| — | Qwen3.8 Max | Partial score | 46.7% | Task release not reportedRun settings not reportedThe source reports paired binary and partial scores but the source record does not identify the task release. | Qwen: Qwen3.8-Max release benchmarks |
| — | Qwen3.8-Flash-Next | Binary completion | 19.4% | Task release not reportedRun settings not reportedThe source reports paired binary and partial scores but the source record does not identify the task release. | Qwen3.8-Flash-Next model card |
| — | Qwen3.8-Flash-Next | Partial score | 52.3% | Task release not reportedRun settings not reportedThe source reports paired binary and partial scores but the source record does not identify the task release. | Qwen3.8-Flash-Next model card |
About OSWorld 2.0
Year
2026
Tasks
108 long-horizon computer-use workflows
Format
Interactive computer-use evaluation; binary completion and partial-score metrics
Difficulty
Long-horizon professional workflows
The paper's v2026.06.24 release covers 108 long-horizon workflows and reports binary completion and partial scores under single-action and batched-action settings. Other sources use later or unspecified task sets and may report different metrics or omit run settings. We rank only rows that match the metric, task release, action mode, step budget, and reasoning effort; other sourced results remain visible without a cross-row rank.
Freshness and provenance
Version
OSWorld 2.0 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
Can I compare every OSWorld 2.0 score directly?
No. The published rows mix binary completion and partial scores, task releases, and action settings. We rank results only inside groups that match the metric, release, action mode, step budget, and reasoning effort. A source row with missing or unmatched settings stays visible with its evidence, but it has no cross-row rank.
Why are Opus 4.8's scores 20.6% and 54.8%?
They are paired results from the paper's v2026.06.24 release: 20.6% is binary completion and 54.8% is partial score, both with batched actions on the 108-task set at 500 steps. They measure different outcomes, so the partial score cannot replace the binary score in a comparison that uses another metric or release.
What does GPT-6 Astra's 72.6% measure?
OpenAI labels 72.6% as a partial score on the v2026.08.08 offline set. That release differs from the paper's v2026.06.24 108-task set, where Opus 4.8 reports 20.6% binary completion and 54.8% partial score. We keep Astra's source value, but do not rank it against rows without a fully matched protocol.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.