Artificial Analysis AutomationBench, strict pass@1 (AA AutomationBench (strict))
We show this table for reference; we do not rank on it.
The strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.
Benchmark score on AA AutomationBench (strict) — November 11, 2026
We compile the AA AutomationBench (strict) rows from provider self-reports. Muse Ember leads the table at 37.8%, followed by GPT-6 Luna (20.2%) and Claude Haiku 5.5 (13.9%). We do not use these results to rank models overall.
Muse Ember
Meta
GPT-6 Luna
OpenAI
Claude Haiku 5.5
Anthropic
3 modelsAgenticCurrentDisplay onlyUpdated November 11, 2026
Benchmark score table (3 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Muse EmberMeta | 37.8% | Not reported | Closed |
| 2 | GPT-6 LunaOpenAI | 20.2% | Not reported | Closed |
| 3 | Claude Haiku 5.5Anthropic | 13.9% | Not reported | Closed |
Among the reported AA AutomationBench (strict) rows, Muse Ember is first at 37.8%. The third row is 23.9 points behind. The broader top-10 range is 23.9 points, so the table still separates the published systems.
3 models have been evaluated on AA AutomationBench (strict). The benchmark falls in the Agentic category. AA AutomationBench (strict) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About AA AutomationBench (strict)
Year
2026
Tasks
End-to-end business-workflow automation tasks
Format
Strict pass@1
Difficulty
Agentic automation
The Artificial Analysis leaderboard headline, which BenchLM stores on the AA AutomationBench page, is a partial score; this strict pass@1 metric is much lower (Claude Haiku 5.5: 35.4 partial, 13.9 strict). BenchLM first added the strict lane from Meta's Muse Ember launch post, which quotes it, and keeps it display-only.
Freshness and provenance
Version
AA AutomationBench (strict) 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does AA AutomationBench (strict) measure?
The strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.
Which model scores highest on AA AutomationBench (strict)?
Muse Ember by Meta currently leads with a score of 37.8% on AA AutomationBench (strict).
How many models are evaluated on AA AutomationBench (strict)?
3 AI models have published results on AA AutomationBench (strict) in the BenchLM catalog.
Compare top models on AA AutomationBench (strict)
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.