# Artificial Analysis AutomationBench, strict pass@1 (AA AutomationBench (strict))

> The strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.

Canonical page: https://benchlm.ai/benchmarks/aaautomationbenchstrict

- Category: [Agentic](/agentic)
- Last updated: November 11, 2026

## About AA AutomationBench (strict)

- Year: 2026
- Tasks: End-to-end business-workflow automation tasks
- Format: Strict pass@1
- Difficulty: Agentic automation
- Paper: [Artificial Analysis AutomationBench Benchmark Leaderboard](https://artificialanalysis.ai/evaluations/automationbench-aa)

The Artificial Analysis leaderboard headline, which BenchLM stores on the AA AutomationBench page, is a partial score; this strict pass@1 metric is much lower (Claude Haiku 5.5: 35.4 partial, 13.9 strict). BenchLM first added the strict lane from Meta's Muse Ember launch post, which quotes it, and keeps it display-only.

AA AutomationBench (strict) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (3 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [Muse Ember](/models/muse-ember) | Meta | 37.8% |
| 2 | [GPT-6 Luna](/models/gpt-6-luna) | OpenAI | 20.2% |
| 3 | [Claude Haiku 5.5](/models/claude-haiku-5-5) | Anthropic | 13.9% |

## FAQ

### What does AA AutomationBench (strict) measure?

The strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.

### Which model scores highest on AA AutomationBench (strict)?

Muse Ember by Meta currently leads with a score of 37.8% on AA AutomationBench (strict).

### How many models are evaluated on AA AutomationBench (strict)?

3 AI models have published results on AA AutomationBench (strict) in the BenchLM catalog.

### Does AA AutomationBench (strict) affect BenchLM's overall score?

Not directly. AA AutomationBench (strict) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Compare Top Models on AA AutomationBench (strict)

- [Muse Ember vs GPT-6 Luna](/compare/gpt-6-luna-vs-muse-ember)
- [GPT-6 Luna vs Claude Haiku 5.5](/compare/claude-haiku-5-5-vs-gpt-6-luna)