AutomationBench, Zapier private evaluation set release 1.0.6 (AutomationBench (Zapier 1.0.6))

The current capture contains 126 configurations from Zapier's private AutomationBench 1.0.6 leaderboard. A task passes only when every final-state assertion passes. The partial-credit reward is a separate diagnostic, and no language model judges the headline score.

Haiku 5.5 rows marked with a section sign use its initial launch version. Zapier says it will update those results after evaluating an updated release.

The leaderboard uses API tools across 47 simulated applications and six business domains. Its held-out tasks differ from the public 600-task set and Artificial Analysis AutomationBench-AA. We keep those results in separate tables. The capture date does not establish when each configuration was evaluated.

Default-fallback configurations rerun refused tasks through the provider's own fallback routing. The Fable 5.1 plus Opus 5 configuration uses Opus for refused steps on about 40% of tasks, and its $2.45 cost covers Fable only. That combined score is not a result from Fable alone.

Row costs are USD per evaluation task, not token tariffs. Starred Gemini costs use standard list prices: promotional task costs are $0.85 for Argon High, $0.77 for Argon Medium, $0.27 for Flash 3.8 Medium, and $0.31 for Flash 3.8 High. Flash 3.7 costs $0.30 per task under a promotion through December 31, 2026. DeepSeek's $0.04 is the cached Fireworks rate; its uncached rate is $0.14. Gemma's cost comes from a dedicated deployment.

We preserve the published configuration names and ordinal ranks, including equal displayed scores. The table is display only and does not enter overall or category rankings. The existing Sonnet 5.5 system-card value remains 44.7%; this source table publishes 44.75% for its max-effort default-fallback configuration.