Harvey LAB-AA v1.1
We show this table for reference; we do not rank on it.
Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.
Evaluation results published by Artificial Analysis and Harvey. Published v1.1 aggregate results; legal task documents and submissions remain private.
Previous release
Harvey LAB-AA v1.0 remains a separate historical table. Version 1.1 changes the task set and grading, so the scores are not comparable across versions.
View Harvey LAB-AA v1.0Hallucination-Gated All-Pass Rate on Harvey LAB-AA v1.1 — 2026-10-15 capture
We mirror the published hallucination-gated all-pass rate view for Harvey LAB-AA v1.1. Muse Wonder Full (Max) leads the public snapshot at 13.1%, followed by Muse Wonder (Max) (10.0%) and Grok 4.7 (Xhigh) (9.4%). We do not use these results to rank models overall.
Muse Wonder Full (Max)
Meta
Max · Stirrup
Muse Wonder (Max)
Meta
Max · Stirrup
Grok 4.7 (Xhigh)
xAI
Xhigh · Stirrup
24 modelsAgenticCurrentDisplay onlyUpdated 2026-10-15 capture
Hallucination-Gated All-Pass Rate table (24 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Muse Wonder Full (Max)MetaMax · Stirrup | 13.1% | Not reported | Closed |
| 2 | Muse Wonder (Max)MetaMax · Stirrup | 10.0% | Not reported | Closed |
| 3 | Grok 4.7 (Xhigh)xAIXhigh · Stirrup | 9.4% | Not reported | Closed |
| 4 | Muse Spark 1.3 (Max)MetaMax · Stirrup | 8.9% | Not reported | Closed |
| 5 | GPT-6 Astra (Max)OpenAIMax · Stirrup | 8.6% | Not reported | Closed |
| 6 | GPT-6.1 Sol (Max)OpenAIMax · Stirrup | 6.9% | Not reported | Closed |
| 7 | Claude Fable 5.1 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 6.4% | Not reported | Closed |
| 8 | Kimi K3 (Max)Moonshot AIMax · Stirrup | 5.3% | Not reported | Pending |
| 9 | Claude Opus 5.5 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 4.2% | Not reported | Closed |
| 10 | GPT-6 Sol (Max)OpenAIMax · Stirrup | 3.6% | Not reported | Closed |
| 11 | GPT-6 Luna (Max)OpenAIMax · Stirrup | 3.3% | Not reported | Closed |
| 12 | Mistral Large 4 PreviewMistralStirrup | 3.1% | Not reported | Pending |
| 13 | Claude Sonnet 5.5 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 2.8% | Not reported | Closed |
| 14 | Claude Haiku 5.5 (Xhigh)AnthropicXhigh · Stirrup | 2.8% | Not reported | Closed |
| 15 | DeepSeek V4.1 Flash (Max)DeepSeekMax · Stirrup | 1.7% | Not reported | Open |
| 16 | Claude Haiku 5.5 (Low)AnthropicLow · Stirrup | 1.7% | Not reported | Closed |
| 17 | Claude Haiku 5.5 (High)AnthropicHigh · Stirrup | 1.1% | Not reported | Closed |
| 18 | GLM-5.3-FlashZ.AIStirrup | 0.8% | Not reported | Open |
| 19 | Claude Haiku 5.5 (Max)AnthropicMax · Stirrup | 0.6% | Not reported | Closed |
| 20 | Claude Haiku 5.5 (Medium)AnthropicMedium · Stirrup | 0.6% | Not reported | Closed |
| 21 | GLM-5.3 (Max)Z.AIMax · Stirrup | 0.3% | Not reported | Open |
| 22 | Gemini 3.8 Flash (High)GoogleHigh · Stirrup | 0.0% | Not reported | Closed |
| 23 | MiniMax-M3MiniMaxStirrup | 0.0% | Not reported | Open |
| 24 | Inkling (Xhigh)Thinking Machines LabXhigh · Stirrup | 0.0% | Not reported | Open |
How to read this leaderboard
Operator receipt: 24 sourced rows are currently displayable on this page; the leading published row is Muse Wonder Full (Max) at 13.1%.
Honest limit: Private tasks and model judges limit independent reproduction. Criterion pass and hallucination-gated all-pass rates measure different outcomes.
How to read Harvey LAB-AA v1.1
Version 1.1 tests legal deliverables on 120 private tasks across 24 practice areas in the Stirrup agent harness. A three-judge panel grades each deliverable, and a separate check compares its claims with the source documents.
The headline hallucination-gated all-pass rate credits a task only when every rubric criterion passes and the deliverable contains no material hallucination, averaged across three judges. Criterion pass rate pools passed criteria across all three judges before the hallucination gate. The two metrics have separate tables.
Artificial Analysis developed v1.1 with Harvey and says its scores are not comparable with v1.0. We keep every published effort configuration and preserve the earlier table separately. These new results stay outside overall and category rankings.
Snapshot
The published Harvey LAB-AA v1.1 snapshot places Muse Wonder Full (Max) first at 13.1%. The third row is 3.6 points behind. The broader top-10 range is 9.4 points, so many of the published results sit in a relatively narrow band.
24 models are shown for Harvey LAB-AA v1.1. The benchmark falls in the Agentic category. Harvey LAB-AA v1.1 is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Harvey LAB-AA v1.1
Year
2026
Tasks
120 private legal tasks across 24 practice areas
Format
Hallucination-Gated All-Pass Rate
Difficulty
Professional legal work
Stirrup agent harness on 120 private legal tasks across 24 practice areas. Version 1.1 changes the task set and grading; scores are not comparable with v1.0. Each effort and fallback setting remains a separate configuration. Results are display only.
Freshness and provenance
Version
Harvey LAB-AA v1.1
Refresh cadence
Published leaderboard snapshots
Staleness state
Current
Question availability
Private legal tasks; only published aggregates mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Harvey LAB-AA v1.1 measure?
Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.
Which model leads the published Harvey LAB-AA v1.1 snapshot?
Muse Wonder Full (Max) currently leads the published Harvey LAB-AA v1.1 snapshot with 13.1% hallucination-gated all-pass rate. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on Harvey LAB-AA v1.1?
The 2026-10-15 capture snapshot contains 24 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.