Harvey LAB-AA v1.1
We show this table for reference; we do not rank on it.
Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.
Evaluation results published by Artificial Analysis and Harvey. Published v1.1 aggregate results; legal task documents and submissions remain private.
All releases
This page carries both Harvey LAB-AA releases: v1.1 above, then its criterion pass rates and the archived v1.0 table. Version 1.1 changes the task set and grading, so the scores are not comparable across versions.
Jump to Harvey LAB-AA v1.0Hallucination-Gated All-Pass Rate on Harvey LAB-AA v1.1 — 2026-11-12 capture
We mirror the published hallucination-gated all-pass rate view for Harvey LAB-AA v1.1. Muse Wonder Full (Max) leads the public snapshot at 13.1%, followed by Muse Wonder (Max) (10.0%) and Grok 4.7 (Xhigh) (9.4%). We do not use these results to rank models overall.
Muse Wonder Full (Max)
Meta
Max · Stirrup
Muse Wonder (Max)
Meta
Max · Stirrup
Grok 4.7 (Xhigh)
xAI
Xhigh · Stirrup
25 modelsAgenticCurrentDisplay onlyUpdated 2026-11-12 capture
Hallucination-Gated All-Pass Rate table (25 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Muse Wonder Full (Max)MetaMax · Stirrup | 13.1% | Not reported | Closed |
| 2 | Muse Wonder (Max)MetaMax · Stirrup | 10.0% | Not reported | Closed |
| 3 | Grok 4.7 (Xhigh)xAIXhigh · Stirrup | 9.4% | Not reported | Closed |
| 4 | Muse Spark 1.3 (Max)MetaMax · Stirrup | 8.9% | Not reported | Closed |
| 5 | GPT-6 Astra (Max)OpenAIMax · Stirrup | 8.6% | Not reported | Closed |
| 6 | Claude Fable 5.2 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 7.2% | Not reported | Closed |
| 7 | GPT-6.1 Sol (Max)OpenAIMax · Stirrup | 6.9% | Not reported | Closed |
| 8 | Claude Fable 5.1 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 6.4% | Not reported | Closed |
| 9 | Kimi K3 (Max)Moonshot AIMax · Stirrup | 5.3% | Not reported | Open |
| 10 | Claude Opus 5.5 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 4.2% | Not reported | Closed |
| 11 | Gemini 4 Argon (High)GoogleHigh · Stirrup | 3.9% | Not reported | Closed |
| 12 | GPT-6 Sol (Max)OpenAIMax · Stirrup | 3.6% | Not reported | Closed |
| 13 | GPT-6 Luna (Max)OpenAIMax · Stirrup | 3.3% | Not reported | Closed |
| 14 | Mistral Large 4 PreviewMistralStirrup | 3.1% | Not reported | Closed |
| 15 | Claude Sonnet 5.5 (Max, Default Fallback)AnthropicMax, Default Fallback · Stirrup | 2.8% | Not reported | Closed |
| 16 | DeepSeek V4.1 Flash (Max)DeepSeekMax · Stirrup | 1.7% | Not reported | Open |
| 17 | GLM-5.3-FlashZ.AIStirrup | 0.8% | Not reported | Open |
| 18 | Claude Haiku 5.5 (Max)AnthropicMax · Stirrup | 0.6% | Not reported | Closed |
| 19 | GLM-5.3 (Max)Z.AIMax · Stirrup | 0.3% | Not reported | Open |
| 20 | Gemini 3.8 Flash (High)GoogleHigh · Stirrup | 0.0% | Not reported | Closed |
| 21 | MiniMax-M3MiniMaxStirrup | 0.0% | Not reported | Open |
| 22 | Inkling SmallThinking Machines LabStirrup | 0.0% | Not reported | Open |
| 23 | Inkling (Xhigh)Thinking Machines LabXhigh · Stirrup | 0.0% | Not reported | Open |
| 24 | Gemini 3.5 Flash-LiteGoogleStirrup | 0.0% | Not reported | Closed |
| 25 | Gemini 3.1 Pro PreviewGoogleStirrup | 0.0% | Not reported | Closed |
Harvey LAB-AA v1.1 Criterion Pass
Criterion Pass Rate is a percentage. Published zero results remain zero; unreported configurations are absent. Capture dates do not establish evaluation dates.
| Source rank | Source model or configuration | Criterion Pass Rate | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Muse Spark 1.3 (Max)Meta | 96.0% | Max · StirrupOriginal result | Not reported | Closed |
| 2 | Muse Wonder Full (Max)Meta | 95.5% | Max · StirrupOriginal result | Not reported | Closed |
| 3 | Muse Wonder (Max)Meta | 95.1% | Max · StirrupOriginal result | Not reported | Closed |
| 4 | Gemini 4 Argon (High)Google | 94.8% | High · StirrupOriginal result | Not reported | Closed |
| 5 | Kimi K3 (Max)Moonshot AI | 93.0% | Max · StirrupOriginal result | Not reported | Open |
| 6 | Mistral Large 4 PreviewMistral | 93.0% | StirrupOriginal result | Not reported | Closed |
| 7 | GLM-5.3 (Max)Z.AI | 93.0% | Max · StirrupOriginal result | Not reported | Open |
| 8 | Grok 4.7 (Xhigh)xAI | 92.8% | Xhigh · StirrupOriginal result | Not reported | Closed |
| 9 | DeepSeek V4.1 Flash (Max)DeepSeek | 92.5% | Max · StirrupOriginal result | Not reported | Open |
| 10 | Claude Fable 5.1 (Max, Default Fallback)Anthropic | 92.4% | Max, Default Fallback · StirrupOriginal result | Not reported | Closed |
| 11 | Claude Sonnet 5.5 (Max, Default Fallback)Anthropic | 92.1% | Max, Default Fallback · StirrupOriginal result | Not reported | Closed |
| 12 | GLM-5.3-FlashZ.AI | 91.1% | StirrupOriginal result | Not reported | Open |
| 13 | Claude Fable 5.2 (Max, Default Fallback)Anthropic | 91.0% | Max, Default Fallback · StirrupOriginal result | Not reported | Closed |
| 14 | GPT-6 Astra (Max)OpenAI | 90.6% | Max · StirrupOriginal result | Not reported | Closed |
| 15 | GPT-6.1 Sol (Max)OpenAI | 90.6% | Max · StirrupOriginal result | Not reported | Closed |
| 16 | Gemini 3.8 Flash (High)Google | 90.5% | High · StirrupOriginal result | Not reported | Closed |
| 17 | Claude Opus 5.5 (Max, Default Fallback)Anthropic | 90.0% | Max, Default Fallback · StirrupOriginal result | Not reported | Closed |
| 18 | Claude Haiku 5.5 (Max)Anthropic | 87.7% | Max · StirrupOriginal result | Not reported | Closed |
| 19 | GPT-6 Sol (Max)OpenAI | 86.9% | Max · StirrupOriginal result | Not reported | Closed |
| 20 | GPT-6 Luna (Max)OpenAI | 86.0% | Max · StirrupOriginal result | Not reported | Closed |
| 21 | MiniMax-M3MiniMax | 85.6% | StirrupOriginal result | Not reported | Open |
| 22 | Inkling (Xhigh)Thinking Machines Lab | 81.3% | Xhigh · StirrupOriginal result | Not reported | Open |
| 23 | Inkling SmallThinking Machines Lab | 77.6% | StirrupOriginal result | Not reported | Open |
| 24 | Gemini 3.5 Flash-LiteGoogle | 64.5% | StirrupOriginal result | Not reported | Closed |
| 25 | Gemini 3.1 Pro PreviewGoogle | 54.7% | StirrupOriginal result | Not reported | Closed |
AA Harvey LAB v1.0
The live source now publishes v1.1 with a three-judge panel and hallucination gate. Earlier results remain on their original protocol and are not comparable with v1.1.
| Source rank | Source model or configuration | Score | Who ran it | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|
| 1 | Kimi K3Moonshot AI | 94.6% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Open |
| 2 | Claude Fable 5Anthropic | 93.6% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| 3 | Claude Opus 5Anthropic | 93.5% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| 4 | Step 5 PreviewStepFun | 93.4% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Open |
| 5 | Claude Sonnet 5.5Anthropic | 93.1% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| 6 | Claude Fable 5.1Anthropic | 93.0% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| 7 | Claude Opus 5.5Anthropic | 91.2% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| No rank on this table | Gemini 3.7 FlashGoogle | 90.7% | Reported by Google | Artificial AnalysisOriginal result | Not reported | Closed |
| 8 | Claude Haiku 5.5Anthropic | 89.9% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Closed |
| 9 | MiniMax M3MiniMax | 88.4% | Independent run: Artificial Analysis | Artificial AnalysisOriginal result | Not reported | Open |
| No rank on this table | Nemotron 3 UltraNVIDIA | 81.7% | Carried from an upstream page | Artificial AnalysisOriginal result | Not reported | Open |
| No rank on this table | Grok 4.7xAI | 19.6% | Reported by xAI | Artificial AnalysisOriginal result | Not reported | Closed |
How to read this leaderboard
Operator receipt: 25 sourced rows are currently displayable on this page; the leading published row is Muse Wonder Full (Max) at 13.1%.
Honest limit: Private tasks and model judges limit independent reproduction. Criterion pass and hallucination-gated all-pass rates measure different outcomes.
How to read Harvey LAB-AA v1.1
Version 1.1 tests legal deliverables on 120 private tasks across 24 practice areas in the Stirrup agent harness. A three-judge panel grades each deliverable, and a separate check compares its claims with the source documents.
The headline hallucination-gated all-pass rate credits a task only when every rubric criterion passes and the deliverable contains no material hallucination, averaged across three judges. Criterion pass rate pools passed criteria across all three judges before the hallucination gate. The two metrics have separate tables.
Artificial Analysis developed v1.1 with Harvey and says its scores are not comparable with v1.0. We keep every published effort configuration and preserve the earlier table separately. These new results stay outside overall and category rankings.
Snapshot
The published Harvey LAB-AA v1.1 snapshot places Muse Wonder Full (Max) first at 13.1%. The third row is 3.6 points behind. The broader top-10 range is 8.9 points, so many of the published results sit in a relatively narrow band.
25 models are shown for Harvey LAB-AA v1.1. The benchmark falls in the Agentic category. Harvey LAB-AA v1.1 is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Harvey LAB-AA v1.1
Year
2026
Tasks
120 private legal tasks across 24 practice areas
Format
Hallucination-Gated All-Pass Rate
Difficulty
Professional legal work
Stirrup agent harness on 120 private legal tasks across 24 practice areas. Version 1.1 changes the task set and grading; scores are not comparable with v1.0. Each effort and fallback setting remains a separate configuration. Results are display only.
Freshness and provenance
Version
Harvey LAB-AA v1.1
Refresh cadence
Published leaderboard snapshots
Staleness state
Current
Question availability
Private legal tasks; only published aggregates mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Harvey LAB-AA v1.1 measure?
Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.
Which model leads the published Harvey LAB-AA v1.1 snapshot?
Muse Wonder Full (Max) currently leads the published Harvey LAB-AA v1.1 snapshot with 13.1% hallucination-gated all-pass rate. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on Harvey LAB-AA v1.1?
The 2026-11-12 capture snapshot contains 25 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.