ExtractBench: schema-guided enterprise document extraction (LlamaIndex leaderboard) (ExtractBench (LlamaIndex))
We show this table for reference; we do not rank on it.
Scores schema-guided extraction from 370 enterprise documents and 4,869 pages on value F1, record completeness, word and page grounding and cost, split by short, medium and long documents.
Overall value F1 on ExtractBench (LlamaIndex) — October 25, 2026 snapshot
We mirror the published overall value f1 view for ExtractBench (LlamaIndex). LlamaExtract Agentic Plus leads the public snapshot at 96.17%, followed by Reducto Deep Extract (96.08%) and Pulse (Effort) (96.07%). We do not use these results to rank models overall.
LlamaExtract Agentic Plus
LlamaIndex
LlamaExtract · Default · 7.50¢ per page
Reducto Deep Extract
Reducto
Specialized extraction API · Default · 6.35¢ per page
Pulse (Effort)
Pulse
Specialized extraction API · Effort · 10.50¢ per page
49 configurationsMultimodal & GroundedCurrentDisplay onlyUpdated October 25, 2026 snapshot
Overall value F1 table (49 configurations)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | LlamaExtract Agentic PlusLlamaIndexLlamaExtract · Default · 7.50¢ per page | 96.17% | Not reported | Unknown |
| 2 | Reducto Deep ExtractReductoSpecialized extraction API · Default · 6.35¢ per page | 96.08% | Not reported | Unknown |
| 3 | Pulse (Effort)PulseSpecialized extraction API · Effort · 10.50¢ per page | 96.07% | Not reported | Unknown |
| 4 | LlamaExtract AgenticLlamaIndexLlamaExtract · Default · 3.12¢ per page | 95.65% | Not reported | Unknown |
| 5 | Claude Sonnet 5.5AnthropicCoding agent · Sonnet 5.5 Evidence · 5.69¢ per page | 94.68% | Not reported | Closed |
| 6 | GPT-6 SolOpenAICoding agent · GPT-6 Sol Evidence · 9.80¢ per page | 94.57% | Not reported | Closed |
| 7 | Pulse (Non-effort)PulseSpecialized extraction API · Non-effort · 3.00¢ per page | 94.51% | Not reported | Unknown |
| 8 | LlamaExtract Cost-EffectiveLlamaIndexLlamaExtract · Default · 1.00¢ per page | 94.21% | Not reported | Unknown |
| 9 | GPT-5.6 SolOpenAICoding agent · GPT-5.6 Sol Evidence · 21.65¢ per page | 93.77% | Not reported | Closed |
| 10 | GPT-5.5OpenAICoding agent · GPT-5.5 · 27.83¢ per page | 93.57% | Not reported | Closed |
| 11 | GPT-5.5OpenAICoding agent · GPT-5.5 Evidence · 33.63¢ per page | 93.35% | Not reported | Closed |
| 12 | GPT-5.6 TerraOpenAICoding agent · GPT-5.6 Terra Evidence · 9.45¢ per page | 92.26% | Not reported | Closed |
| 13 | GPT-6 AstraOpenAICommercial VLM · Default · 11.09¢ per page | 91.91% | Not reported | Closed |
| 14 | GPT-5.6 LunaOpenAICoding agent · GPT-5.6 Luna Evidence · 1.20¢ per page | 91.16% | Not reported | Closed |
| 15 | GPT-6 LunaOpenAICoding agent · GPT-6 Luna Evidence · 0.97¢ per page | 91.01% | Not reported | Closed |
| 16 | Qwen3.8-Flash-NextAlibabaOpen-weight model · Default · no listed cost | 89.88% | Not reported | Open |
| 17 | Qwen3.8-27BAlibabaOpen-weight model · Default · no listed cost | 89.75% | Not reported | Open |
| 18 | Extend (Max Context)ExtendSpecialized extraction API · Max Context · 10.00¢ per page | 88.62% | Not reported | Unknown |
| 19 | Qwen3.6-35B-A3BAlibabaOpen-weight model · Default · no listed cost | 88.11% | Not reported | Open |
| 20 | Qwen3.5-35B-A3BAlibabaOpen-weight model · Default · no listed cost | 88.00% | Not reported | Open |
| 21 | DeepSeek V4.1 Flash 1017DeepSeekCommercial VLM · Default · 0.32¢ per page | 87.11% | Not reported | Open |
| 22 | Claude Opus 4.8AnthropicCoding agent · Opus 4.8 · 16.17¢ per page | 87.09% | Not reported | Closed |
| 23 | Claude Opus 4.8AnthropicCoding agent · Opus 4.8 Evidence · 63.24¢ per page | 86.84% | Not reported | Closed |
| 24 | Qwen3.5 9BAlibabaOpen-weight model · Default · no listed cost | 86.10% | Not reported | Unknown |
| 25 | Claude Opus 5.5AnthropicCoding agent · Opus 5.5 Evidence · 12.14¢ per page | 86.06% | Not reported | Closed |
| 26 | Extend ExtractExtendSpecialized extraction API · Default · 6.80¢ per page | 85.72% | Not reported | Unknown |
| 27 | Datalab (Accurate + Balanced)DatalabSpecialized extraction API · Accurate + Balanced · 3.50¢ per page | 85.70% | Not reported | Unknown |
| 28 | Reducto ExtractReductoSpecialized extraction API · Default · 4.57¢ per page | 83.79% | Not reported | Unknown |
| 29 | Kimi K3Moonshot AICommercial VLM · Default · 6.75¢ per page | 83.17% | Not reported | Pending |
| 30 | Qwen3.5 4BAlibabaOpen-weight model · Default · no listed cost | 82.43% | Not reported | Unknown |
| 31 | NuExtract3NuMindOpen-weight model · Default · no listed cost | 81.02% | Not reported | Unknown |
| 32 | GLM-5.3-FlashZ.AICommercial VLM · Default · 0.34¢ per page | 80.75% | Not reported | Open |
| 33 | Gemini 3.8 FlashGoogleCommercial VLM · Default · 0.43¢ per page | 80.71% | Not reported | Closed |
| 34 | Gemini 3.5 FlashGoogleCommercial VLM · Default · 1.00¢ per page | 79.84% | Not reported | Closed |
| 35 | Reducto Extract (v4)ReductoSpecialized extraction API · v4 · 4.59¢ per page | 79.58% | Not reported | Unknown |
| 36 | Lift Datalab 9BDatalabOpen-weight model · Default · no listed cost | 78.43% | Not reported | Unknown |
| 37 | GPT-5.4 nanoOpenAICommercial VLM · Default · 0.21¢ per page | 74.90% | Not reported | Closed |
| 38 | Gemma 4 26B A4BGoogleOpen-weight model · Default · no listed cost | 70.27% | Not reported | Open |
| 39 | Gemma 4 E4BGoogleOpen-weight model · Default · no listed cost | 69.64% | Not reported | Open |
| 40 | InternVL3.5 14BOpenGVLabOpen-weight model · Default · no listed cost | 66.35% | Not reported | Unknown |
| 41 | InternVL3.5 8BOpenGVLabOpen-weight model · Default · no listed cost | 64.63% | Not reported | Unknown |
| 42 | Qwen3.5 2BAlibabaOpen-weight model · Default · no listed cost | 63.40% | Not reported | Unknown |
| 43 | Kimi-VL A3BMoonshot AIOpen-weight model · Default · no listed cost | 57.77% | Not reported | Unknown |
| 44 | Gemma 4 E2BGoogleOpen-weight model · Default · no listed cost | 51.85% | Not reported | Open |
| 45 | NuExtract 2.0 8BNuMindOpen-weight model · Default · no listed cost | 49.42% | Not reported | Unknown |
| 46 | Qwen3.5 0.8BAlibabaOpen-weight model · Default · no listed cost | 46.63% | Not reported | Unknown |
| 47 | MiniCPM-V 4.5OpenBMBOpen-weight model · Default · no listed cost | 45.85% | Not reported | Unknown |
| 48 | Gemma 4 12BGoogleOpen-weight model · Default · no listed cost | 45.39% | Not reported | Open |
| 49 | GLM-4.6V FlashZ.AIOpen-weight model · Default · no listed cost | 45.19% | Not reported | Unknown |
ExtractBench (Kaggle)
Published Kaggle aggregates only. Model configurations and evaluation dates stay attached to each result. These results do not enter overall or category rankings. Task data, prompts, answers, and per-case outputs are not included. How Kaggle Benchmarks reruns tests.
| Source rank | Source model or configuration | ExtractBench overall score | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | GPT-5.6 Sol | 91.0% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
| 2 | GPT-6 Astra | 90.3% | Task version 6; Evaluated 2026-09-23Original result | Not reported | Unknown |
| 3 | GPT-5.6 Terra | 90.0% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 4 | Gemini 4 Argon | 89.8% | Task version 6; Evaluated 2026-10-09Original result | Not reported | Unknown |
| 5 | GPT-5.5 | 89.1% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
| 6 | Claude Sonnet 5.5 | 89.0% | Task version 6; Evaluated 2026-09-30Original result | Not reported | Unknown |
| 7 | Gemini 3 Flash Preview | 89.0% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 8 | GPT-5.6 Luna | 89.0% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 9 | Claude Opus 5 | 88.8% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 10 | Claude Sonnet 5 | 88.6% | Task version 6; Evaluated 2026-08-21Original result | Not reported | Unknown |
| 11 | Gemini 3.1 Pro Preview | 87.6% | Task version 6; Evaluated 2026-08-21Original result | Not reported | Unknown |
| 12 | Gemini 3.8 Flash | 87.1% | Task version 6; Evaluated 2026-09-02Original result | Not reported | Unknown |
| 13 | GPT-5.4 mini | 86.4% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
| 14 | Gemini 3.5 Flash | 85.6% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 15 | Gemini 3.7 Flash | 85.5% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 16 | Claude Haiku 4.5 | 81.6% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 17 | Gemma 4 31B | 79.9% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
| 18 | Gemini 3.5 Flash-Lite | 79.4% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 19 | Gemma 4 26B A4B | 77.8% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
| 20 | GPT-5.4 nano | 68.9% | Task version 6; Evaluated 2026-08-17Original result | Not reported | Unknown |
| 21 | Gemini 3.6 Flash | 66.3% | Task version 6; Evaluated 2026-08-18Original result | Not reported | Unknown |
How to read this leaderboard
A higher overall value F1 means more correct values across all documents. Check the long-document column before trusting a high overall figure: several vision-language models score above 90 on short documents and under 35 on long ones because they truncate record lists. Grounding of zero means the system returned no usable evidence, not that it was wrong.
Operator receipt: 49 sourced rows are currently displayable on this page; the leading published row is LlamaExtract Agentic Plus at 96.17%.
Honest limit: Coding-agent rows are agent products, not plain model calls, and their cost reflects the harness. Open-weight rows list no cost. The publisher sells LlamaExtract, which leads the board; the dataset and evaluation code are public. The "Evidence" variants ask the agent for source evidence and are separate rows from the plain runs.
How we show ExtractBench
LlamaIndex gives each system a document and a JSON Schema and scores the values it returns. Value F1 counts every record, so truncating a long list costs recall; grounding is scored at word and page level against the cited evidence. The set has 370 documents and 4,869 pages across 67 document types. We mirror the leaderboard CSV the public site loads: 49 configurations across 47 systems.
Coding agents such as Codex and Claude Code appear as extraction systems: the agent reads the document and returns the schema. We map each run to the model it used and name the harness in the row; the cost reflects the harness, not a single model call. Check the long-document column before trusting an overall figure, since several vision-language models truncate record lists.
This is LlamaIndex's ExtractBench, not the unrelated Contextual AI benchmark of the same name. LlamaIndex sells LlamaExtract, which leads the board; the dataset and evaluation code are public. The table is display only and does not enter model rankings. Kaggle Benchmarks reruns the task through its own harness with different numbers; that board is the separate table below.
Snapshot
The published ExtractBench (LlamaIndex) snapshot places LlamaExtract Agentic Plus first at 96.17%. The third row is 0.10 points behind. The broader top-10 range is 2.60 points, so many of the published results sit in a relatively narrow band.
49 configurations have been evaluated on ExtractBench (LlamaIndex). The benchmark falls in the Multimodal & Grounded category. ExtractBench (LlamaIndex) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About ExtractBench (LlamaIndex)
Year
2026
Tasks
370 documents, 4,869 pages, 8 business domains, 67 document types, each with a JSON Schema
Format
Order-insensitive value F1 overall and by document length; word and page grounding F1; cost and latency
Difficulty
Short documents are near solved; long record lists and grounding separate systems
Each ExtractBench document comes with a JSON Schema; the system returns structured values with source evidence. Value accuracy is an order-insensitive value F1 that counts every record in long lists, so truncating a list costs recall. Grounding is scored at word level and page level against the evidence the system cites. Documents are grouped as short, medium and long, and each row reports cost per page and latency per document. The board covers LlamaExtract, specialized extraction APIs, coding agents such as Codex and Claude Code running the extraction as a task, commercial vision-language models and open-weight models. We mirror the leaderboard CSV from the ExtractBench repository, which the public site loads at runtime, and store each catalog model's best overall value F1 on its row, including runs made through a coding-agent harness. The ExtractBench name is also used by Contextual AI for an unrelated benchmark; this page is LlamaIndex's.
Freshness and provenance
Version
ExtractBench leaderboard (LlamaIndex repository CSV)
Refresh cadence
Repository leaderboard updates
Staleness state
Current
Question availability
The 370-document dataset, schemas and evaluation code are public under Apache 2.0
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does ExtractBench measure?
ExtractBench gives a system a document and a JSON Schema and scores the values it returns. Value F1 counts every record, so a truncated list loses recall. Grounding checks whether the cited evidence points at the right words and pages. Cost per page and latency per document are reported with every row.
Why do coding agents appear on a document benchmark?
LlamaIndex ran Codex and Claude Code as extraction systems: the agent reads the document, writes code where it helps and returns the schema. Those rows are accurate but cost far more per page than an extraction API. We map each run to the model it used and name the harness in the row note.
Is this the same ExtractBench as Contextual AI's?
No. Contextual AI publishes an unrelated benchmark under the same name. This page mirrors LlamaIndex's ExtractBench, with its own dataset, schema-guided task and leaderboard. Kaggle Benchmarks also reruns the LlamaIndex task through its own harness; that board is shown as a separate table on this page because its numbers differ.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.