ExtractBench Pro: long-tail document extraction challenges (LlamaIndex) (ExtractBench Pro)
We show this table for reference; we do not rank on it.
A harder companion to ExtractBench: 100 synthetic documents and 2,622 pages across 86 document types that test document conventions, house vocabulary, counting rules, drawn relations and charts, scored by perfect-document rate.
Perfect-document rate on ExtractBench Pro — October 25, 2026 snapshot
We mirror the published perfect-document rate view for ExtractBench Pro. GPT-6 Astra leads the public snapshot at 58%, followed by Claude Opus 5.5 (51%) and LlamaExtract Agentic Plus (38%). We do not use these results to rank models overall.
GPT-6 Astra
OpenAI
Coding agent · Codex harness · value F1 94.53% · 95.20¢ per page
Claude Opus 5.5
Anthropic
Coding agent · Claude Code harness · value F1 74.87% · 41.20¢ per page
LlamaExtract Agentic Plus
LlamaIndex
LlamaExtract · Default · value F1 77.70% · 7.50¢ per page
13 systemsMultimodal & GroundedCurrentDisplay onlyUpdated October 25, 2026 snapshot
Perfect-document rate table (13 systems)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | GPT-6 AstraOpenAICoding agent · Codex harness · value F1 94.53% · 95.20¢ per page | 58% | Not reported | Closed |
| 2 | Claude Opus 5.5AnthropicCoding agent · Claude Code harness · value F1 74.87% · 41.20¢ per page | 51% | Not reported | Closed |
| 3 | LlamaExtract Agentic PlusLlamaIndexLlamaExtract · Default · value F1 77.70% · 7.50¢ per page | 38% | Not reported | Unknown |
| 4 | Extend OperatorExtendSpecialized extraction API · Default · value F1 73.87% · 9.88¢ per page | 34% | Not reported | Unknown |
| 5 | Claude Sonnet 5.5AnthropicCoding agent · Claude Code harness · value F1 73.62% · 19.98¢ per page | 32% | Not reported | Closed |
| 6 | Gemini 4 ArgonGoogleCoding agent · Default · value F1 42.98% · 36.16¢ per page | 30% | Not reported | Closed |
| 7 | LlamaExtract AgenticLlamaIndexLlamaExtract · Default · value F1 71.32% · 3.12¢ per page | 20% | Not reported | Unknown |
| 8 | Pulse EffortPulseSpecialized extraction API · Default · value F1 70.53% · 10.50¢ per page | 18% | Not reported | Unknown |
| 9 | GPT-6.1 SolOpenAICoding agent · Codex harness · value F1 74.06% · 13.86¢ per page | 12% | Not reported | Closed |
| 10 | Reducto DeepReductoSpecialized extraction API · Default · value F1 56.35% · 6.18¢ per page | 3% | Not reported | Unknown |
| 11 | Datalab AccurateDatalabSpecialized extraction API · Default · value F1 43.93% · 15.75¢ per page | 2% | Not reported | Unknown |
| 12 | GPT-6 LunaOpenAICoding agent · Codex harness · value F1 48.75% · 1.14¢ per page | 0% | Not reported | Closed |
| 13 | LlamaExtract Cost EffectiveLlamaIndexLlamaExtract · Default · value F1 47.19% · 1.00¢ per page | 0% | Not reported | Unknown |
How to read this leaderboard
The score is the share of documents with every value right, so it is far stricter than value F1: a system can reach 77% value F1 with 38% of documents perfect. A zero means no document was extracted without error, not that nothing was extracted. Compare the per-challenge rows to see where a system fails.
Operator receipt: 13 sourced rows are currently displayable on this page; the leading published row is GPT-6 Astra at 58%.
Honest limit: The launch set has 12 systems, five of them coding agents whose cost reflects the harness rather than a model call. Documents are synthetic by design, and the 30-minute limit zeroes slow runs. The publisher sells LlamaExtract. The dataset and evaluation code are public.
How we show ExtractBench Pro
ExtractBench Pro is a separate synthetic set of 100 documents and 2,622 pages across 86 document types, built around six challenges: document conventions, house vocabulary, house counting rules, drawn relations, charts inside documents and long chart sheets. The shown value is the perfect-document rate, the share of documents extracted with every value right; value F1, grounding and cost per page travel with each row.
Systems ran under a 30-minute limit per document and a document over the limit scores zero. We mirror the Pro CSV from the ExtractBench repository, which carries the overall row and the six challenge rows for each of the 13 systems; coding-agent rows map to the model they ran with the harness named in the row.
The perfect-document rate is far stricter than value F1: the leader reaches 94.5% value F1 with 58% of documents perfect, and several systems complete none. LlamaIndex sells LlamaExtract, one of the systems; the dataset and evaluation code are public. The table is display only and does not enter model rankings.
Snapshot
The published ExtractBench Pro snapshot places GPT-6 Astra first at 58%. The third row is 20 points behind. The broader top-10 range is 55 points, so the table still separates the published systems.
13 systems have been evaluated on ExtractBench Pro. The benchmark falls in the Multimodal & Grounded category. ExtractBench Pro is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About ExtractBench Pro
Year
2026
Tasks
100 synthetic documents, 2,622 pages, 86 document types, six challenge groups
Format
Perfect-document rate, value F1, word and page grounding F1 and cost, overall and per challenge
Difficulty
The best system completes 58 of 100 documents perfectly; several complete none
ExtractBench Pro is a separate dataset and leaderboard that extends ExtractBench to the long tail of enterprise documents. Its six challenges are document conventions (22 documents), house vocabulary (14), house counting rules (12), drawn relations (20), charts inside documents (25) and long chart sheets (7). A document counts as perfect only when every value matches, so the headline metric is the share of the 100 documents extracted with 100% value F1; value F1, word and page grounding F1 and cost per page are reported alongside. Systems ran under a 30-minute limit per document, and documents over the limit score zero. We mirror the Pro CSV from the ExtractBench repository, which carries the overall row and all six challenge rows per system, and store each catalog model's perfect-document rate on its row.
Freshness and provenance
Version
ExtractBench Pro launch leaderboard (LlamaIndex repository CSV)
Refresh cadence
Repository leaderboard updates
Staleness state
Current
Question availability
The 100-document synthetic dataset and evaluation code are public under Apache 2.0
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
How does ExtractBench Pro differ from ExtractBench?
ExtractBench scores value accuracy on real enterprise documents. Pro is a separate synthetic set built around failure modes those documents rarely show: house vocabulary, counting rules, relations that are drawn rather than written, and charts. Its headline number is the share of documents extracted with no error at all.
Why is the headline metric a perfect-document rate?
A document with one wrong value is unusable for the workflows Pro models, such as a filing where one holding is miscounted. The perfect-document rate counts only fully correct documents, which is why it sits far below value F1 for every system on the board, including the leader.
Why is ExtractBench Pro display only?
The launch board has 12 systems, five of them coding agents whose cost and behavior reflect the harness, and the documents are synthetic. The publisher sells one of the systems. We show every row with its challenge breakdown, but the results compare those exact runs and do not enter BenchLM's model rankings.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.