# ExtractBench Pro: long-tail document extraction challenges (LlamaIndex) (ExtractBench Pro)

> A harder companion to ExtractBench: 100 synthetic documents and 2,622 pages across 86 document types that test document conventions, house vocabulary, counting rules, drawn relations and charts, scored by perfect-document rate.

Canonical page: https://benchlm.ai/benchmarks/extractbench-pro

- Category: [Multimodal & Grounded](/multimodal-grounded)
- Last updated: October 25, 2026 snapshot

## About ExtractBench Pro

- Year: 2026
- Tasks: 100 synthetic documents, 2,622 pages, 86 document types, six challenge groups
- Format: Perfect-document rate, value F1, word and page grounding F1 and cost, overall and per challenge
- Difficulty: The best system completes 58 of 100 documents perfectly; several complete none
- Paper: [ExtractBench Pro leaderboard and challenge descriptions](https://www.extractbench.ai/pro)

ExtractBench Pro is a separate dataset and leaderboard that extends ExtractBench to the long tail of enterprise documents. Its six challenges are document conventions (22 documents), house vocabulary (14), house counting rules (12), drawn relations (20), charts inside documents (25) and long chart sheets (7). A document counts as perfect only when every value matches, so the headline metric is the share of the 100 documents extracted with 100% value F1; value F1, word and page grounding F1 and cost per page are reported alongside. Systems ran under a 30-minute limit per document, and documents over the limit score zero. We mirror the Pro CSV from the ExtractBench repository, which carries the overall row and all six challenge rows per system, and store each catalog model's perfect-document rate on its row.

ExtractBench Pro is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (13 systems)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [GPT-6 Astra](/models/gpt-6-astra) | Coding agent · Codex harness · value F1 94.53% · 95.20¢ per page | OpenAI | 58% |
| 2 | [Claude Opus 5.5](/models/claude-opus-5-5) | Coding agent · Claude Code harness · value F1 74.87% · 41.20¢ per page | Anthropic | 51% |
| 3 | [LlamaExtract Agentic Plus](https://www.extractbench.ai/pro#leaderboard) | LlamaExtract · Default · value F1 77.70% · 7.50¢ per page | LlamaIndex | 38% |
| 4 | [Extend Operator](https://www.extractbench.ai/pro#leaderboard) | Specialized extraction API · Default · value F1 73.87% · 9.88¢ per page | Extend | 34% |
| 5 | [Claude Sonnet 5.5](/models/claude-sonnet-5-5) | Coding agent · Claude Code harness · value F1 73.62% · 19.98¢ per page | Anthropic | 32% |
| 6 | [Gemini 4 Argon](/models/gemini-4-argon) | Coding agent · Default · value F1 42.98% · 36.16¢ per page | Google | 30% |
| 7 | [LlamaExtract Agentic](https://www.extractbench.ai/pro#leaderboard) | LlamaExtract · Default · value F1 71.32% · 3.12¢ per page | LlamaIndex | 20% |
| 8 | [Pulse Effort](https://www.extractbench.ai/pro#leaderboard) | Specialized extraction API · Default · value F1 70.53% · 10.50¢ per page | Pulse | 18% |
| 9 | [GPT-6.1 Sol](/models/gpt-6-1-sol) | Coding agent · Codex harness · value F1 74.06% · 13.86¢ per page | OpenAI | 12% |
| 10 | [Reducto Deep](https://www.extractbench.ai/pro#leaderboard) | Specialized extraction API · Default · value F1 56.35% · 6.18¢ per page | Reducto | 3% |
| 11 | [Datalab Accurate](https://www.extractbench.ai/pro#leaderboard) | Specialized extraction API · Default · value F1 43.93% · 15.75¢ per page | Datalab | 2% |
| 12 | [GPT-6 Luna](/models/gpt-6-luna) | Coding agent · Codex harness · value F1 48.75% · 1.14¢ per page | OpenAI | 0% |
| 13 | [LlamaExtract Cost Effective](https://www.extractbench.ai/pro#leaderboard) | LlamaExtract · Default · value F1 47.19% · 1.00¢ per page | LlamaIndex | 0% |

## FAQ

### How does ExtractBench Pro differ from ExtractBench?

ExtractBench scores value accuracy on real enterprise documents. Pro is a separate synthetic set built around failure modes those documents rarely show: house vocabulary, counting rules, relations that are drawn rather than written, and charts. Its headline number is the share of documents extracted with no error at all.

### Why is the headline metric a perfect-document rate?

A document with one wrong value is unusable for the workflows Pro models, such as a filing where one holding is miscounted. The perfect-document rate counts only fully correct documents, which is why it sits far below value F1 for every system on the board, including the leader.

### Why is ExtractBench Pro display only?

The launch board has 12 systems, five of them coding agents whose cost and behavior reflect the harness, and the documents are synthetic. The publisher sells one of the systems. We show every row with its challenge breakdown, but the results compare those exact runs and do not enter BenchLM's model rankings.