MuirBench: Robust Multi-Image Understanding (MuirBench)
We show this table for reference; we do not rank on it.
A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.
Accuracy on MuirBench — MuirBench ICLR 2025 Table 1 (captured November 20, 2026)
We mirror the published accuracy view for MuirBench. GPT-4o leads the public snapshot at 68.0%, followed by GPT-4 Turbo (62.3%) and Gemini Pro (49.4%). We do not use these results to rank models overall.
GPT-4o
OpenAI
GPT-4 Turbo
OpenAI
Gemini Pro
20 modelsMultimodal & GroundedRefreshingDisplay onlyUpdated MuirBench ICLR 2025 Table 1 (captured November 20, 2026)
Accuracy table (20 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | GPT-4oOpenAI | 68.0% | Not reported | Closed |
| 2 | GPT-4 TurboOpenAI | 62.3% | Not reported | Closed |
| 3 | Gemini ProGoogle | 49.4% | Not reported | Unknown |
| 4 | Mantis-8B-Idefics2TIGER-Lab | 44.5% | Not reported | Unknown |
| 5 | Mantis-8B-clip-llama3TIGER-Lab | 37.4% | Not reported | Unknown |
| 6 | Mantis-8B-siglip-llama3TIGER-Lab | 36.1% | Not reported | Unknown |
| 7 | Idefics-9B-InstructHugging Face | 35.4% | Not reported | Unknown |
| 8 | Emu2-Chat (37B)BAAI | 33.6% | Not reported | Unknown |
| 9 | LLaVA-NeXT-34BLLaVA | 33.3% | Not reported | Unknown |
| 10 | LLaVA-v1.5-7B-xtunerInternLM | 33.2% | Not reported | Unknown |
| 11 | VILA1.5-13BNVIDIA | 33.1% | Not reported | Unknown |
| 12 | Yi-VL-6B01.AI | 28.7% | Not reported | Unknown |
| 13 | LLaVA-internLM2-7BInternLM | 28.1% | Not reported | Unknown |
| 14 | Idefics2-8BHugging Face | 26.1% | Not reported | Unknown |
| 15 | LLaVA-v1.5-13BLLaVA | 24.4% | Not reported | Unknown |
| 16 | OpenFlamingo-v2-9BLAION | 23.7% | Not reported | Unknown |
| 17 | LLaVA-v1.5-7BLLaVA | 23.5% | Not reported | Unknown |
| 18 | LLaVA-v1.5-13B-xtunerInternLM | 21.7% | Not reported | Unknown |
| 19 | CogVLMZ.ai | 20.9% | Not reported | Unknown |
| 20 | MiniGPT-4-v2KAUST | 17.4% | Not reported | Unknown |
Results reported by model developersUnranked
Labs ran these themselves. They take no rank here.
| Model | Reported value | What was measured | Against the official board | Who ran it |
|---|---|---|---|---|
| Inkling 2 MiniThinking Machines Lab | 77.5%Overall accuracy | Reported in the Inkling 2 Mini launch tableNot part of the original MuirBench ICLR 2025 paper table. | Comparability not established | Reported by Thinking Machines Lab |
A lab’s own result is shown for reference. It never takes a rank on this board.
How we label sourcesHow to read this leaderboard
Operator receipt: 20 sourced rows are currently displayable on this page; the leading published row is GPT-4o at 68.0%.
Honest limit: The mirrored paper table covers 20 models from the original ICLR 2025 evaluation; newer lab-reported rows are shown separately because prompts and image preprocessing can differ across labs.
How we show MuirBench
We mirror Table 1 of the MuirBench ICLR 2025 paper (arXiv:2406.09411v2): 20 multimodal models evaluated on 2,600 pairwise multiple-choice questions across 11,264 images and 12 multi-image tasks.
For reference, the paper reports a human baseline of 93.15% and a random-choice baseline of 23.99%. Every answerable question is paired with a minimally altered unanswerable variant.
We keep MuirBench display only and exclude it from overall and category rankings. Newer models whose developers publish MuirBench numbers outside the paper table appear below under lab-reported results.
Snapshot
The published MuirBench snapshot places GPT-4o first at 68.0%. The third row is 18.6 points behind. The broader top-10 range is 34.8 points, so the table still separates the published systems.
20 models have been evaluated on MuirBench. The benchmark falls in the Multimodal & Grounded category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. MuirBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About MuirBench
Year
2024
Tasks
2,600 pairwise multiple-choice questions across 11,264 images and 12 multi-image tasks
Format
Accuracy (%)
Difficulty
Multi-image visual reasoning and robustness
Created by Wang et al. (ICLR 2025) to test temporal ordering, scene understanding, visual retrieval, geographic reasoning, and multi-panel diagram interpretation across multiple images. We mirror Table 1 of the ICLR 2025 paper (20 models; human baseline 93.15%, random 23.99%). Provider-reported results for newer models not yet on the paper table appear below under lab-reported results.
Freshness and provenance
Version
MuirBench 2024
Refresh cadence
Annual
Staleness state
Refreshing
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does MuirBench measure?
A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.
Which model leads the published MuirBench snapshot?
GPT-4o currently leads the published MuirBench snapshot with 68.0% accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on MuirBench?
The MuirBench ICLR 2025 Table 1 (captured November 20, 2026) snapshot contains 20 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.