Skip to main content
BenchLM
Data

MuirBench: Robust Multi-Image Understanding (MuirBench)

We show this table for reference; we do not rank on it.

A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.

Accuracy on MuirBench — MuirBench ICLR 2025 Table 1 (captured November 20, 2026)

We mirror the published accuracy view for MuirBench. GPT-4o leads the public snapshot at 68.0%, followed by GPT-4 Turbo (62.3%) and Gemini Pro (49.4%). We do not use these results to rank models overall.

20 modelsMultimodal & GroundedRefreshingDisplay onlyUpdated MuirBench ICLR 2025 Table 1 (captured November 20, 2026)

Accuracy table (20 models)

Accuracy results for MuirBench
RankModel / configurationScoreParameters (B)Open / closed
1GPT-4oOpenAI
68.0%
Not reportedClosed
2GPT-4 TurboOpenAI
62.3%
Not reportedClosed
3Gemini ProGoogle
49.4%
Not reportedUnknown
4Mantis-8B-Idefics2TIGER-Lab
44.5%
Not reportedUnknown
5Mantis-8B-clip-llama3TIGER-Lab
37.4%
Not reportedUnknown
6Mantis-8B-siglip-llama3TIGER-Lab
36.1%
Not reportedUnknown
7Idefics-9B-InstructHugging Face
35.4%
Not reportedUnknown
8Emu2-Chat (37B)BAAI
33.6%
Not reportedUnknown
9LLaVA-NeXT-34BLLaVA
33.3%
Not reportedUnknown
10LLaVA-v1.5-7B-xtunerInternLM
33.2%
Not reportedUnknown
11VILA1.5-13BNVIDIA
33.1%
Not reportedUnknown
12Yi-VL-6B01.AI
28.7%
Not reportedUnknown
13LLaVA-internLM2-7BInternLM
28.1%
Not reportedUnknown
14Idefics2-8BHugging Face
26.1%
Not reportedUnknown
15LLaVA-v1.5-13BLLaVA
24.4%
Not reportedUnknown
16OpenFlamingo-v2-9BLAION
23.7%
Not reportedUnknown
17LLaVA-v1.5-7BLLaVA
23.5%
Not reportedUnknown
18LLaVA-v1.5-13B-xtunerInternLM
21.7%
Not reportedUnknown
19CogVLMZ.ai
20.9%
Not reportedUnknown
20MiniGPT-4-v2KAUST
17.4%
Not reportedUnknown

Results reported by model developersUnranked

Labs ran these themselves. They take no rank here.

MuirBench results reported by model developers, unranked
Inkling 2 MiniThinking Machines Lab77.5%Overall accuracyReported in the Inkling 2 Mini launch tableNot part of the original MuirBench ICLR 2025 paper table.Comparability not establishedReported by Thinking Machines Lab

A lab’s own result is shown for reference. It never takes a rank on this board.

How we label sources

How to read this leaderboard

Operator receipt: 20 sourced rows are currently displayable on this page; the leading published row is GPT-4o at 68.0%.

Honest limit: The mirrored paper table covers 20 models from the original ICLR 2025 evaluation; newer lab-reported rows are shown separately because prompts and image preprocessing can differ across labs.

How we show MuirBench

We mirror Table 1 of the MuirBench ICLR 2025 paper (arXiv:2406.09411v2): 20 multimodal models evaluated on 2,600 pairwise multiple-choice questions across 11,264 images and 12 multi-image tasks.

For reference, the paper reports a human baseline of 93.15% and a random-choice baseline of 23.99%. Every answerable question is paired with a minimally altered unanswerable variant.

We keep MuirBench display only and exclude it from overall and category rankings. Newer models whose developers publish MuirBench numbers outside the paper table appear below under lab-reported results.

Snapshot

20 models2,600 questions12 multi-image tasksHuman 93.2%Display only

The published MuirBench snapshot places GPT-4o first at 68.0%. The third row is 18.6 points behind. The broader top-10 range is 34.8 points, so the table still separates the published systems.

20 models have been evaluated on MuirBench. The benchmark falls in the Multimodal & Grounded category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. MuirBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About MuirBench

Year

2024

Tasks

2,600 pairwise multiple-choice questions across 11,264 images and 12 multi-image tasks

Format

Accuracy (%)

Difficulty

Multi-image visual reasoning and robustness

Created by Wang et al. (ICLR 2025) to test temporal ordering, scene understanding, visual retrieval, geographic reasoning, and multi-panel diagram interpretation across multiple images. We mirror Table 1 of the ICLR 2025 paper (20 models; human baseline 93.15%, random 23.99%). Provider-reported results for newer models not yet on the paper table appear below under lab-reported results.

Freshness and provenance

Version

MuirBench 2024

Refresh cadence

Annual

Staleness state

Refreshing

Question availability

Public benchmark set

RefreshingDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does MuirBench measure?

A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.

Which model leads the published MuirBench snapshot?

GPT-4o currently leads the published MuirBench snapshot with 68.0% accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on MuirBench?

The MuirBench ICLR 2025 Table 1 (captured November 20, 2026) snapshot contains 20 AI models.

Last updated: MuirBench ICLR 2025 Table 1 (captured November 20, 2026) · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 7,000+ subscribers.

One email each week. Unsubscribe anytime.