# MuirBench: Robust Multi-Image Understanding (MuirBench)

> A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.

Canonical page: https://benchlm.ai/benchmarks/muirbench

- Category: [Multimodal & Grounded](/multimodal-grounded)
- Last updated: MuirBench ICLR 2025 Table 1 (captured November 20, 2026)

## About MuirBench

- Year: 2024
- Tasks: 2,600 pairwise multiple-choice questions across 11,264 images and 12 multi-image tasks
- Format: Accuracy (%)
- Difficulty: Multi-image visual reasoning and robustness
- Paper: [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://arxiv.org/abs/2406.09411)

Created by Wang et al. (ICLR 2025) to test temporal ordering, scene understanding, visual retrieval, geographic reasoning, and multi-panel diagram interpretation across multiple images. We mirror Table 1 of the ICLR 2025 paper (20 models; human baseline 93.15%, random 23.99%). Provider-reported results for newer models not yet on the paper table appear below under lab-reported results.

MuirBench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (20 models)

| Rank | Model | Creator | Score |
|------|-------|---------|-------|
| 1 | [GPT-4o](/models/gpt-4o) | OpenAI | 68.0% |
| 2 | [GPT-4 Turbo](/models/gpt-4-turbo) | OpenAI | 62.3% |
| 3 | [Gemini Pro](https://arxiv.org/abs/2406.09411) | Google | 49.4% |
| 4 | [Mantis-8B-Idefics2](https://arxiv.org/abs/2406.09411) | TIGER-Lab | 44.5% |
| 5 | [Mantis-8B-clip-llama3](https://arxiv.org/abs/2406.09411) | TIGER-Lab | 37.4% |
| 6 | [Mantis-8B-siglip-llama3](https://arxiv.org/abs/2406.09411) | TIGER-Lab | 36.1% |
| 7 | [Idefics-9B-Instruct](https://arxiv.org/abs/2406.09411) | Hugging Face | 35.4% |
| 8 | [Emu2-Chat (37B)](https://arxiv.org/abs/2406.09411) | BAAI | 33.6% |
| 9 | [LLaVA-NeXT-34B](https://arxiv.org/abs/2406.09411) | LLaVA | 33.3% |
| 10 | [LLaVA-v1.5-7B-xtuner](https://arxiv.org/abs/2406.09411) | InternLM | 33.2% |
| 11 | [VILA1.5-13B](https://arxiv.org/abs/2406.09411) | NVIDIA | 33.1% |
| 12 | [Yi-VL-6B](https://arxiv.org/abs/2406.09411) | 01.AI | 28.7% |
| 13 | [LLaVA-internLM2-7B](https://arxiv.org/abs/2406.09411) | InternLM | 28.1% |
| 14 | [Idefics2-8B](https://arxiv.org/abs/2406.09411) | Hugging Face | 26.1% |
| 15 | [LLaVA-v1.5-13B](https://arxiv.org/abs/2406.09411) | LLaVA | 24.4% |
| 16 | [OpenFlamingo-v2-9B](https://arxiv.org/abs/2406.09411) | LAION | 23.7% |
| 17 | [LLaVA-v1.5-7B](https://arxiv.org/abs/2406.09411) | LLaVA | 23.5% |
| 18 | [LLaVA-v1.5-13B-xtuner](https://arxiv.org/abs/2406.09411) | InternLM | 21.7% |
| 19 | [CogVLM](https://arxiv.org/abs/2406.09411) | Z.ai | 20.9% |
| 20 | [MiniGPT-4-v2](https://arxiv.org/abs/2406.09411) | KAUST | 17.4% |

## Results reported by model developers

Unranked. Labs ran these themselves. A lab’s own result is shown for reference. It never takes a rank on this board.

| Model | Reported value | What was measured | Against the official board | Who ran it | Source |
|-------|----------------|-------------------|---------------------------|------------|--------|
| [Inkling 2 Mini](/models/inkling-2-mini) | 77.5% (Overall accuracy) | Reported in the Inkling 2 Mini launch table; not part of the original MuirBench ICLR 2025 paper table. | Comparability not established | Reported by Thinking Machines Lab | [Thinking Machines Lab Inkling 2 Mini launch post](https://thinkingmachines.ai/news/inkling-2-mini/) |

## FAQ

### What does MuirBench measure?

A multi-image understanding benchmark of 2,600 multiple-choice questions across 11,264 images and 12 tasks, pairing every answerable question with a minimally altered unanswerable variant.

### Which model leads the published MuirBench snapshot?

GPT-4o currently leads the published MuirBench snapshot with a score of 68.0%.

### How many models are evaluated on MuirBench?

The MuirBench ICLR 2025 Table 1 (captured November 20, 2026) contains 20 AI models.

### Does MuirBench affect BenchLM's overall score?

Not directly. MuirBench is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.
