# Multi-IF

> Multi-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models.

Canonical page: https://benchlm.ai/benchmarks/multi-if

- Category: [Instruction Following](/instruction-following)
- Last updated: Paper v2, November 13, 2024; checked October 1, 2026

Evaluation results published by [Multi-IF authors](https://arxiv.org/html/2410.15553v2). Selected author results on 4,501 three-turn conversations across eight languages.

The original article is CC BY-NC-SA 4.0. This page independently reports selected numerical results with its own explanation and layout; it does not reproduce the paper table or task contents. [Original article terms](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.en).

This selected report keeps the author paper's three turn scores separate for four dated API models. Each value averages instruction-level strict, conversation-level strict, instruction-level loose, and conversation-level loose accuracy, then averages the eight language scores. Values use the paper's native 0-1 scale.

The third-turn value is shown first for sorting. It is a third-turn result, not a mean across all three turns. The rows use the main Table 1 evaluation; the paper's later false-refusal filtering experiment uses a separate protocol.

The API version and Appendix A generation settings stay beside each row. The o1-preview row uses temperature 1, top-p 1, and a 25,000-token cap; the other selected rows use temperature 0.6, top-p 0.9, and a 1,024-token cap. Reasoning effort and the evaluation date are undisclosed. November 13, 2024 is the paper revision date.

This selected 2024 cohort uses different generation settings; turn 3 is separate from the other turns and does not establish current API performance.

- [Original author results, Table 1](https://arxiv.org/html/2410.15553v2#S3.T1)
- [Source API versions and generation settings](https://arxiv.org/html/2410.15553v2#A1)
- [Paper v2 and article license declaration](https://arxiv.org/abs/2410.15553v2)
- [Paper CC BY-NC-SA 4.0 terms](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.en)

## About Multi-IF

- Year: 2024
- Tasks: 4,501 three-turn conversations in eight languages
- Format: Turn-specific accuracy average on a 0-1 scale
- Paper: [Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following](https://arxiv.org/abs/2410.15553v2)

The four metrics are instruction-level strict, conversation-level strict, instruction-level loose, and conversation-level loose accuracy. Turn 3 is separate from the other turns; it is not an all-turn mean. The main Table 1 results retain false refusals. API snapshots are explicit, but evaluation dates and reasoning efforts are unreported. This source cohort stays display only.

Multi-IF is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (4 dated API model reports)

| Source model | Evaluated API version | Turn 1 | Turn 2 | Turn 3 | Generation settings | Original source |
|---|---|---|---|---|---|---|
| [o1-preview](/models/o1-preview) | o1-preview-2024-09-12 | 0.877 | 0.783 | 0.707 | T=1; top-p=1; max new tokens=25000 | [Author result](https://arxiv.org/html/2410.15553v2#S3.T1); [API versions and settings](https://arxiv.org/html/2410.15553v2#A1) |
| [GPT-4o](/models/gpt-4o) | gpt-4o-2024-08-06 | 0.843 | 0.724 | 0.631 | T=0.6; top-p=0.9; max new tokens=1024 | [Author result](https://arxiv.org/html/2410.15553v2#S3.T1); [API versions and settings](https://arxiv.org/html/2410.15553v2#A1) |
| [GPT-4](/models/gpt-4-turbo) | gpt-4-turbo-2024-04-09 | 0.815 | 0.705 | 0.609 | T=0.6; top-p=0.9; max new tokens=1024 | [Author result](https://arxiv.org/html/2410.15553v2#S3.T1); [API versions and settings](https://arxiv.org/html/2410.15553v2#A1) |
| [Claude-3-Haiku](/models/claude-3-haiku) | claude-3-haiku-20240307 | 0.729 | 0.632 | 0.542 | T=0.6; top-p=0.9; max new tokens=1024 | [Author result](https://arxiv.org/html/2410.15553v2#S3.T1); [API versions and settings](https://arxiv.org/html/2410.15553v2#A1) |

Each native 0-1 value averages four accuracy metrics across eight languages for that turn. Sorting uses turn 3; it is not an all-turn mean. The paper revision is dated November 13, 2024; evaluation dates and reasoning efforts are undisclosed. Profile links refer to named generations for navigation.

## FAQ

### What does Multi-IF measure?

Multi-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models.

### Which source row has the highest reported Multi-IF value?

o1-preview has the highest reported Multi-IF value at 0.707. This selected 2024 cohort uses different generation settings; turn 3 is separate from the other turns and does not establish current API performance.

### How many models are evaluated on Multi-IF?

The Paper v2, November 13, 2024; checked October 1, 2026 contains 4 AI models.

### Does Multi-IF affect BenchLM's overall score?

Not directly. Multi-IF is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.