Multi-IF
We show this table for reference; we do not rank on it.
Multi-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models.
Evaluation results published by Multi-IF authors. Selected author results on 4,501 three-turn conversations across eight languages.
The original article is CC BY-NC-SA 4.0. This page independently reports selected numerical results with its own explanation and layout; it does not reproduce the paper table or task contents. Original article terms.
Turn 3 average on Multi-IF — Paper v2, November 13, 2024; checked October 1, 2026
We mirror the published turn 3 average view for Multi-IF. o1-preview has the highest reported value at 0.707. This selected 2024 cohort uses different generation settings; turn 3 is separate from the other turns and does not establish current API performance. We do not use these results to rank models overall.
o1-preview
OpenAI
o1-preview-2024-09-12
GPT-4o
OpenAI
gpt-4o-2024-08-06
GPT-4
OpenAI
gpt-4-turbo-2024-04-09
4 dated API model reportsInstruction FollowingHistorical dated API cohortDisplay onlyUpdated Paper v2, November 13, 2024; checked October 1, 2026
| Source model and API version | Turn 1 | Turn 2 | Turn 3 | Generation settings and source | Parameters (B) | Open / closed |
|---|---|---|---|---|---|---|
| o1-previewo1-preview-2024-09-12Named generation reference | 0.877 | 0.783 | 0.707 | T=1; top-p=1; max new tokens=25000 Original author result · API versions and settings | Not reported | Closed |
| GPT-4ogpt-4o-2024-08-06Named generation reference | 0.843 | 0.724 | 0.631 | T=0.6; top-p=0.9; max new tokens=1024 Original author result · API versions and settings | Not reported | Closed |
| GPT-4gpt-4-turbo-2024-04-09Named generation reference | 0.815 | 0.705 | 0.609 | T=0.6; top-p=0.9; max new tokens=1024 Original author result · API versions and settings | Not reported | Closed |
| Claude-3-Haikuclaude-3-haiku-20240307Named generation reference | 0.729 | 0.632 | 0.542 | T=0.6; top-p=0.9; max new tokens=1024 Original author result · API versions and settings | Not reported | Closed |
Each native 0-1 value averages four accuracy metrics across eight languages for that turn. Sorting uses turn 3; it is not an all-turn mean. The paper revision is dated November 13, 2024; evaluation dates and reasoning efforts are undisclosed.
How we show these Multi-IF results
This selected report keeps the author paper's three turn scores separate for four dated API models. Each value averages instruction-level strict, conversation-level strict, instruction-level loose, and conversation-level loose accuracy, then averages the eight language scores. Values use the paper's native 0-1 scale.
The third-turn value is shown first for sorting. It is a third-turn result, not a mean across all three turns. The rows use the main Table 1 evaluation; the paper's later false-refusal filtering experiment uses a separate protocol.
The API version and Appendix A generation settings stay beside each row. The o1-preview row uses temperature 1, top-p 1, and a 25,000-token cap; the other selected rows use temperature 0.6, top-p 0.9, and a 1,024-token cap. Reasoning effort and the evaluation date are undisclosed. November 13, 2024 is the paper revision date.
Snapshot
This selected 2024 cohort uses different generation settings; turn 3 is separate from the other turns and does not establish current API performance. The table retains each source configuration and its reported value.
4 dated API model reports are shown for Multi-IF. The benchmark falls in the Instruction Following category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. Multi-IF is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Multi-IF
Year
2024
Tasks
4,501 three-turn conversations in eight languages
Format
Turn-specific accuracy average on a 0-1 scale
The four metrics are instruction-level strict, conversation-level strict, instruction-level loose, and conversation-level loose accuracy. Turn 3 is separate from the other turns; it is not an all-turn mean. The main Table 1 results retain false refusals. API snapshots are explicit, but evaluation dates and reasoning efforts are unreported. This source cohort stays display only.
Freshness and provenance
Version
arXiv:2410.15553v2 selected author reports
Refresh cadence
Pinned paper revision
Staleness state
Historical dated API cohort
Question availability
Public upstream; no task contents mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Multi-IF measure?
Multi-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models.
Which source row has the highest reported Multi-IF value?
o1-preview has the highest reported Multi-IF value at 0.707. This selected 2024 cohort uses different generation settings; turn 3 is separate from the other turns and does not establish current API performance. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on Multi-IF?
The Paper v2, November 13, 2024; checked October 1, 2026 snapshot contains 4 AI models.
Is the third-turn score an average over all three turns?
The third-turn score averages four accuracy metrics across eight languages for turn 3 only. The first- and second-turn columns remain separate. The paper does not publish an all-turn mean in Table 1.
Can a separately reported Multi-IF percentage be added to this table?
Its metric, turn and language aggregation, dataset version, and generation settings must be established first. A matching benchmark name alone does not show that the percentage measures the same result.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.