Belebele

We show this table for reference; we do not rank on it.

Belebele tests passage comprehension through questions with four answer options. Its parallel language variants let researchers study how reading performance changes across languages.

Original benchmark results

Belebele reports multiple-choice accuracy over 122 language variants, with 900 questions per variant. Few-shot, zero-shot, English fine-tuning, translated training, and translated tests keep separate configurations. All published per-language values are included. Summary and appendix averages sometimes disagree: GPT-3.5 is 51.1 versus 50.6, and the 91-language Llama-2-chat subset is 44.1 versus 44.0. Table 2 labels a Llama 1 row 70B while Table 7 identifies 65B; the profile uses the appendix identity and retains the conflicting label. BLOOMZ used FLORES material during training, which can advantage its evaluation.

7 source tables · 20 evaluated configurations · reviewed October 1, 2026. Download full results (JSON)

Paper summary across 122 language variants

Source vocabulary and parameter sizes are metadata, not scores. Source summary averages remain unchanged despite appendix discrepancies.

Published source

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants — Lucas Bandarkar and colleagues. CC BY SA 4.0. Copyright 2024 the authors. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.

Paper summary across 122 language variants. Published source metrics with original precision; unreported cells are not zero.
Model	Size or variant	Vocabulary size	Average accuracy (%)	Language share ≥50% accuracy (%)	Language share ≥70% accuracy (%)	English accuracy (%)	Non-English average (%)
5-Shot In-Context Learning (examples in English)
Llama 1	7B	32K	27.7	0.0%	0.0%	37.3	27.6
Llama 1	13B	32K	30.4	0.8%	0.0%	53.3	30.2
Llama 1	30B	32K	36.2	18.0%	0.8%	73.1	35.9
Llama 1	70B	32K	40.9	25.4%	12.3%	82.5	40.5
Llama 2 base	70B	32K	48.0	38.5%	26.2%	90.9	47.7
Falcon	40B	65K	37.3	16.4%	1.6%	77.2	36.9
Zero-Shot for Instructed Models (English instructions)
BLOOMZ**	7.1B	251K	43.2	28.7%	9.0%	79.6	42.9
Llama-2-chat	7B	32K	34.4	4.1%	0.0%	58.6	34.1
Llama-2-chat	70B	32K	41.5	27.0%	2.5%	78.8	41.2
GPT3.5-turbo	unk	100K	51.1	44.2%	29.2%	87.7	50.7
Full Finetuning in English
XLM-R	large (550M)	250K	54.0	64.8%	15.6%	76.2	53.8
XLM-V	large (1.2B)	902K	55.6	69.7%	21.2%	76.2	54.9
InfoXLM	large (550M)	250K	56.2	67.2%	28.7%	79.3	56.0
Translate-Train-All
XLM-R	large (550M)	250K	58.9	69.7%	36.1%	78.7	58.8
XLM-V	large (1.2B)	902K	60.2	76.2%	32.8%	77.8	60.1
InfoXLM	large (550M)	250K	60.0	70.5%	36.9%	81.2	59.8

All 122 language results: multilingual encoders

Accuracy values are percentages. PCT rows are the percentage of language variants meeting an accuracy threshold; summary and appendix PCT values differ and are both retained.

Published source

All 122 language results: multilingual encoders. Published source metrics with original precision; unreported cells are not zero.
Language or summary metric	XLM-V large (1.2B) (Belebele English fine-tuning)	InfoXLM large (550M) (Belebele English fine-tuning)	XLM-R large (550M) (Belebele English fine-tuning)	XLM-V large (1.2B) (Belebele Translate-Train-All)	InfoXLM large (550M) (Belebele Translate-Train-All)	XLM-R large (550M) (Belebele Translate-Train-All)
AVG	55.6	56.2	54.0	60.2	60.0	58.9
PCT Above 50	69.7%	67.2%	64.8%	76.2%	70.5%	69.7%
PCT Above 70	21.9%	28.9%	15.7%	33.1%	37.2%	36.4%
eng_Latn	76.2	79.3	76.2	77.8	81.2	78.7
acm_Arab	51.2	57.3	55.4	55.3	57.6	59.2
afr_Latn	69.3	72.7	69.1	72.3	75.1	74.3
als_Latn	68.4	68.9	64.9	70.8	72.2	71.4
amh_Ethi	53.1	52.9	52.6	61.6	60.0	60.7
apc_Arab	56.1	58.8	57.9	57.7	60.6	61.9
arb_Arab	67.2	71.0	69.8	70.6	75.0	74.3
arb_Latn	29.3	32.2	27.6	31.6	33.4	30.6
ars_Arab	55.6	59.9	58.9	61.1	65.8	65.9
ary_Arab	43.8	48.7	44.0	48.0	52.8	52.6
arz_Arab	56.9	60.2	57.6	61.4	64.9	66.1
asm_Beng	53.7	53.6	49.3	58.6	58.8	56.9
azj_Latn	59.7	61.3	59.0	65.0	65.6	65.1
bam_Latn	34.2	34.9	33.2	39.2	39.1	36.9
ben_Beng	60.0	63.4	59.6	65.6	69.6	63.7
ben_Latn	46.8	36.9	38.8	53.0	42.7	48.1
bod_Tibt	24.0	24.9	23.7	24.8	23.3	36.9
bul_Cyrl	72.6	72.0	70.1	74.0	75.3	74.2
cat_Latn	71.6	74.4	72.0	75.7	78.1	74.7
ceb_Latn	45.4	44.1	42.3	52.0	52.6	50.7
ces_Latn	69.9	72.3	69.9	72.3	76.2	74.4
ckb_Arab	29.7	52.3	30.3	36.9	58.0	36.9
dan_Latn	70.8	74.1	72.9	73.0	76.3	74.7
deu_Latn	72.6	75.7	72.9	74.1	78.7	76.7
ell_Grek	70.3	72.3	70.3	73.1	74.9	73.0
est_Latn	63.2	67.2	64.8	68.7	70.7	70.4
eus_Latn	63.6	66.1	64.8	68.2	70.8	70.3
fin_Latn	69.1	72.4	72.2	73.0	75.2	74.9
fra_Latn	73.1	74.2	72.1	74.6	76.8	75.6
fuv_Latn	29.7	27.7	26.4	32.8	30.7	31.1
gaz_Latn	48.8	33.8	36.4	52.6	36.0	43.3
grn_Latn	53.9	37.8	37.9	59.6	40.6	41.9
guj_Gujr	58.7	57.0	54.1	63.3	65.9	63.1
hat_Latn	57.1	39.6	35.2	63.2	44.1	39.8
hau_Latn	51.0	41.1	48.2	53.4	48.1	53.0
heb_Hebr	67.2	68.2	64.8	69.3	72.3	70.6
hin_Deva	57.9	60.2	57.4	63.8	64.3	63.4
hin_Latn	53.1	49.7	46.8	57.6	55.4	58.9
hrv_Latn	70.0	72.4	69.9	71.2	75.3	74.0
hun_Latn	69.7	70.8	70.0	73.1	74.2	72.8
hye_Armn	59.4	61.0	58.9	65.9	66.1	64.7
ibo_Latn	40.1	32.2	31.2	46.8	32.0	32.2
ilo_Latn	37.4	36.3	33.8	38.1	40.6	39.7
ind_Latn	68.9	70.7	68.0	71.3	73.1	70.4
isl_Latn	67.3	66.0	63.8	70.1	68.9	69.0
ita_Latn	70.6	72.8	70.0	71.8	76.4	73.3
jav_Latn	64.2	59.8	60.8	67.2	63.3	66.8
jpn_Jpan	66.4	70.1	67.6	71.3	71.8	71.0
kac_Latn	32.0	29.1	32.1	33.8	34.0	33.3
kan_Knda	61.1	62.0	59.7	66.6	68.4	69.1
kat_Geor	64.7	64.8	63.6	68.0	68.9	67.4
kaz_Cyrl	60.1	61.6	56.8	64.9	65.3	64.7
kea_Latn	44.0	45.2	44.9	48.7	47.7	48.1
khk_Cyrl	56.7	58.8	57.8	61.1	64.6	64.2
khm_Khmr	60.0	59.0	57.7	63.0	64.2	63.8
kin_Latn	35.9	33.6	34.3	39.1	39.1	38.6
kir_Cyrl	65.4	63.4	61.8	68.3	68.2	67.7
kor_Hang	70.1	71.4	68.7	72.9	74.6	74.8
lao_Laoo	55.8	57.6	53.0	63.2	63.6	63.0
lin_Latn	44.7	33.2	30.6	50.9	35.3	34.4
lit_Latn	68.3	69.4	67.2	71.7	72.9	72.0
lug_Latn	39.9	29.4	31.6	47.8	34.7	34.7
luo_Latn	30.3	30.9	30.8	33.7	34.9	33.2
lvs_Latn	70.1	71.3	68.7	74.1	75.6	73.0
mal_Mlym	62.0	65.0	62.7	69.1	68.3	67.1
mar_Deva	62.6	65.2	60.8	69.2	68.8	67.2
mkd_Cyrl	67.8	69.3	65.7	71.0	73.8	72.8
mlt_Latn	37.9	57.1	38.1	40.2	63.7	42.7
mri_Latn	32.0	30.6	32.2	33.0	35.7	34.0
mya_Mymr	56.6	59.1	53.6	62.2	65.1	62.9
nld_Latn	68.4	71.7	71.0	68.6	74.0	72.8
nob_Latn	71.8	73.6	70.7	72.8	75.4	74.2
npi_Deva	58.4	60.7	55.7	64.4	65.8	62.7
npi_Latn	38.3	35.8	33.8	37.4	36.4	34.8
nso_Latn	45.9	31.3	30.0	53.2	34.1	34.7
nya_Latn	31.0	29.2	29.8	34.2	33.0	30.8
ory_Orya	60.8	62.1	58.6	65.6	65.4	63.9
pan_Guru	58.1	59.2	57.8	63.1	62.6	62.0
pbt_Arab	55.4	56.0	51.0	60.6	62.6	61.1
pes_Arab	68.3	69.1	68.2	70.8	73.6	72.0
plt_Latn	55.7	45.6	52.7	61.7	53.4	58.1
pol_Latn	69.0	70.4	67.4	72.1	73.7	72.7
por_Latn	70.9	74.3	70.6	73.8	77.1	74.0
ron_Latn	72.3	72.9	71.3	74.0	76.2	74.8
rus_Cyrl	71.9	73.8	72.2	75.4	76.8	77.1
shn_Mymr	26.9	25.2	26.3	25.0	26.4	27.0
sin_Latn	24.9	34.2	30.7	41.7	38.3	37.3
sin_Sinh	64.4	67.2	62.7	69.8	70.2	68.6
slk_Latn	69.3	71.9	70.2	72.6	76.7	73.0
slv_Latn	69.7	72.2	68.6	71.8	75.4	73.9
sna_Latn	34.8	37.2	33.2	37.1	38.6	35.9
snd_Arab	55.2	56.6	51.9	60.0	61.3	61.3
som_Latn	46.0	39.1	42.6	50.7	46.3	50.7
sot_Latn	46.8	29.3	31.3	52.0	31.9	32.7
spa_Latn	71.0	73.3	71.4	72.7	75.3	76.4
srp_Cyrl	71.0	70.9	71.1	73.6	76.1	75.9
ssw_Latn	39.8	30.6	34.3	47.1	34.3	38.9
sun_Latn	60.9	50.7	55.3	64.2	55.8	59.4
swe_Latn	73.0	75.0	74.2	74.2	76.9	75.1
swh_Latn	64.9	65.3	62.8	69.3	69.2	68.7
tam_Taml	61.8	64.6	61.7	67.4	69.4	65.3
tel_Telu	55.6	57.8	53.6	62.1	63.2	61.1
tgk_Cyrl	38.2	58.6	33.8	39.2	64.3	39.6
tgl_Latn	69.2	67.4	64.7	72.0	70.4	70.0
tha_Thai	63.8	68.1	65.8	69.0	68.9	70.1
tir_Ethi	33.3	36.7	33.8	39.9	42.1	37.7
tsn_Latn	49.0	35.0	30.8	49.8	35.7	34.3
tso_Latn	37.9	36.3	34.2	41.7	39.7	37.1
tur_Latn	66.7	70.2	66.8	70.6	72.0	72.0
ukr_Cyrl	70.4	70.9	71.0	72.3	74.9	75.0
urd_Arab	61.6	63.8	59.3	65.6	68.6	66.3
urd_Latn	42.2	42.6	40.8	49.4	48.9	48.4
uzn_Latn	65.2	66.9	64.4	69.1	70.6	70.2
vie_Latn	69.6	71.1	69.4	73.7	72.9	71.4
war_Latn	46.4	44.7	43.7	47.6	49.3	46.6
wol_Latn	36.8	32.2	30.4	40.6	32.3	32.2
xho_Latn	48.7	36.1	39.0	54.4	40.2	45.4
yor_Latn	35.0	29.3	28.7	38.6	32.0	27.9
zho_Hans	69.8	74.6	71.0	73.7	76.2	74.8
zho_Hant	69.2	72.4	67.1	73.1	74.3	71.3
zsm_Latn	69.1	72.6	69.9	72.4	73.3	72.2
zul_Latn	46.9	36.4	39.0	54.2	39.8	44.1

Multilingual encoder training settings

These are training settings, not benchmark scores. XLM-R Translate-Train-All learning rate is printed as 3-6 in the source; no exponent is inferred.

Published source

Multilingual encoder training settings. Published source metrics with original precision; unreported cells are not zero.
Training setting	XLM-V large (1.2B) (Belebele English fine-tuning)	InfoXLM large (550M) (Belebele English fine-tuning)	XLM-R large (550M) (Belebele English fine-tuning)	XLM-V large (1.2B) (Belebele Translate-Train-All)	InfoXLM large (550M) (Belebele Translate-Train-All)	XLM-R large (550M) (Belebele Translate-Train-All)
epochs	3	4	3	1	1	1
training set size	67.5k	67.5k	67.5k	650k	650k	650k
learning rate	5e-6	4e-6	5e-6	3-6	3e-6	3e-6
weight decay	0.01	0.01	0.01	0.001	0.001	0.001
batch size	64	64	64	64	64	64

All 122 language results: large language models

The GPT-3.5 appendix average is 50.6; the summary prints 51.1. The Llama 1 checkpoint is labeled 65B in this appendix.

Published source

All 122 language results: large language models. Published source metrics with original precision; unreported cells are not zero.
Language or summary metric	GPT3.5-turbo unk (Belebele zero-shot)	Llama-2-chat 70B (Belebele zero-shot)	Llama 2 base 70B (Belebele 5-shot)	Llama 1 65B (Belebele 5-shot)	Falcon 40B (Belebele 5-shot)	XLM-V large (1.2B) (Belebele Translate-Train-All)
AVG	50.6	41.5	48.0	40.9	37.3	60.2
PCT Above 50	43.4 %	27.1%	38.5 %	25.4%	16.4%	76.2%
PCT Above 70	28.9 %	2.5%	26.2 %	12.3%	1.6%	33.1%
eng_Latn	87.7	78.8	90.9	82.5	77.2	77.8
acm_Arab	51.6	35.9	47.9	37.9	37.6	55.3
afr_Latn	78.3	57.9	75.9	60.7	53.4	72.3
als_Latn	67.1	36.0	45.4	34.9	36.6	70.8
amh_Ethi	28.7	28.9	27.5	27.8	24.8	61.6
apc_Arab	55.6	38.8	51.2	39.6	36.3	57.7
arb_Arab	69.3	42.3	61.7	44.1	38.3	70.6
arb_Latn	31.1	30.2	26.8	28.0	26.3	31.6
ars_Arab	55.1	37.4	50.2	40.7	32.1	61.1
ary_Arab	45.7	32.6	40.6	33.1	32.3	48.0
arz_Arab	56.7	37.3	50.7	37.4	33.0	61.4
asm_Beng	36.0	35.7	32.3	28.9	22.4	58.6
azj_Latn	54.9	33.4	42.2	33.6	34.1	65.0
bam_Latn	31.7	29.4	30.3	28.4	29.7	39.2
ben_Beng	43.6	34.9	39.1	33.4	22.6	65.6
ben_Latn	34.6	30.4	29.6	29.2	32.1	53.0
bod_Tibt	26.6	28.3	25.7	24.9	26.8	24.8
bul_Cyrl	76.0	65.0	80.4	69.3	41.9	74.0
cat_Latn	78.4	68.2	84.6	76.3	58.8	75.7
ceb_Latn	53.3	40.6	50.4	38.9	39.2	52.0
ces_Latn	76.9	65.0	81.1	70.7	65.0	72.3
ckb_Arab	31.8	32.8	28.7	31.6	28.9	36.9
dan_Latn	80.7	66.2	83.6	73.6	56.2	73.0
deu_Latn	83.3	69.4	84.6	76.0	70.1	74.1
ell_Grek	73.0	50.7	64.9	44.2	31.2	73.1
est_Latn	73.1	36.6	53.0	36.3	34.9	68.7
eus_Latn	40.9	31.1	34.7	32.8	38.9	68.2
fin_Latn	77.9	62.7	79.3	55.7	42.8	73.0
fra_Latn	83.1	72.2	86.4	77.5	69.7	74.6
fuv_Latn	26.1	29.8	24.9	25.4	25.1	32.8
gaz_Latn	30.3	29.3	27.8	29.1	24.9	52.6
grn_Latn	34.2	32.2	32.4	30.3	33.8	59.6
guj_Gujr	38.4	31.1	27.1	25.7	24.7	63.3
hat_Latn	51.6	34.1	37.4	33.7	36.2	63.2
hau_Latn	32.2	32.1	28.0	26.4	28.9	53.4
heb_Hebr	64.2	41.4	54.9	41.4	31.1	69.3
hin_Deva	49.1	42.0	52.6	38.4	27.1	63.8
hin_Latn	52.3	39.2	49.0	34.2	40.0	57.6
hrv_Latn	78.4	64.7	79.8	66.9	48.7	71.2
hun_Latn	74.6	61.1	78.8	66.7	37.7	73.1
hye_Armn	35.0	31.9	34.1	32.1	25.4	65.9
ibo_Latn	28.4	30.1	27.4	25.3	30.2	46.8
ilo_Latn	37.1	33.2	36.6	32.1	35.1	38.1
ind_Latn	74.2	61.3	81.4	55.7	52.1	71.3
isl_Latn	62.3	38.0	54.3	42.1	36.4	70.1
ita_Latn	80.0	68.6	84.5	76.1	66.4	71.8
jav_Latn	46.7	37.0	40.3	33.0	36.8	67.2
jpn_Jpan	70.9	56.6	77.6	53.9	49.6	71.3
kac_Latn	30.9	30.7	27.7	28.6	27.8	33.8
kan_Knda	40.6	21.9	25.7	24.4	24.0	66.6
kat_Geor	33.0	34.6	37.8	34.3	23.4	68.0
kaz_Cyrl	35.0	32.4	29.3	32.4	32.6	64.9
kea_Latn	46.0	38.1	45.4	38.1	38.0	48.7
khk_Cyrl	32.0	31.1	29.8	28.4	27.4	61.1
khm_Khmr	30.4	30.6	27.0	28.2	25.0	63.0
kin_Latn	35.2	30.6	29.8	28.5	31.9	39.1
kir_Cyrl	37.9	32.2	34.6	32.5	31.9	68.3
kor_Hang	67.1	56.3	77.8	52.9	40.2	72.9
lao_Laoo	30.0	26.5	24.3	26.2	28.1	63.2
lin_Latn	33.8	31.0	28.0	30.4	29.3	50.9
lit_Latn	72.0	39.7	52.1	39.6	39.3	71.7
lug_Latn	28.4	30.9	29.2	28.3	28.9	47.8
luo_Latn	27.1	31.2	29.4	29.3	29.9	33.7
lvs_Latn	70.8	41.0	51.3	39.0	37.6	74.1
mal_Mlym	34.9	30.1	32.4	30.0	21.2	69.1
mar_Deva	38.3	34.8	41.2	32.9	25.0	69.2
mkd_Cyrl	69.4	55.7	72.5	56.2	38.1	71.0
mlt_Latn	44.8	36.2	44.9	36.7	35.4	40.2
mri_Latn	33.3	31.8	28.5	32.0	29.7	33.0
mya_Mymr	30.3	31.3	24.1	24.2	22.6	62.2
nld_Latn	80.4	66.2	82.2	73.3	66.7	68.6
nob_Latn	79.0	65.7	81.8	70.9	60.8	72.8
npi_Deva	40.4	32.9	40.4	33.0	25.4	64.4
npi_Latn	35.1	30.4	30.2	30.0	30.9	37.4
nso_Latn	33.6	30.1	30.4	27.4	29.3	53.2
nya_Latn	33.2	29.3	27.3	28.7	29.3	34.2
ory_Orya	Unreported	29.2	24.8	23.9	23.7	65.6
pan_Guru	39.1	33.1	26.3	27.1	23.4	63.1
pbt_Arab	32.3	30.2	30.8	29.4	29.4	60.6
pes_Arab	61.8	41.8	53.9	41.0	35.9	70.8
plt_Latn	32.3	30.5	29.6	31.0	31.4	61.7
pol_Latn	74.7	61.7	79.2	67.0	59.9	72.1
por_Latn	83.0	70.2	86.1	75.4	68.3	73.8
ron_Latn	77.4	65.6	83.4	73.2	66.6	74.0
rus_Cyrl	78.4	67.0	82.7	73.1	48.1	75.4
shn_Mymr	Unreported	28.2	25.6	22.7	24.0	25.0
sin_Latn	30.4	31.9	33.8	27.9	32.6	41.7
sin_Sinh	32.6	33.4	25.2	29.4	27.7	69.8
slk_Latn	77.3	58.8	75.2	60.4	57.0	72.6
slv_Latn	77.4	62.4	76.7	65.6	43.7	71.8
sna_Latn	35.4	30.2	27.4	28.3	31.6	37.1
snd_Arab	34.1	29.7	30.9	28.9	30.2	60.0
som_Latn	32.4	30.3	27.8	27.6	29.9	50.7
sot_Latn	33.9	30.0	28.9	26.8	29.9	52.0
spa_Latn	79.2	68.4	85.0	74.8	69.2	72.7
srp_Cyrl	74.8	65.1	81.0	70.7	40.2	73.6
ssw_Latn	32.0	30.7	27.7	28.0	30.1	47.1
sun_Latn	38.9	34.9	37.8	30.7	34.1	64.2
swe_Latn	81.7	67.4	82.7	73.7	67.3	74.2
swh_Latn	70.3	35.1	39.6	34.4	36.7	69.3
tam_Taml	32.8	34.4	33.2	31.6	24.4	67.4
tel_Telu	34.6	27.5	25.9	26.6	22.4	62.1
tgk_Cyrl	37.7	32.5	34.0	33.1	32.7	39.2
tgl_Latn	66.7	49.6	68.1	48.3	47.7	72.0
tha_Thai	55.7	38.9	46.2	35.0	33.0	69.0
tir_Ethi	28.4	29.6	24.5	23.5	25.0	39.9
tsn_Latn	31.8	30.1	28.5	24.7	31.2	49.8
tso_Latn	33.4	30.0	30.4	28.0	29.7	41.7
tur_Latn	69.9	47.3	65.4	42.1	39.6	70.6
ukr_Cyrl	72.8	65.7	80.8	69.7	41.9	72.3
urd_Arab	48.3	37.0	43.2	34.7	31.7	65.6
urd_Latn	40.3	34.1	38.0	30.1	34.2	49.4
uzn_Latn	44.1	33.1	35.1	30.6	33.1	69.1
vie_Latn	72.9	59.6	78.4	43.5	41.4	73.7
war_Latn	48.9	39.3	44.4	37.4	38.6	47.6
wol_Latn	29.0	28.9	27.6	26.0	26.8	40.6
xho_Latn	30.0	29.9	28.2	27.6	30.2	54.4
yor_Latn	29.1	30.1	28.3	27.7	27.2	38.6
zho_Hans	77.6	62.4	83.7	64.6	66.0	73.7
zho_Hant	76.3	59.3	82.0	57.7	62.2	73.1
zsm_Latn	74.0	56.4	76.3	51.7	51.3	72.4
zul_Latn	30.4	30.2	29.7	27.1	30.7	54.2

Multiple-script language comparison

Per-language accuracy (%). The final AVG column is a cross-system summary, not an additional model.

Published source

Multiple-script language comparison. Published source metrics with original precision; unreported cells are not zero.
Language	GPT3.5-turbo unk (Belebele zero-shot)	Llama 2 base 70B (Belebele 5-shot)	Falcon 40B (Belebele 5-shot)	InfoXLM large (550M) (Belebele English fine-tuning)	Average across shown systems (%)
arb_Arab	69.3	61.7	38.3	71.0	60.1
arb_Latn	31.1	26.8	26.3	32.2	29.1
ben_Beng	43.6	39.1	22.6	63.4	42.2
ben_Latn	34.6	29.6	32.1	36.9	33.3
hin_Deva	49.1	52.6	27.1	60.2	47.3
hin_Latn	52.3	49.0	40.0	49.7	47.8
npi_Deva	40.4	40.4	25.4	60.7	41.7
npi_Latn	35.1	30.2	30.9	35.8	33.0
sin_Sinh	32.6	25.2	27.7	67.2	38.2
sin_Latn	30.4	33.8	32.6	34.2	32.8
urd_Arab	48.3	43.2	31.7	63.8	46.7
urd_Latn	40.3	38.0	34.2	42.6	38.8
zho_Hant	76.3	82.0	62.2	72.4	73.3
zho_Hans	77.6	83.7	66.0	74.6	75.5

All translated-test language results

The 91 non-English-language subset also shows an English reference row. Its averages are not the 122-language averages. In-language appendix average is 44.0; the summary prints 44.1.

Published source

All translated-test language results. Published source metrics with original precision; unreported cells are not zero.
Language or summary metric	Llama-2-chat 70B (Belebele 91-language in-language)	Llama-2-chat 70B (Belebele 91-language Translate-Test)	XLM-V large (1.2B) (Belebele Translate-Train-All)
AVG	44.0	57.1	64.9
PCT Above 50	35.2%	78.0%	90.1%
PCT Above 70	2.2%	2.2%	42.9%
eng_Latn	78.8	78.8	77.8
fra_Latn	72.2	70.6	73.1
por_Latn	70.2	69.9	70.9
deu_Latn	69.4	65.7	72.6
ita_Latn	68.6	66.1	70.6
spa_Latn	68.4	69.3	71.0
cat_Latn	68.2	67.0	71.6
swe_Latn	67.4	66.1	73.0
rus_Cyrl	67.0	67.3	71.9
dan_Latn	66.2	66.8	70.8
nld_Latn	66.2	67.2	68.4
nob_Latn	65.7	68.3	71.8
ukr_Cyrl	65.7	66.0	70.4
ron_Latn	65.6	67.0	72.3
srp_Cyrl	65.1	66.2	71.0
bul_Cyrl	65.0	67.7	72.6
ces_Latn	65.0	65.6	69.9
hrv_Latn	64.7	65.3	70.0
fin_Latn	62.7	61.1	69.1
slv_Latn	62.4	61.2	69.7
zho_Hans	62.4	71.2	69.8
pol_Latn	61.7	63.0	69.0
ind_Latn	61.3	64.8	68.9
hun_Latn	61.1	62.9	69.7
vie_Latn	59.6	59.4	69.6
zho_Hant	59.3	65.8	69.2
slk_Latn	58.8	66.2	69.3
afr_Latn	57.9	65.0	69.3
jpn_Jpan	56.6	54.8	66.4
zsm_Latn	56.4	67.0	69.1
kor_Hang	56.3	56.7	70.1
mkd_Cyrl	55.7	66.7	67.8
ell_Grek	50.7	67.6	70.3
tgl_Latn	49.6	62.2	69.2
tur_Latn	47.3	62.6	66.7
arb_Arab	42.3	60.7	67.2
hin_Deva	42.0	62.8	57.9
pes_Arab	41.8	59.6	68.3
heb_Hebr	41.4	62.0	67.2
lvs_Latn	41.0	60.9	70.1
ceb_Latn	40.6	62.6	45.4
lit_Latn	39.7	60.8	68.3
hin_Latn	39.2	52.7	53.1
tha_Thai	38.9	54.1	63.8
isl_Latn	38.0	58.1	67.3
jav_Latn	37.0	55.3	64.2
urd_Arab	37.0	59.4	61.6
est_Latn	36.6	59.4	63.2
als_Latn	36.0	63.1	68.4
asm_Beng	35.7	57.7	53.7
swh_Latn	35.1	57.8	64.9
ben_Beng	34.9	61.0	60.0
sun_Latn	34.9	50.8	60.9
mar_Deva	34.8	60.0	62.6
kat_Geor	34.6	57.7	64.7
tam_Taml	34.4	55.9	61.8
urd_Latn	34.1	43.0	42.2
hat_Latn	34.1	56.3	57.1
azj_Latn	33.4	55.6	59.7
sin_Sinh	33.4	57.7	64.4
pan_Guru	33.1	57.6	58.1
npi_Deva	32.9	62.0	58.4
ckb_Arab	32.8	51.3	29.7
kaz_Cyrl	32.4	53.2	60.1
hau_Latn	32.1	43.4	51.0
hye_Armn	31.9	58.0	59.4
mya_Mymr	31.3	46.6	56.6
khk_Cyrl	31.1	52.2	56.7
guj_Gujr	31.1	59.6	58.7
lin_Latn	31.0	40.3	44.7
lug_Latn	30.9	38.7	39.9
ssw_Latn	30.7	43.2	39.8
khm_Khmr	30.6	52.8	60.0
plt_Latn	30.5	46.7	55.7
ben_Latn	30.4	45.1	46.8
som_Latn	30.3	40.8	46.0
pbt_Arab	30.2	48.8	55.4
zul_Latn	30.2	44.4	46.9
nso_Latn	30.1	43.4	45.9
tsn_Latn	30.1	40.4	49.0
yor_Latn	30.1	37.7	35.0
ibo_Latn	30.1	35.3	40.1
mal_Mlym	30.1	63.0	62.0
xho_Latn	29.9	49.2	48.7
fuv_Latn	29.8	29.4	29.7
gaz_Latn	29.3	37.0	48.8
ory_Orya	29.2	57.8	60.8
amh_Ethi	28.9	50.4	53.1
wol_Latn	28.9	39.0	36.8
tel_Telu	27.5	54.3	55.6
lao_Laoo	26.5	47.4	55.8
kan_Knda	21.9	62.0	61.1

Translation and instruction-language comparisons

The first two rows cover 91 non-English languages; the instruction-language comparison covers 89. These subsets are distinct from the full 122-language result.

Published source

Translation and instruction-language comparisons. Published source metrics with original precision; unreported cells are not zero.
Model	Variant	Evaluation setting	Average accuracy (%)	Language share ≥50% accuracy (%)	Language share ≥70% accuracy (%)	English accuracy (%)
Translate-Test (English) on 91 non-English languages in Zero-Shot
Llama-2-chat	70B	Translate-Test	57.1	78.0%	2.2%	78.8
Llama-2-chat	70B	In-Language	44.1	35.2%	2.2%	78.8
Translated Instructions in 89 non-English languages Zero-Shot
Llama-2-chat	70B	In-Language Translated Instructions	38.7	36.0%	7.9%	78.8
Llama-2-chat	70B	English Instructions	44.9	37.1%	3.4%	78.8

Original benchmark results and configurations

Numeric results and configurations from the seven benchmark-owner papers and maintained result tables. This is not a census of all downstream papers or unpublished evaluations.

Snapshot

7 source tables20 configurationsDisplay only

Original benchmark report Decision Models Download full numeric results (JSON)

About Belebele

Year

2023

Tasks

Multilingual multiple-choice reading comprehension

Format

Decision classification

Difficulty

Depends on the evaluated split and label mapping

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants Public benchmark source

Freshness and provenance

Version

Belebele 2023

Refresh cadence

Static

Staleness state

Stale

Question availability

Public benchmark set

StaleDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

Which Belebele results are included?

This page includes 7 numeric result and configuration tables from Belebele’s original paper and available benchmark-owner updates, covering 20 evaluated systems or configurations. Source links, precision, metrics, and evaluation settings remain attached to each table. This coverage does not include every downstream paper or unpublished evaluation.

Can I compare these Belebele scores with the Perplexity panel?

Compare Belebele scores only when the sample, labels, input representation, and evaluation protocol match. Perplexity’s decision panel uses a fixed sample and its own harness. The original tables preserve different splits, metrics, and model configurations, so their numbers cannot establish a direct ranking against that panel.

Where can I download the Belebele results?

The results download on this page provides every imported Belebele table as JSON, with model links, published values, source URLs, and evaluation notes. Each linked configuration profile also exports its own result tables as JSON and numeric metrics as CSV. Missing source measurements remain unreported rather than zero.

Last updated: October 1, 2026 source review · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 2,000+ readers.

One email each week. Unsubscribe anytime.