Vals MedQA (MedQA)

We show this table for reference; we do not rank on it.

Data verified April 16, 2026

Evaluating language model bias in medical questions.

Evaluation results published by Vals AI. Published leaderboard aggregates and evaluation settings only. Dataset contents are not included.

MedQA score on MedQA — April 16, 2026

We mirror the published medqa score view for MedQA. o1 leads the public snapshot at 96.52%, followed by GPT-5.1 (96.38%) and Gemini 3.1 Pro Preview (96.37%). We do not use these results to rank models overall.

1Closed

OpenAI

high reasoning

96.52%

Overall 35.8Context 200K

2Closed

GPT-5.1

OpenAI

high reasoning

96.38%

Overall 56.39Context 200K

Gemini 3.1 Pro Preview

Google

high reasoning

96.37%

95 modelsKnowledgeCurrentDisplay onlyUpdated April 16, 2026

MedQA score table (95 models)

MedQA score results for MedQA
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	o1OpenAIhigh reasoning	96.52%	Not reported	Closed
2	GPT-5.1OpenAIhigh reasoning	96.38%	Not reported	Closed
3	Gemini 3.1 Pro PreviewGooglehigh reasoning	96.37%	Not reported	Unknown
4	GPT-5OpenAIhigh reasoning	96.32%	Not reported	Unknown
5	GPT-5.4OpenAIxhigh reasoning	96.09%	Not reported	Closed
6	o3OpenAIhigh reasoning	96.06%	Not reported	Closed
7	GPT-5 miniOpenAIhigh reasoning	96.06%	Not reported	Closed
8	Gemini 3 Pro PreviewGooglehigh reasoning	96.03%	Not reported	Unknown
9	o4-miniOpenAIhigh reasoning	96.02%	Not reported	Closed
10	Claude Opus 4.5 ThinkingAnthropic	95.88%	Not reported	Closed
11	Gemini 3 Flash PreviewGooglehigh reasoning	95.81%	Not reported	Unknown
12	Claude Opus 4.6 (Adaptive)Anthropic	95.41%	Not reported	Closed
13	Qwen3.5 Plus ThinkingAlibaba	95.21%	Not reported	Unknown
14	o3-miniOpenAIhigh reasoning	94.83%	Not reported	Closed
15	Claude Sonnet 4.5 ThinkingAnthropic	94.71%	Not reported	Closed
16	Grok 4.20 0309 ReasoningxAI	94.55%	Not reported	Unknown
17	Kimi K2.5 ThinkingMoonshot AI	94.37%	Not reported	Unknown
18	GLM 5 ThinkingZhipu AI	94.27%	Not reported	Unknown
19	GPT-5.2OpenAIxhigh reasoning	94.13%	Not reported	Closed
20	DeepSeek V3p2 ThinkingFireworks AIhigh reasoning	93.92%	Not reported	Unknown
21	GLM-4.7Z.AI	93.74%	Not reported	Open
22	Claude Opus 4.1 20250805 ThinkingAnthropic	93.59%	Not reported	Unknown
23	GPT-5 nanoOpenAIhigh reasoning	93.26%	Not reported	Closed
24	Claude Opus 4.5Anthropic	93.16%	Not reported	Closed
25	Gemini 2.5 Pro Exp 03 25Google	93.14%	Not reported	Unknown
26	o1-previewOpenAIhigh reasoning	93.01%	Not reported	Closed
27	Claude Opus 4Anthropic	92.87%	Not reported	Unknown
28	Claude Sonnet 4 20250514 ThinkingAnthropic	92.71%	Not reported	Unknown
29	Kimi K2 ThinkingMoonshot AI	92.59%	Not reported	Unknown
30	Claude Opus 4.1Anthropic	92.53%	Not reported	Unknown
31	MiniMax M2.5MiniMax	92.53%	Not reported	Closed
32	Grok 4 0709xAI	92.49%	Not reported	Unknown
33	Grok 2 1212xAI	92.32%	Not reported	Unknown
34	GLM-4.6Z.AI	92.22%	Not reported	Open
35	Grok 4.1 Fast (Reasoning)xAI	92.08%	Not reported	Closed
36	Grok 4 Fast (Reasoning)xAI	92.07%	Not reported	Closed
37	Claude Sonnet 4.6Anthropic	92.06%	Not reported	Closed
38	Gemini 2.5 Flash Preview 09 2025Google	91.43%	Not reported	Unknown
39	GPT-OSS 120BOpenAI	91.36%	Not reported	Open
40	GPT-4.1OpenAIhigh reasoning	91.18%	Not reported	Closed
41	Gemini 2.5 Flash Preview 09 2025 ThinkingGoogle	91.17%	Not reported	Unknown
42	MiniMax M2.1MiniMax	91.16%	Not reported	Unknown
43	Gemini 2.5 Flash Preview 04 17 ThinkingGoogle	91.02%	Not reported	Unknown
44	DeepSeek-R1DeepSeek	90.80%	Not reported	Open
45	Qwen3 235B-A22BAlibaba	90.62%	Not reported	Open
46	Claude Sonnet 4Anthropic	90.35%	Not reported	Unknown
47	o1-miniOpenAIhigh reasoning	90.22%	Not reported	Closed
48	Claude 3.7 Sonnet ThinkingAnthropic	90.22%	Not reported	Closed
49	Grok 3 Mini Fast High ReasoningxAIhigh reasoning	90.10%	Not reported	Unknown
50	GLM-4.5Z.AI	89.97%	Not reported	Closed
51	Magistral Medium 2509Mistral AI	89.47%	Not reported	Unknown
52	DeepSeek V3p2Fireworks AInone reasoning	89.45%	Not reported	Unknown
53	Gemini 2.5 Flash Lite Preview 09 2025 ThinkingGoogle	88.87%	Not reported	Unknown
54	Grok 3 Mini Fast Low ReasoningxAIlow reasoning	88.65%	Not reported	Unknown
55	Meta Llama Meta Llama 3.1 405B Instruct TurboTogether AI	88.24%	Not reported	Unknown
56	GPT-4oOpenAIhigh reasoning	88.16%	Not reported	Closed
57	Qwen3 Max PreviewAlibaba	87.38%	Not reported	Unknown
58	Qwen3 MaxAlibaba	87.37%	Not reported	Closed
59	Gemini 2.5 Flash Preview 04 17Google	86.73%	Not reported	Unknown
60	Meta Llama Meta Llama 3.1 70B Instruct TurboTogether AI	84.78%	Not reported	Unknown
61	GPT-4.1 miniOpenAIhigh reasoning	84.63%	Not reported	Closed
62	Moonshotai Kimi K2 InstructTogether AI	83.97%	Not reported	Unknown
63	Grok 3xAI	83.85%	Not reported	Unknown
64	Claude 3.5 SonnetAnthropic	83.19%	Not reported	Closed
65	GPT-OSS 20BOpenAI	82.88%	Not reported	Open
66	Magistral Small 2509Mistral AI	82.36%	Not reported	Unknown
67	Mistral Large 2512Mistral AI	82.23%	Not reported	Unknown
68	DeepSeek V3 0324DeepSeek	82.00%	Not reported	Open
69	GPT-4 TurboOpenAIhigh reasoning	81.99%	Not reported	Closed
70	Gemini 2.0 Flash 001Google	81.47%	Not reported	Unknown
71	DeepSeek V3DeepSeek	80.90%	Not reported	Open
72	Command A 03 2025Cohere	80.55%	Not reported	Unknown
73	Gemini 2.5 Flash Lite Preview 09 2025Google	80.33%	Not reported	Unknown
74	Claude Haiku 4.5 ThinkingAnthropic	79.57%	Not reported	Closed
75	Mistral Medium 2505Mistral AI	78.23%	Not reported	Unknown
76	Qwen Qwen2.5 72B Instruct TurboTogether AI	77.39%	Not reported	Unknown
77	Gemini 1.5 Pro 002Google	76.53%	Not reported	Unknown
78	Mistral Large 2411Mistral AI	76.22%	Not reported	Unknown
79	Grok 4.1 Fast Non ReasoningxAI	76.03%	Not reported	Unknown
80	Grok 4 Fast Non ReasoningxAI	75.36%	Not reported	Unknown
81	GPT-4o miniOpenAIhigh reasoning	72.44%	Not reported	Closed
82	Mistral Small 2503Mistral AI	69.10%	Not reported	Unknown
83	GPT-4.1 nanoOpenAIhigh reasoning	68.22%	Not reported	Closed
84	Jamba 1.5 LargeAI21 Labs	68.11%	Not reported	Unknown
85	Meta Llama Meta Llama 3.1 8B Instruct TurboTogether AI	62.61%	Not reported	Unknown
86	Mixtral 8x22B Instruct v0.1Mistral	62.14%	Not reported	Open
87	GPT-3.5 TurboOpenAIhigh reasoning	58.47%	Not reported	Unknown
88	Mistral Small 2402Mistral AI	56.98%	Not reported	Unknown
89	Jamba 1.5 MiniAI21 Labs	55.18%	Not reported	Unknown
90	Mistralai Mixtral 8x7B V0.1Together AI	53.22%	Not reported	Unknown
91	Jamba Mini 1.6AI21 Labs	52.52%	Not reported	Unknown
92	Meta Llama Llama 4 Scout 17B 16E InstructTogether AI	50.90%	Not reported	Unknown
93	Jamba Large 1.6AI21 Labs	50.70%	Not reported	Unknown
94	Llama4 Maverick Instruct BasicFireworks AI	43.30%	Not reported	Unknown
95	Command R PlusCohere	2.65%	Not reported	Unknown

How MedQA is shown here

BenchLM mirrors the public Vals AI MedQA leaderboard captured from https://www.vals.ai/benchmarks/medqa and updated by Vals on April 16, 2026. The snapshot preserves overall scores, uncertainty, latency, cost-per-test metadata, and task-level scores where Vals publishes them.

MedQA is display only on BenchLM. Vals proprietary or Vals-hosted aggregate views are useful context, but BenchLM does not use them as weighted ranking inputs or as a replacement for benchmark-native source records.

Snapshot

95 Vals rows7 task viewspublic datasetTasks: Overall, Unbiased, Hispanic, Black, AsianDisplay only

MedQA on Vals AI Vals methodology Vals home

The published MedQA snapshot places o1 first at 96.52%. The third row is 0.15 points behind. The broader top-10 range is 0.64 points, so many of the published results sit in a relatively narrow band.

95 models have been evaluated on MedQA. The benchmark falls in the Knowledge category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. MedQA is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About MedQA

Year

2026

Tasks

Medical question answering

Format

Accuracy score

Difficulty

Medical knowledge and bias evaluation

BenchLM mirrors the public Vals AI MedQA leaderboard as display-only external evidence. The captured snapshot preserves overall scores, task-level scores where Vals publishes them, uncertainty, latency, and cost-per-test metadata. It is excluded from BenchLM weighted rankings.

MedQA Public benchmark source

Freshness and provenance

Version

MedQA 2026

Refresh cadence

Quarterly

Staleness state

Current

Question availability

Public benchmark set

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does MedQA measure?

Evaluating language model bias in medical questions.

Which model leads the published MedQA snapshot?

o1 currently leads the published MedQA snapshot with 96.52% medqa score. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on MedQA?

The April 16, 2026 snapshot contains 95 AI models.

Last updated: April 16, 2026 · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 7,000+ subscribers.

One email each week. Unsubscribe anytime.