# Vals MedQA (MedQA)

> Evaluating language model bias in medical questions.

Canonical page: https://benchlm.ai/benchmarks/valsmedqa

- Category: [Knowledge](/knowledge)
- Last updated: April 16, 2026

## About MedQA

- Year: 2026
- Tasks: Medical question answering
- Format: Accuracy score
- Difficulty: Medical knowledge and bias evaluation
- Paper: [MedQA](https://www.vals.ai/benchmarks/medqa)

BenchLM mirrors the public Vals AI MedQA leaderboard as display-only external evidence. The captured snapshot preserves overall scores, task-level scores where Vals publishes them, uncertainty, latency, and cost-per-test metadata. It is excluded from BenchLM weighted rankings.

MedQA is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (95 models)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [o1](/models/o1) | high reasoning | OpenAI | 96.52% |
| 2 | [GPT-5.1](/models/gpt-5-1) | high reasoning | OpenAI | 96.38% |
| 3 | [Gemini 3.1 Pro Preview](https://www.vals.ai/models/google_gemini-3.1-pro-preview) | high reasoning | Google | 96.37% |
| 4 | [GPT-5](https://www.vals.ai/models/openai_gpt-5-2025-08-07) | high reasoning | OpenAI | 96.32% |
| 5 | [GPT-5.4](/models/gpt-5-4) | xhigh reasoning | OpenAI | 96.09% |
| 6 | [o3](/models/o3) | high reasoning | OpenAI | 96.06% |
| 7 | [GPT-5 mini](/models/gpt-5-mini) | high reasoning | OpenAI | 96.06% |
| 8 | [Gemini 3 Pro Preview](https://www.vals.ai/models/google_gemini-3-pro-preview) | high reasoning | Google | 96.03% |
| 9 | [O4 Mini](https://www.vals.ai/models/openai_o4-mini-2025-04-16) | high reasoning | OpenAI | 96.02% |
| 10 | [Claude Opus 4.5 Thinking](/models/claude-opus-4-5-thinking) | — | Anthropic | 95.88% |
| 11 | [Gemini 3 Flash Preview](https://www.vals.ai/models/google_gemini-3-flash-preview) | high reasoning | Google | 95.81% |
| 12 | [Claude Opus 4.6 (Adaptive)](/models/claude-opus-4-6-thinking) | — | Anthropic | 95.41% |
| 13 | [Qwen3.5 Plus Thinking](https://www.vals.ai/models/alibaba_qwen3.5-plus-thinking) | — | Alibaba | 95.21% |
| 14 | [o3-mini](/models/o3-mini) | high reasoning | OpenAI | 94.83% |
| 15 | [Claude Sonnet 4.5 Thinking](/models/claude-sonnet-4-5-thinking) | — | Anthropic | 94.71% |
| 16 | [Grok 4.20 0309 Reasoning](https://www.vals.ai/models/grok_grok-4.20-0309-reasoning) | — | xAI | 94.55% |
| 17 | [Kimi K2.5 Thinking](https://www.vals.ai/models/kimi_kimi-k2.5-thinking) | — | Moonshot AI | 94.37% |
| 18 | [GLM 5 Thinking](https://www.vals.ai/models/zai_glm-5-thinking) | — | Zhipu AI | 94.27% |
| 19 | [GPT-5.2](/models/gpt-5-2) | xhigh reasoning | OpenAI | 94.13% |
| 20 | [DeepSeek V3p2 Thinking](https://www.vals.ai/models/fireworks_deepseek-v3p2-thinking) | high reasoning | Fireworks AI | 93.92% |
| 21 | [GLM-4.7](/models/glm-4-7) | — | Z.AI | 93.74% |
| 22 | [Claude Opus 4.1 20250805 Thinking](https://www.vals.ai/models/anthropic_claude-opus-4-1-20250805-thinking) | — | Anthropic | 93.59% |
| 23 | [GPT-5 nano](/models/gpt-5-nano) | high reasoning | OpenAI | 93.26% |
| 24 | [Claude Opus 4.5](/models/claude-opus-4-5) | — | Anthropic | 93.16% |
| 25 | [Gemini 2.5 Pro Exp 03 25](https://www.vals.ai/models/google_gemini-2.5-pro-exp-03-25) | — | Google | 93.14% |
| 26 | [o1-preview](/models/o1-preview) | high reasoning | OpenAI | 93.01% |
| 27 | [Claude Opus 4](https://www.vals.ai/models/anthropic_claude-opus-4-20250514) | — | Anthropic | 92.87% |
| 28 | [Claude Sonnet 4 20250514 Thinking](https://www.vals.ai/models/anthropic_claude-sonnet-4-20250514-thinking) | — | Anthropic | 92.71% |
| 29 | [Kimi K2 Thinking](https://www.vals.ai/models/kimi_kimi-k2-thinking) | — | Moonshot AI | 92.59% |
| 30 | [Claude Opus 4.1](https://www.vals.ai/models/anthropic_claude-opus-4-1-20250805) | — | Anthropic | 92.53% |
| 31 | [MiniMax M2.5](/models/minimax-m2-5) | — | MiniMax | 92.53% |
| 32 | [Grok 4 0709](https://www.vals.ai/models/grok_grok-4-0709) | — | xAI | 92.49% |
| 33 | [Grok 2 1212](https://www.vals.ai/models/grok_grok-2-1212) | — | xAI | 92.32% |
| 34 | [GLM-4.6](/models/glm-4-6) | — | Z.AI | 92.22% |
| 35 | [Grok 4.1 Fast (Reasoning)](/models/grok-4-1-fast-reasoning) | — | xAI | 92.08% |
| 36 | [Grok 4 Fast (Reasoning)](/models/grok-4-fast-reasoning) | — | xAI | 92.07% |
| 37 | [Claude Sonnet 4.6](/models/claude-sonnet-4-6) | — | Anthropic | 92.06% |
| 38 | [Gemini 2.5 Flash Preview 09 2025](https://www.vals.ai/models/google_gemini-2.5-flash-preview-09-2025) | — | Google | 91.43% |
| 39 | [GPT-OSS 120B](/models/gpt-oss-120b) | — | OpenAI | 91.36% |
| 40 | [GPT-4.1](/models/gpt-4-1) | high reasoning | OpenAI | 91.18% |
| 41 | [Gemini 2.5 Flash Preview 09 2025 Thinking](https://www.vals.ai/models/google_gemini-2.5-flash-preview-09-2025-thinking) | — | Google | 91.17% |
| 42 | [MiniMax M2.1](https://www.vals.ai/models/minimax_MiniMax-M2.1) | — | MiniMax | 91.16% |
| 43 | [Gemini 2.5 Flash Preview 04 17 Thinking](https://www.vals.ai/models/google_gemini-2.5-flash-preview-04-17-thinking) | — | Google | 91.02% |
| 44 | [DeepSeek-R1](/models/deepseek-r1) | — | DeepSeek | 90.80% |
| 45 | [Qwen3 235b A22b](https://www.vals.ai/models/fireworks_qwen3-235b-a22b) | — | Fireworks AI | 90.62% |
| 46 | [Claude Sonnet 4](https://www.vals.ai/models/anthropic_claude-sonnet-4-20250514) | — | Anthropic | 90.35% |
| 47 | [O1 Mini](https://www.vals.ai/models/openai_o1-mini-2024-09-12) | high reasoning | OpenAI | 90.22% |
| 48 | [Claude 3.7 Sonnet 20250219 Thinking](https://www.vals.ai/models/anthropic_claude-3-7-sonnet-20250219-thinking) | — | Anthropic | 90.22% |
| 49 | [Grok 3 Mini Fast High Reasoning](https://www.vals.ai/models/grok_grok-3-mini-fast-high-reasoning) | high reasoning | xAI | 90.10% |
| 50 | [GLM-4.5](/models/glm-4-5) | — | Z.AI | 89.97% |
| 51 | [Magistral Medium 2509](https://www.vals.ai/models/mistralai_magistral-medium-2509) | — | Mistral AI | 89.47% |
| 52 | [DeepSeek V3p2](https://www.vals.ai/models/fireworks_deepseek-v3p2) | none reasoning | Fireworks AI | 89.45% |
| 53 | [Gemini 2.5 Flash Lite Preview 09 2025 Thinking](https://www.vals.ai/models/google_gemini-2.5-flash-lite-preview-09-2025-thinking) | — | Google | 88.87% |
| 54 | [Grok 3 Mini Fast Low Reasoning](https://www.vals.ai/models/grok_grok-3-mini-fast-low-reasoning) | low reasoning | xAI | 88.65% |
| 55 | [Meta Llama Meta Llama 3.1 405B Instruct Turbo](https://www.vals.ai/models/together_meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo) | — | Together AI | 88.24% |
| 56 | [GPT-4o](/models/gpt-4o) | high reasoning | OpenAI | 88.16% |
| 57 | [Qwen3 Max Preview](https://www.vals.ai/models/alibaba_qwen3-max-preview) | — | Alibaba | 87.38% |
| 58 | [Qwen3 Max](/models/qwen3-max) | — | Alibaba | 87.37% |
| 59 | [Gemini 2.5 Flash Preview 04 17](https://www.vals.ai/models/google_gemini-2.5-flash-preview-04-17) | — | Google | 86.73% |
| 60 | [Meta Llama Meta Llama 3.1 70B Instruct Turbo](https://www.vals.ai/models/together_meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo) | — | Together AI | 84.78% |
| 61 | [GPT-4.1 mini](/models/gpt-4-1-mini) | high reasoning | OpenAI | 84.63% |
| 62 | [Moonshotai Kimi K2 Instruct](https://www.vals.ai/models/together_moonshotai/Kimi-K2-Instruct) | — | Together AI | 83.97% |
| 63 | [Grok 3](https://www.vals.ai/models/grok_grok-3) | — | xAI | 83.85% |
| 64 | [Claude 3.5 Sonnet](/models/claude-3-5-sonnet) | — | Anthropic | 83.19% |
| 65 | [GPT-OSS 20B](/models/gpt-oss-20b) | — | OpenAI | 82.88% |
| 66 | [Magistral Small 2509](https://www.vals.ai/models/mistralai_magistral-small-2509) | — | Mistral AI | 82.36% |
| 67 | [Mistral Large 2512](https://www.vals.ai/models/mistralai_mistral-large-2512) | — | Mistral AI | 82.23% |
| 68 | [DeepSeek V3 0324](https://www.vals.ai/models/fireworks_deepseek-v3-0324) | — | Fireworks AI | 82.00% |
| 69 | [GPT-4 Turbo](/models/gpt-4-turbo) | high reasoning | OpenAI | 81.99% |
| 70 | [Gemini 2.0 Flash 001](https://www.vals.ai/models/google_gemini-2.0-flash-001) | — | Google | 81.47% |
| 71 | [DeepSeek V3](/models/deepseek-v3) | — | DeepSeek | 80.90% |
| 72 | [Command A 03 2025](https://www.vals.ai/models/cohere_command-a-03-2025) | — | Cohere | 80.55% |
| 73 | [Gemini 2.5 Flash Lite Preview 09 2025](https://www.vals.ai/models/google_gemini-2.5-flash-lite-preview-09-2025) | — | Google | 80.33% |
| 74 | [Claude Haiku 4.5 Thinking](/models/claude-haiku-4-5-thinking) | — | Anthropic | 79.57% |
| 75 | [Mistral Medium 2505](https://www.vals.ai/models/mistralai_mistral-medium-2505) | — | Mistral AI | 78.23% |
| 76 | [Qwen Qwen2.5 72B Instruct Turbo](https://www.vals.ai/models/together_Qwen/Qwen2.5-72B-Instruct-Turbo) | — | Together AI | 77.39% |
| 77 | [Gemini 1.5 Pro 002](https://www.vals.ai/models/google_gemini-1.5-pro-002) | — | Google | 76.53% |
| 78 | [Mistral Large 2411](https://www.vals.ai/models/mistralai_mistral-large-2411) | — | Mistral AI | 76.22% |
| 79 | [Grok 4.1 Fast Non Reasoning](https://www.vals.ai/models/grok_grok-4-1-fast-non-reasoning) | — | xAI | 76.03% |
| 80 | [Grok 4 Fast Non Reasoning](https://www.vals.ai/models/grok_grok-4-fast-non-reasoning) | — | xAI | 75.36% |
| 81 | [GPT-4o mini](/models/gpt-4o-mini) | high reasoning | OpenAI | 72.44% |
| 82 | [Mistral Small 2503](https://www.vals.ai/models/mistralai_mistral-small-2503) | — | Mistral AI | 69.10% |
| 83 | [GPT-4.1 nano](/models/gpt-4-1-nano) | high reasoning | OpenAI | 68.22% |
| 84 | [Jamba 1.5 Large](https://www.vals.ai/models/ai21labs_jamba-1.5-large) | — | AI21 Labs | 68.11% |
| 85 | [Meta Llama Meta Llama 3.1 8B Instruct Turbo](https://www.vals.ai/models/together_meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo) | — | Together AI | 62.61% |
| 86 | [Mixtral 8x22B Instruct v0.1](/models/mixtral-8x22b-instruct-v0-1) | — | Mistral | 62.14% |
| 87 | [GPT-3.5 Turbo](https://www.vals.ai/models/openai_gpt-3.5-turbo) | high reasoning | OpenAI | 58.47% |
| 88 | [Mistral Small 2402](https://www.vals.ai/models/mistralai_mistral-small-2402) | — | Mistral AI | 56.98% |
| 89 | [Jamba 1.5 Mini](https://www.vals.ai/models/ai21labs_jamba-1.5-mini) | — | AI21 Labs | 55.18% |
| 90 | [Mistralai Mixtral 8x7B V0.1](https://www.vals.ai/models/together_mistralai/Mixtral-8x7B-v0.1) | — | Together AI | 53.22% |
| 91 | [Jamba Mini 1.6](https://www.vals.ai/models/ai21labs_jamba-mini-1.6) | — | AI21 Labs | 52.52% |
| 92 | [Meta Llama Llama 4 Scout 17B 16E Instruct](https://www.vals.ai/models/together_meta-llama/Llama-4-Scout-17B-16E-Instruct) | — | Together AI | 50.90% |
| 93 | [Jamba Large 1.6](https://www.vals.ai/models/ai21labs_jamba-large-1.6) | — | AI21 Labs | 50.70% |
| 94 | [Llama4 Maverick Instruct Basic](https://www.vals.ai/models/fireworks_llama4-maverick-instruct-basic) | — | Fireworks AI | 43.30% |
| 95 | [Command R Plus](https://www.vals.ai/models/cohere_command-r-plus) | — | Cohere | 2.65% |

## FAQ

### What does MedQA measure?

Evaluating language model bias in medical questions.

### Which model leads the published MedQA snapshot?

o1 currently leads the published MedQA snapshot with a score of 96.52%.

### How many models are evaluated on MedQA?

The April 16, 2026 contains 95 AI models.

### Does MedQA affect BenchLM's overall score?

Not directly. MedQA is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.