JudgeBench
We show this table for reference; we do not rank on it.
JudgeBench evaluates whether a judge can identify the better response in pairs with objective correctness labels. It tests judgment on challenging questions rather than agreement with writing style preferences.
Original benchmark results
JudgeBench’s paper and maintained leaderboard report judges under different prompts, reasoning efforts, and response-model splits. Accuracy is a percentage. The paper’s 700-pair GPT-4o split is distinct from the 270-pair Claude split. The app appends the same Nemotron CSV to both tabs without specifying a response split; we show that supplement once with its split unresolved. Live one-decimal values and paper two-decimal values are preserved separately.
11 source tables · 54 evaluated configurations · reviewed October 1, 2026. Download full results (JSON)
Maintainer results on GPT-4o response pairs
GPT-4o-2024-05-13 response split. The public app rounds to one decimal; paper tables retain two decimals.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Type | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|---|
| Arena-Hard (o3-mini-2025-01-31 (high)) | Prompted Judge | 67.5 | 89.8 | 87.5 | 100.0 | 80.9 |
| Arena-Hard (o3-mini-2025-01-31 (medium)) | Prompted Judge | 62.3 | 86.7 | 85.7 | 92.9 | 76.6 |
| Arena-Hard (o1-preview-2024-09-12) | Prompted Judge | 66.2 | 79.6 | 85.7 | 85.7 | 75.4 |
| Arena-Hard (DeepSeek-R1-250120) | Prompted Judge | 59.1 | 82.7 | 80.4 | 92.9 | 73.1 |
| Arena-Hard (o3-mini-2025-01-31 (low)) | Prompted Judge | 63.0 | 69.4 | 83.9 | 83.3 | 70.6 |
| Arena-Hard (o1-mini-2024-09-12) | Prompted Judge | 58.4 | 62.2 | 82.1 | 78.6 | 65.7 |
| Arena-Hard (claude-3-5-sonnet-20240620) | Prompted Judge | 62.3 | 66.3 | 66.1 | 64.3 | 64.3 |
| Skywork-Reward-Gemma-2-27B | Reward Model | 59.7 | 66.3 | 83.9 | 50.0 | 64.3 |
| InternLM2-20B-Reward | Reward Model | 62.3 | 69.4 | 66.1 | 50.0 | 63.4 |
| CompassJudger-1-32B | Fine-Tuned Judge | 58.4 | 62.2 | 75.0 | 59.5 | 62.3 |
| Skywork-Reward-Llama-3.1-8B | Reward Model | 59.1 | 64.3 | 76.8 | 50.0 | 62.3 |
| GRM-Gemma-2B | Reward Model | 63.0 | 53.1 | 64.3 | 54.8 | 59.4 |
| InternLM2-7B-Reward | Reward Model | 56.5 | 61.2 | 71.4 | 50.0 | 59.4 |
| Skywork-Critic-Llama-3.1-70B | Fine-Tuned Judge | 55.8 | 55.1 | 73.2 | 47.6 | 57.4 |
| Arena-Hard (Llama-3.1-405B-Instruct) | Prompted Judge | 55.8 | 54.1 | 69.6 | 50.0 | 56.9 |
| Arena-Hard (gpt-4o-2024-05-13) | Prompted Judge | 50.6 | 54.1 | 75.0 | 59.5 | 56.6 |
| CompassJudger-1-14B | Fine-Tuned Judge | 51.9 | 60.2 | 67.9 | 42.9 | 55.7 |
| Skywork-Critic-Llama-3.1-8B | Fine-Tuned Judge | 51.3 | 54.1 | 73.2 | 33.3 | 53.4 |
| Arena-Hard (Llama-3.1-70B-Instruct) | Prompted Judge | 51.3 | 49.0 | 60.7 | 52.4 | 52.3 |
| Vanilla (gpt-4o-2024-05-13) | Prompted Judge | 44.2 | 48.0 | 66.1 | 61.9 | 50.9 |
| Arena-Hard (gpt-4o-mini-2024-07-18) | Prompted Judge | 48.1 | 43.9 | 69.6 | 45.2 | 50.0 |
| Arena-Hard (gemini-1.5-pro-001) | Prompted Judge | 49.4 | 42.9 | 64.3 | 26.2 | 47.1 |
| CompassJudger-1-7B | Fine-Tuned Judge | 42.2 | 37.8 | 69.6 | 47.6 | 46.0 |
| CompassJudger-1-1.5B | Fine-Tuned Judge | 48.1 | 39.8 | 50.0 | 35.7 | 44.6 |
| VertexAI Evaluation (gemini-1.5-pro-001) | Prompted Judge | 45.5 | 44.9 | 53.6 | 28.6 | 44.6 |
| Arena-Hard (Llama-3.1-8B-Instruct) | Prompted Judge | 38.3 | 45.9 | 44.6 | 33.3 | 40.9 |
| Prometheus2-8x7b | Fine-Tuned Judge | 41.6 | 39.8 | 50.0 | 23.8 | 40.3 |
| Arena-Hard (gemini-1.5-flash-001) | Prompted Judge | 42.9 | 36.7 | 50.0 | 21.4 | 39.7 |
| Prometheus2-bgb-8x7b | Fine-Tuned Judge | 45.5 | 30.6 | 46.4 | 28.6 | 39.4 |
| Auto-J | Fine-Tuned Judge | 40.3 | 29.6 | 44.6 | 28.6 | 36.6 |
| JudgeLM-33B-v1.0 | Fine-Tuned Judge | 32.5 | 49.0 | 33.9 | 19.0 | 35.7 |
| Prometheus2-7b | Fine-Tuned Judge | 38.3 | 25.5 | 35.7 | 42.9 | 34.9 |
| ChatEval (gpt-4o-2024-05-13) | Multi-Agent Judge | 32.5 | 31.6 | 44.6 | 31.0 | 34.0 |
| Arena-Hard (claude-3-haiku-20240307) | Prompted Judge | 35.1 | 34.7 | 33.9 | 21.4 | 33.1 |
| JudgeLM-13B-v1.0 | Fine-Tuned Judge | 26.6 | 29.6 | 28.6 | 19.0 | 26.9 |
| JudgeLM-7B-v1.0 | Fine-Tuned Judge | 23.4 | 29.6 | 32.1 | 11.9 | 25.1 |
| PandaLM-7B-v1 | Fine-Tuned Judge | 9.1 | 21.4 | 7.1 | 16.7 | 13.1 |
Maintainer results on Claude response pairs
Claude-3-5-sonnet-20240620 response split. Nemotron supplement rows are omitted here because their response split is unspecified.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Type | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|---|
| Arena-Hard (gpt-4o-2024-05-13) | Prompted Judge | 52.6 | 51.0 | 61.8 | 38.7 | 51.9 |
| Arena-Hard (Llama-3.1-70B-Instruct) | Prompted Judge | 50.6 | 43.1 | 41.2 | 25.8 | 45.2 |
| Arena-Hard (claude-3-5-sonnet-20240620) | Prompted Judge | 42.2 | 52.9 | 55.9 | 32.3 | 44.8 |
| Arena-Hard (Llama-3.1-8B-Instruct) | Prompted Judge | 33.1 | 43.1 | 50.0 | 29.0 | 36.7 |
| Arena-Hard (claude-3-haiku-20240307) | Prompted Judge | 37.7 | 29.4 | 32.4 | 9.7 | 32.2 |
Maintainer Nemotron supplement — response split unspecified
The maintainer app inserts this CSV into both response-model views without a split field. These are 11 source-published results, not 22 independently measured rows.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|
| Llama-3.3-Nemotron-Super-49B-GenRM | 71.4 | 73.5 | 87.5 | 76.2 | 75.1 |
| Llama-3.3-Nemotron-Super-49B-GenRM + voting@32 | 70.8 | 83.7 | 87.5 | 83.3 | 78.6 |
| Llama-3.3-Nemotron-Super-49B-GenRM-Multilingual | 64.9 | 74.5 | 87.5 | 73.8 | 72.3 |
| Llama-3.3-Nemotron-Super-49B-GenRM-Multilingual + voting@32 | 65.6 | 82.7 | 87.5 | 85.7 | 76.3 |
| Llama-3.3-Nemotron-70B-Reward | 70.8 | 76.5 | 82.1 | 66.7 | 73.7 |
| Llama-3.3-Nemotron-70B-Reward-Multilingual | 66.2 | 71.4 | 82.1 | 59.5 | 69.4 |
| Llama-3.1-Nemotron-70B-Reward | 62.3 | 72.5 | 76.8 | 57.1 | 66.9 |
| Qwen-3-Nemotron-32B-Reward | 70.1 | 67.4 | 78.6 | 83.3 | 72.3 |
| Qwen-2.5-Nemotron-32B-Reward | 61.7 | 74.5 | 76.2 | 82.1 | 70.3 |
| Qwen3-Nemotron-32B-GenRM-Principle | 74.6 | 85.7 | 85.7 | 90.5 | 81.4 |
| Llama-3.3-Nemotron-70B-Reward-Principle | 74.0 | 74.5 | 82.1 | 81.0 | 76.3 |
Paper: judge methods
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|
| Prompted Judges | |||||
| Vanilla (GPT-4o) | 44.16 | 47.96 | 66.07 | 61.90 | 50.86 |
| Arena-Hard Judge (GPT-4o) | 50.65 | 54.08 | 75.00 | 59.52 | 56.57 |
| VertexAI Evaluation (Gemini-1.5-pro) | 45.45 | 44.90 | 53.57 | 28.57 | 44.57 |
| Fine-tuned Judges | |||||
| PandaLM | 9.09 | 21.43 | 7.14 | 16.67 | 13.14 |
| Prometheus2-7b | 38.31 | 25.51 | 35.71 | 42.86 | 34.86 |
| Prometheus2-8x7b | 41.56 | 39.80 | 50.00 | 23.81 | 40.29 |
| Prometheus2-bgb-8x7b | 45.45 | 30.61 | 46.43 | 28.57 | 39.43 |
| JudgeLM-7B | 23.38 | 29.59 | 32.14 | 11.90 | 25.14 |
| JudgeLM-13B | 26.62 | 29.59 | 28.57 | 19.05 | 26.86 |
| JudgeLM-33B | 32.47 | 48.98 | 33.93 | 19.05 | 35.71 |
| AutoJ | 40.26 | 29.59 | 44.64 | 28.57 | 36.57 |
| Skywork-LLaMA-3.1B-8B | 51.30 | 54.08 | 73.21 | 33.33 | 53.43 |
| Skywork-LLaMA-3.1B-70B | 55.84 | 55.10 | 73.21 | 47.62 | 57.43 |
| Multi-Agent Judges | |||||
| ChatEval | 32.47 | 31.63 | 44.64 | 30.95 | 34.00 |
Paper: Arena-Hard prompted judges
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge model | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|
| GPT-4o | 50.65 | 54.08 | 75.00 | 59.52 | 56.57 |
| GPT-4o-mini | 48.05 | 43.88 | 69.64 | 45.24 | 50.00 |
| o1-preview | 66.23 | 79.59 | 85.71 | 85.71 | 75.43 |
| o1-mini | 58.44 | 62.24 | 82.14 | 78.57 | 65.71 |
| o3-mini (high) | 67.53 | 89.80 | 87.50 | 100.0 | 80.86 |
| o3-mini (medium) | 62.34 | 86.73 | 85.71 | 92.86 | 76.57 |
| o3-mini (low) | 62.99 | 69.39 | 83.93 | 83.33 | 70.57 |
| Claude-3.5-Sonnet | 62.34 | 66.33 | 66.07 | 64.29 | 64.29 |
| Claude-3-Haiku | 35.06 | 34.69 | 33.93 | 21.43 | 33.14 |
| Llama-3.1-405B-Instruct | 55.84 | 54.08 | 69.64 | 50.00 | 56.86 |
| Llama-3.1-70B-Instruct | 51.30 | 48.98 | 60.71 | 52.38 | 52.29 |
| Llama-3.1-8B-Instruct | 38.31 | 45.92 | 44.64 | 33.33 | 40.86 |
| Gemini-1.5-pro | 49.35 | 42.86 | 64.29 | 26.19 | 47.14 |
| Gemini-1.5-flash | 42.86 | 36.73 | 50.00 | 21.43 | 39.71 |
| Deepseek-R1 | 59.09 | 82.65 | 80.36 | 92.86 | 73.14 |
Paper: reward models
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Reward model | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|
| Skywork-Reward-Gemma-2-27B | 59.74 | 66.33 | 83.93 | 50.00 | 64.29 |
| Skywork-Reward-Llama-3.1-8B | 59.09 | 64.29 | 76.79 | 50.00 | 62.29 |
| InternLM2-20B-Reward | 62.34 | 69.39 | 66.07 | 50.00 | 63.43 |
| InternLM2-7B-Reward | 56.49 | 61.22 | 71.43 | 50.00 | 59.43 |
| GRM-Gemma-2B | 62.99 | 53.06 | 64.29 | 54.76 | 59.43 |
Paper: solving versus judging
Paper v2; configurations and precision remain as published. Solver results are direct answers rather than judge preferences.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Setup | Knowledge (%) | Reasoning (%) | Math (%) | Coding (%) | Overall (%) |
|---|---|---|---|---|---|
| GPT-4o Solver | 48.70 | 53.06 | 58.93 | 73.81 | 54.57 |
| GPT-4o Judge | 50.65 | 54.08 | 75.00 | 59.52 | 56.57 |
| Claude-3.5-Sonnet Solver | 61.04 | 62.24 | 60.71 | 88.10 | 64.57 |
| Claude-3.5-Sonnet Judge | 62.34 | 66.33 | 66.07 | 64.29 | 64.29 |
| Llama-3.1-405B-Instruct Solver | 48.05 | 67.86 | 63.27 | 66.67 | 57.71 |
| Llama-3.1-405B-Instruct Judge | 55.84 | 54.08 | 69.64 | 50.00 | 56.86 |
| Gemini-1.5-pro Solver | 33.12 | 42.86 | 37.50 | 64.29 | 40.29 |
| Gemini-1.5-pro Judge | 49.35 | 42.86 | 64.29 | 26.19 | 47.14 |
Paper: judgment outcomes
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | A>B count | A<B count | Tie count | Invalid count |
|---|---|---|---|---|
| PandaLM-7B | 45 | 114 | 479 | 62 |
| Prometheus2-7b | 395 | 232 | 0 | 73 |
| Prometheus2-8x7b | 331 | 328 | 0 | 41 |
| Prometheus2-bgb-8x7b | 239 | 215 | 0 | 246 |
| JudgeLM-7B | 399 | 229 | 72 | 0 |
| JudgeLM-13B | 355 | 312 | 33 | 0 |
| JudgeLM-33B | 344 | 264 | 92 | 0 |
| AutoJ | 289 | 378 | 33 | 0 |
| Skywork-LLaMA-3.1B-8B | 346 | 354 | 0 | 0 |
| Skywork-LLaMA-3.1B-70B | 390 | 310 | 0 | 0 |
Paper: order inconsistency
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Inconsistent (%) |
|---|---|
| PandaLM-7B | 29.14% |
| Prometheus2-7b | 52.29% |
| Prometheus2-8x7b | 40.00% |
| Prometheus2-bgb-8x7b | 43.71% |
| JudgeLM-7B | 59.71% |
| JudgeLM-13B | 54.57% |
| JudgeLM-33B | 38.00% |
| AutoJ | 43.71% |
| Skywork-Llama-3.1B-8B | 18.86% |
| Skywork-Llama-3.1B-70B | 18.29% |
Paper: Prometheus and its base model
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge | Knowledge accuracy (%) |
|---|---|
| Prometheus2-7b | 38.31 |
| Vanilla (Mistral-7B-v0.1-Instruct) | 7.43 |
| Arena-Hard (Mistral-7B-v0.1-Instruct) | 6.57 |
Paper: original versus augmented knowledge
Paper v2; source group labels and published precision are retained.
JudgeBench: A Benchmark for Evaluating LLM-Based Judges — Sijun Tan and colleagues. Numeric results transcribed and reformatted; evaluation notes and model links added by BenchLM. Published numeric precision is retained.
| Judge model | Original accuracy (%) and rank | Augmented accuracy (%) and rank |
|---|---|---|
| gpt-4o | 50.65 (3rd) | 46.49 (3rd) |
| gpt-4o-mini | 48.05 (4th) | 44.03 (4th) |
| claude-3.5-sonnet | 62.34 (1st) | 63.25 (1st) |
| claude-3-haiku | 35.06 (6th) | 39.35 (6th) |
| llama-3.1-70b-instruct | 51.30 (2nd) | 52.60 (2nd) |
| llama-3.1-8b-instruct | 38.31 (5th) | 40.00 (5th) |
Original benchmark results and configurations
JudgeBench’s paper and maintained leaderboard report judges under different prompts, reasoning efforts, and response-model splits. Accuracy is a percentage. The paper’s 700-pair GPT-4o split is distinct from the 270-pair Claude split. The app appends the same Nemotron CSV to both tabs without specifying a response split; we show that supplement once with its split unresolved. Live one-decimal values and paper two-decimal values are preserved separately.
Numeric results and configurations from the seven benchmark-owner papers and maintained result tables. This is not a census of all downstream papers or unpublished evaluations.
Snapshot
About JudgeBench
Year
2024
Tasks
Choosing the better of two model responses
Format
Decision classification
Difficulty
Depends on the evaluated split and label mapping
JudgeBench’s paper and maintained leaderboard report judges under different prompts, reasoning efforts, and response-model splits. Accuracy is a percentage. The paper’s 700-pair GPT-4o split is distinct from the 270-pair Claude split. The app appends the same Nemotron CSV to both tabs without specifying a response split; we show that supplement once with its split unresolved. Live one-decimal values and paper two-decimal values are preserved separately.
Freshness and provenance
Version
JudgeBench 2024
Refresh cadence
Annual
Staleness state
Refreshing
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
Which JudgeBench results are included?
This page includes 11 numeric result and configuration tables from JudgeBench’s original paper and available benchmark-owner updates, covering 54 evaluated systems or configurations. Source links, precision, metrics, and evaluation settings remain attached to each table. This coverage does not include every downstream paper or unpublished evaluation.
Can I compare these JudgeBench scores with the Perplexity panel?
Compare JudgeBench scores only when the sample, labels, input representation, and evaluation protocol match. Perplexity’s decision panel uses a fixed sample and its own harness. The original tables preserve different splits, metrics, and model configurations, so their numbers cannot establish a direct ranking against that panel.
Where can I download the JudgeBench results?
The results download on this page provides every imported JudgeBench table as JSON, with model links, published values, source URLs, and evaluation notes. Each linked configuration profile also exports its own result tables as JSON and numeric metrics as CSV. Missing source measurements remain unreported rather than zero.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 2,000+ readers.
One email each week. Unsubscribe anytime.