# How to evaluate an LLM before you pick one

> Evaluate an LLM in two passes: public benchmarks for a shortlist, then 30 to 50 of your own cases. The model's developer publishes 54% of the scores we show.

- Published: 2026-11-10
- Last reviewed: 2026-11-10
- Data as of: 2026-11-10
- Article slug: how-to-evaluate-an-llm
- Author: [Glevd](https://x.com/glevd)
- Reading time: 8 minutes
- Topics: llm-evaluation, benchmarking, evals, model-selection, guide
- Data policy: Dated analysis; static benchmark claims are retained.
- Canonical URL: https://benchlm.ai/blog/posts/how-to-evaluate-an-llm

Evaluate an LLM in two passes. Public benchmark scores cut the field to two or three models, and 30 to 50 cases from your own work decide between them.

Public scores can't do the second pass for you. As of November 10, 2026, our model pages show 6,203 benchmark results, and the model's own developer published 54% of them.

So a leaderboard tells you which models to test. It can't tell you which one will handle your support tickets or your codebase.

## An overall rank hides the test that matches your job

Overall ranks average work you may never ask for. One model can lead on fixing code and sit mid-table on legal reasoning, so make the first cut from the benchmark that looks most like your task and leave the overall table for later.

"Looks like your task" means the test asks for the same kind of output and grades it the way you would. Here are five we track, and what each one checks.

| If the job is | Look at | What the test grades |
| --- | --- | --- |
| Fixing bugs in an existing codebase | [SWE-bench Verified](/benchmarks/swe-bench-verified) | Whether the patch resolves a real GitHub issue |
| Multi-step work in a terminal | [Terminal-Bench](/benchmarks/terminal-bench) | Whether a professional command-line task ends in the right state |
| Support with tools and a policy to follow | [τ²-bench](/benchmarks/tau2-bench) | Whether the agent reaches the right outcome for a simulated customer |
| Operating a desktop or a browser | [OSWorld-Verified](/benchmarks/osworld-verified) | Whether the task is finished on a real computer |
| Hard science questions | [GPQA Diamond](/benchmarks/gpqa-diamond) | Multiple-choice answers to graduate-level questions |

Our [job pages](/best/for) do this step for common jobs, ranking models on the benchmarks closest to each one. If nothing public resembles your work, skip ahead, because your own cases will carry the whole decision.

Take the top of the closest table, then remove what you can't use. That's any model over your budget, any model that isn't offered where your data has to stay, and any model that needs hardware you don't have. You want two or three names left, plus the model you run today.

## Most scores are published by the model's developer

We count every result on our model pages by who published it. On November 10, 2026, the company selling the model was the largest source, ahead of the benchmarks' own leaderboards.

| Who published the result | Results | Share |
| --- | --- | --- |
| The model's developer | 3,377 | 54% |
| The benchmark's own leaderboard | 2,251 | 36% |
| An independent evaluator | 295 | 5% |
| Another lab's comparison table | 241 | 4% |
| Carried from an upstream source | 39 | 1% |

Nothing is wrong with a developer's number just because the developer ran it. But the developer picked the settings: the prompt, the reasoning effort, the tools the model could call, and how many attempts counted. Two parties can report different scores for one model on one benchmark for those reasons alone.

So compare two models on results that one party published under one setup. When all you have is each developer's own figure, treat the gap between them as unconfirmed.

We deal with this in the ranking by labeling each position. Supported means we have enough independent evidence to stand behind the row. Estimated means the rank is visible and the uncertainty is wider. On November 10, 77 of the 214 ranked models were Supported and 137 were Estimated, and [where our scores come from](/score-sources) shows how each kind of source counts.

## A two-point lead is usually a tie

Many public tests are crowded at the top. Of the 192 score tables in our data with at least ten models on a 0 to 100 scale, 24 have their five highest scores within two points of each other. On [MATH-500](/benchmarks/math-500) the top five run from 98.8 to 98.0, and on [MMLU-Pro](/benchmarks/mmlu-pro) from 89.6 to 87.8.

Such a test can still show you that a model is far behind. It can't choose between the leaders, because the order among them mostly reflects setup differences and run-to-run variation.

Sample size sets a floor under every score, too. If a model gets 80% on a 500-question test, sampling alone puts a margin of about 3.5 points either way on that score, at the usual 95% level. On 100 questions the margin is about 8 points.

A published table shows only each model's total, so you can't check a small lead question by question. Treat any lead smaller than the margin as a tie, and let price, speed, or your own test break it.

## Your own cases make the decision

Run the shortlist on a test built from your work. Your best source is the folder of outputs that were already wrong: tickets, documents, or requests where someone read the answer and knew why it failed. Each of those comes with a pass rule attached, and writing the rule down as an expected value is most of the cost.

Thirty to fifty reviewed cases is enough to start. Then keep the comparison fair.

1. Freeze the cases, the expected answers, and the prompt before any model runs.
2. Give every model the same prompt, temperature, and token budget.
3. Interleave the runs, so a provider outage hits every model and not only the one that ran last.
4. Run each case three times. One failure in three runs is a different problem from three in three.
5. Grade with exact checks wherever the right answer can be written down, and keep a person on the cases where it can't.

We built and filed a test this way on September 14, 2026: 30 synthetic support tickets, three runs each, two configurations. Of the first configuration's 90 attempts, 87 completed, 86 returned valid JSON, and 84 matched every reviewed field. Stop the report at "valid JSON" and two wrong answers count as right. You can read the walkthrough and download the kit in [Test the replacement on what already fails](/blog/posts/building-custom-llm-benchmark).

Pick whichever test framework your team can maintain. On September 22, 2026, we replayed one set of 372 saved outputs through Promptfoo and through DeepEval, twice each, and all four runs produced the same passes and failures. Cases and checks decided the result. Our [comparison of the two](/llm-evaluation-frameworks) covers what each tool is like to live with.

## A cheaper model can cost more per accepted answer

Report the result as accepted answers out of attempts for each model, with the failed cases listed. Without that list, a percentage hides whether the failures are the ones your application can't tolerate.

Then divide what the run cost by the accepted answers. In our September 14 run that was $0.20128 for 84 accepted responses, about a quarter of a cent each, at list rates of $2 and $10 per million input and output tokens.

Retries change that math. When a cheaper model needs more of them, it can cost more per accepted answer than a dearer one that passes the first time, and you only see it when failed attempts stay in the cost. Current rates are on the [pricing table](/llm-pricing).

Keeping the model you have is a valid result.

If two models differ by a couple of cases on a set of thirty, the test hasn't separated them. Add cases before you switch.

## Last month's result describes last month's models

Providers release new versions and retire old ones, and an alias that means "latest" can point at a different version a few months later. Date every run, and record the exact model ID beside it.

Keep the frozen cases, the prompt, and the run record in your repository. With those in place, testing a new release takes an afternoon.

When a new model appears on a public table, look at two things before the score: whether its position is Supported or Estimated, and whether anyone besides its developer has published a result. If the answer to the second is no, the launch numbers are a reason to add the model to your next run. They aren't a reason to switch.

Carry one figure between runs: accepted answers per dollar on your own cases, with the date and the list of failures beside it.

## Frequently asked questions

### What is LLM evaluation?

LLM evaluation is the work of measuring whether a language model's output is good enough for a defined task. It covers public benchmarks, which compare many models on shared tests, and private evaluations, often called evals, which test one application's prompts and data. Benchmarks narrow the choice of model, and evals show whether your system works.

### How many test cases do I need to evaluate an LLM?

Start with 30 to 50 reviewed cases drawn from real work, including known failures, and run each one three times. That many cases will catch a model that breaks a required field or rule. They can't measure a general error rate, so add cases before you treat a difference of two or three as real.

### Should I use an LLM as a judge?

Use exact checks first. Schema validity, identifiers, labels, and quoted fields can all be compared with code. For open-ended output such as summaries or tone, a model grader fits. Check it against a sample a person has graded, and keep the judge model and its prompt fixed, because changing either one changes the scores.

### What is the difference between a benchmark and an eval?

Benchmarks are public tests that many models take, so their results can be compared across models. Evals are tests you build for one application, with your inputs and your pass rule. Benchmark scores tell you which models are worth testing, and an eval tells you whether one of them does your job.