Choosing the Best LLM for RAG

As of July 28, 2026, we do not publish a universal RAG-model winner. The best LLM for your RAG system is the cheapest candidate that clears a grounded-answer eval on your corpus, questions, retrieval settings, citation rules, and latency budget. Long-context and instruction-following benchmarks can build a shortlist. They cannot substitute for the end-to-end test.

The earlier version of this page blended unrelated public benchmark rows into named RAG winners. We removed those matches. A model can lead a knowledge or long-context benchmark and still fail because retrieval returned the wrong document, the prompt demanded an unsupported citation, or the application passed noisy chunks.

Separate retrieval from generation

RAG hides two systems behind one answer:

1. Retrieval finds the evidence.
2. Generation uses that evidence and stays inside the requested format.

Score them separately. If the correct passage never reaches the model, changing the generator is mostly theatre. If the passage is present and the answer is still unsupported, the model or prompt owns the failure.

Table 1 Download CSV

Failure	Component to inspect first	Useful measure
Correct document never retrieved	Index, query, embedding, filters, reranker	Recall at K
Correct document retrieved but ignored	Prompt packing and model	Evidence-use rate
Claim lacks support	Model, citation contract, evaluator	Grounded-claim precision
Citation points to wrong passage	Chunk identity and output parser	Citation accuracy
Valid answer breaks JSON or schema	Model and constrained output	Schema pass rate
Answer is correct but too slow or expensive	Retrieval depth, context, model	End-to-end latency and cost

This table is the reason a single "RAG score" is tempting and usually dishonest. Each row has a different owner.

Use public benchmarks as screens

LongBench v2 tests long-context understanding under a defined benchmark setup. The current sourced rows are:

Table 2 Download CSV

Rank	Model	Score	Evidence
1	Qwen3.8 Max	66.3%	Provider exact
2	Claude Opus 4.5	64.4%	Secondary exact
3	Qwen3.5 397B	63.2%	Provider exact
4	Qwen3.6 Plus	62%	Provider exact
5	Nemotron 3 Ultra	61.9%	Provider exact
6	Kimi K2.5	61%	Provider exact
7	GLM-5	60.8%	Secondary exact
8	Qwen3.5-27B	60.6%	Provider exact

IFEval tests verifiable instruction constraints. It can help screen models for citation formats, concise answers, or structured output, but it does not measure whether a citation supports a claim.

Table 3 Download CSV

Rank	Model	Score	Evidence
1	Qwen3.5-27B	95%	Provider exact
2	Agents-A1	94.8%	Provider exact
3	Qwen3.7 Plus	94.6%	Provider exact
4	Qwen3.7 Max	94.3%	Provider exact
5	Qwen3.6 Plus	94.3%	Provider exact
6	dots3-note Preview	93.9%	Provider exact
7	o3-mini	93.9%	Provider exact
8	Kimi K2.5	93.9%	Secondary exact

These tables rebuild from displayable verified rows. They are not a RAG ranking. Benchmark coverage varies, provider and secondary evaluations may use different harnesses, and a candidate missing from a table has missing evidence rather than a score of zero.

Knowledge benchmarks are even easier to misuse. They test what a model can answer or reason about under their own conditions. RAG asks whether the model will use supplied evidence, including evidence that contradicts its prior. A strong knowledge score may help with comprehension; it does not prove grounding.

Build the eval before choosing the model

Start with questions, not vendors. A useful first set can be small enough to review by hand and broad enough to catch the system's real failure modes.

Include:

• direct lookups with one supporting passage;
• questions where the corpus contains no answer;
• two documents that disagree by date or version;
• multi-document questions that require combining evidence;
• requests with strict output and citation rules; and
• adversarial chunks that contain instructions unrelated to the user's task.

For every question, store the expected source documents and the acceptable answer facts. Do not require one exact sentence when several formulations are correct. Do require every material claim to point to supporting evidence.

Run retrieval once and freeze the returned contexts for the model comparison. That isolates generation. Then run the full pipeline separately to measure how retrieval and generation interact. Without both views, a new embedding model can make the LLM look better and a reranker regression can make it look worse.

One more refusal: an LLM grading another LLM is not a complete evaluation plan. Use deterministic checks for citations and schemas, human review on a stratified sample, and a model judge only where its rubric has been tested against those human decisions.

Choose by the failure you can afford

For customer-support lookup, the useful model is often the fast, inexpensive one that cites one approved article and abstains when the article is missing. Multi-hop brilliance is wasted if the product's hard rule is "never infer beyond the help center."

For research synthesis, a slower reasoning path may earn its cost when the answer must reconcile several documents. Route direct lookups to the simpler model and reserve the expensive path for questions that actually need it.

For structured extraction, instruction following and schema pass rate can dominate broad knowledge. A model that returns beautiful prose around invalid JSON creates operational work downstream.

For long documents, test retrieval before increasing the context window. Passing an entire manual can improve recall and worsen distraction, cost, and latency at the same time. The long-context ranking shows sourced capability evidence; your eval decides how much context to send.

For self-hosting, compare the full operating cost: GPUs, replicas, batching, idle capacity, model loading, observability, and the engineering time to keep inference healthy. Zero API price is not zero cost. The self-host calculator is the appropriate starting surface.

Price the accepted answer

Token rates are only one line in the budget. Count:

Table 4 Download CSV

Cost	Why it moves
Embedding and indexing	Corpus size and refresh cadence
Retrieval and reranking	Queries, candidates, and reranker model
Input tokens	Chunk count, chunk length, history, and system prompt
Output tokens	Answer length and retries
Cache behavior	Repeated context and provider rules
Failed or abstained calls	Retry policy and fallback model
Human review	Risk tier and evaluation sampling

Use the live LLM pricing page and cost calculator after the quality gate. Comparing list prices before a model passes the eval optimizes a system you should not ship.

The metric we would keep is cost per accepted answer: total pipeline cost divided by answers that meet retrieval, grounding, citation, format, latency, and safety thresholds. It makes a cheap model with many retries look expensive and a pricier model with a high pass rate compete fairly.

Fix the corpus before blaming the model

Web data arrives with duplicate pages, obsolete versions, navigation text, and uncertain permissions. The current data-pipeline article shows why source authority and validation sit before publication. The web-scraping roundup maps acquisition tools without pretending that "clean markdown" proves corpus quality.

For RAG, retain the source URL, retrieval time, document version, and access or licensing basis. Deduplicate before embedding. Keep titles and headings so citations can point to something a user recognizes. Delete revoked content from both the primary store and derived index.

And test freshness. A system that retrieves last year's correct policy is a confident archive, not a knowledge assistant.