Best LLMs for Voice AI Agents (2026)

As of August 3, 2026, there is no universal best LLM for voice agents. The right model is the fastest one that clears your own task, instruction-following, and tool-use tests. We use measured first-answer latency to build the shortlist, but we do not turn that one metric into a fake voice-agent ranking. Your speech recognition, network path, text-to-speech layer, and call flow still decide what the caller hears.

For the speech layer, see Best Text-to-Speech APIs. ElevenLabs is one option in that separate decision; its pricing calculator models API character volume and peak concurrency without treating voice quality as a measured result.

Start with one latency budget

A voice turn is a chain: speech recognition, model inference, speech generation, transport, and playback. Each stage adds delay. ElevenLabs' own latency guide makes the same distinction between model inference and the end-to-end time a user experiences. That distinction matters because a vendor's model-latency number is not your caller's time-to-first-audio.

Measure from the end of the user's speech to the first audible sample from the agent. Keep median and tail latency. A beautiful median can hide the calls that feel broken.

Then define the quality gates. A useful voice model must:

• keep answers short enough to hear;
• follow escalation and safety instructions;
• call the right tool with the right arguments;
• recover after interruptions and partial transcripts; and
• pronounce the structured output your speech layer receives.

The model with the highest overall score may clear those gates. It may also arrive late and read JSON aloud. Voice is impolite to abstractions.

Use the live table as a shortlist

The table below rebuilds from the current runtime and model catalogs. It includes models above a minimum overall-quality threshold and orders them by first-answer latency.

Table 1 Download CSV

Model	Latency (first answer)	Output speed	Type	Overall score
Gemini 3 Flash	0.84s	181 t/s	Non-Reasoning	59
GPT-5 (high)	1.12s	77 t/s	Reasoning	58
Claude Sonnet 4.6	1.30s	43 t/s	Non-Reasoning	64
Claude Opus 4.5	1.37s	45 t/s	Non-Reasoning	63
GLM-4.5	1.45s	51 t/s	Non-Reasoning	57
MiniMax M2.5	2.12s	46 t/s	Non-Reasoning	58
Claude Opus 4.6	2.60s	37 t/s	Non-Reasoning	68
Kimi K2.6	2.91s	30 t/s	Reasoning	59
GPT-5.4 nano	3.64s	191 t/s	Reasoning	66
GPT-5.4 mini	3.85s	201 t/s	Reasoning	56

The latency column is first answer from the runtime source, not full voice-to-voice delay. Output speed and overall score are screening signals. The speed dashboard carries the current rows and source notes.

We deliberately stop short of naming the first row as the winner. Runtime observations can change with provider load, region, prompt length, and serving updates. More importantly, this table does not test your system prompt, tool schema, accents, background noise, or call-transfer rules.

Take three candidates into a call replay set. Use real, consented transcripts or synthetic cases that match the same turn lengths and tool calls. Record task success, wrong-tool rate, first audio, interruptions, and cost. That small test tells you more than another paragraph about model families.

Turn the shortlist into 100 calls

Build a replay set before changing the production route. Fifty calls should represent the ordinary work that drives volume. The other fifty should concentrate failures: noise, accents, names, account numbers, corrections, interruptions, tool timeouts, ambiguous confirmation, unsupported requests, and human transfer.

Hold the speech recognizer, voice, prompt, tools, region, and call transport constant while testing the text models. If the candidate requires a different architecture, such as native audio, preserve the input clips and score the user-visible outcome rather than pretending the internal stages match.

Use one trace per conversation and record:

• task completion and expected final state;
• correct tool name, arguments, and authorization;
• duplicate or unauthorized actions;
• end-of-speech to first audible response at p50 and p95;
• interruptions, false turn endings, and repeated questions;
• input, output, audio, and cached usage;
• transfers and the context delivered to the human; and
• model and platform cost per completed task.

Score failures by severity. A slightly slow greeting and an unauthorized account change do not belong in the same average. Set hard release blockers for data exposure, wrong high-impact actions, duplicate writes, and lost transfers; use weighted scores only for the remaining product tradeoffs.

Run each candidate more than once if provider variance is material. Randomize model order so time-of-day load does not favor one endpoint, and preserve the raw timing events. When the result is close, choose the system the team can operate, debug, and roll back.

The replay produces a local, defensible winner with an audit trail. It does not create a universal best model, which is exactly the point.

Keep Mercury 2's two clocks separate

Inception says Mercury 2 generates 1,009 tokens per second on NVIDIA Blackwell. Its separate voice-agent report describes a roughly 300 ms reasoning trace and customer-observed model latency below about 500 ms. Those are provider-published results from specified deployments, not this page's first-answer measurement.

The current cross-provider runtime row measures Mercury 2 at 802 output tokens per second and 3.53 seconds to the first answer. That combination makes it a throughput candidate, not the measured TTFT winner. It also explains why a vendor can honestly describe fast generation while a different test records a slower initial response: output rate and first-answer delay are separate clocks.

Mercury 2 belongs in a cascaded speech-to-text, text-model, and text-to-speech stack. It does not accept or return native audio. Compare that design with GPT-Realtime 2.1 and Gemini 3.1 Flash Live Preview before treating text throughput as voice latency.

Keep slow work off the conversational path

A two-model design is useful when ordinary turns are simple but some tasks require deeper work.

Table 2 Download CSV

Path	Job	Selection rule
Conversational model	Greeting, clarification, short answers, routine tool calls	Lowest measured latency that clears the call eval
Background model	Policy synthesis, multi-document lookup, complex calculations	Highest task success inside the allowed wait and cost

The conversational model can acknowledge the request, start the tool call, and continue when the result returns. ElevenAgents documents optional tool-call sounds for filling a long operation; the more general lesson is to tell the caller that work is happening instead of leaving an unexplained silence.

This pattern has a limit. Routing adds its own failure states: duplicate actions, stale results, and a fast model summarizing a slow model incorrectly. Keep one trace across the turn and test the handoff, not just the two models separately.

Map the platform before choosing it

We have not run a controlled benchmark across voice-agent platforms. The table is a product-architecture map built from current first-party documentation, not a quality or latency ranking.

Table 3 Download CSV

Approach	Documented architecture	Model choice	Where it loses
OpenAI Realtime	Native realtime audio and speech-to-speech	OpenAI realtime models	Less component freedom
Vapi	Orchestrates transcription, model, voice, and telephony	Multiple providers or compatible custom endpoints	More vendor boundaries to debug
Retell	Voice-agent and telephony orchestration	Retell or configured response engine	Platform-specific call configuration
ElevenAgents	Speech, turn-taking, tools, workflows, and monitoring	Supported or custom LLM	More of the stack sits with one vendor
Build the chain	Your STT, model, TTS, transport, and state	Full control	You own interruption handling and operations

OpenAI's route removes boundaries between speech and reasoning. Vapi's documentation exposes replaceable transcriber, model, and voice components. ElevenAgents documents custom models, tools, telephony, testing, and analytics in one platform. Retell exposes agent and response-engine configuration. Those are verifiable differences; "sounds most human" is not, because we have not run the listening test.

If that managed speech-and-agent architecture fits the replay test, use the ElevenLabs pricing calculator to size the API quota and concurrency limit before opening an account.

Pick by failure mode

For a customer-service phone agent, start with the fastest three models that pass the task suite. Tool errors and escalation failures matter more than a small leaderboard gap.

For an internal assistant, a slower model may be acceptable when users can see that work is in progress. Measure the actual workflow rather than borrowing a consumer-call threshold.

For complex backend work, split the conversational and background paths. Make the acknowledgment explicit, attach an idempotency key to actions, and prevent the background result from firing the same tool twice.

For multilingual calls, cross-check the multilingual leaderboard with the languages and voices your speech provider actually supports. Then test code-switching and names. A language appearing on two feature lists does not prove that the pair works well together.