Time to First Token Explained

Time to first token is the wait for the first generated token or usable response chunk.

It captures the start of an answer, not the speed of the complete response.

We use TTFT as one timing boundary alongside output rate, completion time, and task success.

TTFT is useful only when the timing boundary and workload are explicit.

The TTFT clock starts before generation

A typical request crosses several stages:

1. the client opens or reuses a connection,
2. the provider authenticates and admits the request,
3. a scheduler queues it for compute,
4. the model processes the input prompt, often called prefill,
5. the model produces its first output, and
6. the serving layer sends the first usable chunk back to the client.

TTFT normally includes all six from the client's point of view. A server-side metric may exclude DNS, connection setup, or internet transit. A provider may report model time after queueing. Neither is automatically wrong, but the labels are not interchangeable.

BenchLM's speed dashboard uses the runtime feed's first-answer measurement. It should not be read as full voice latency, browser render time, or a guarantee for another region.

TTFT and output speed answer different questions

Suppose Model A begins in 0.4 seconds and produces 40 tokens per second. Model B begins in 2.0 seconds and produces 300 tokens per second.

For a ten-token classifier response, Model A probably finishes first. For a 2,000-token report, Model B can overtake it after the initial delay. The crossover depends on output length:

estimated completion time = TTFT + output tokens ÷ output tokens per second

That rough formula ignores stop time, network jitter, variable decoding rate, and reasoning tokens, but it exposes the trade. A single “fastest model” label hides two clocks.

Table 1 Download CSV

Workload	Metric that usually dominates	Why
Voice turn	End-of-speech to first audio	Silence feels broken before a long answer exists
Chat answer	TTFT and readable streaming	Early text confirms that work started
Router or classifier	Full completion time	The application needs the short result before acting
Long report	Output rate and completion time	Most time is spent after the first token
Background batch	Throughput and cost	No user is watching the initial wait

Current first-answer rows are screening evidence

This table rebuilds from the current runtime and model catalogs. The quality floor prevents a very fast but weak or unverified model from becoming a universal recommendation.

Table 2 Download CSV

Model	Latency (first answer)	Output speed	Type	Overall score
Gemini 3 Flash	0.84s	181 t/s	Non-Reasoning	59
GPT-5 (high)	1.12s	77 t/s	Reasoning	58
Claude Sonnet 4.6	1.30s	43 t/s	Non-Reasoning	64
Claude Opus 4.5	1.37s	45 t/s	Non-Reasoning	63
GLM-4.5	1.45s	51 t/s	Non-Reasoning	57
MiniMax M2.5	2.12s	46 t/s	Non-Reasoning	58

The table is a shortlist, not a service-level promise. A provider update can change the order without changing the model name. Region, account tier, concurrency, prompt length, reasoning setting, and cache state also affect the request.

Use the full speed dashboard to sort by first answer, output rate, price, and score. Then repeat the measurement on the deployed route.

Prompt processing can become the hidden cost

Before decoding output, a transformer processes the input tokens and builds the state needed for generation. Longer prompts increase that prefill work. Large tool schemas, retrieved documents, conversation history, images, and system instructions all belong to the input even when the user typed one sentence.

Prompt caching can reduce repeated work, but cache rules differ. A request may miss because one early token changed, the cached prefix expired, the region changed, or the provider routed to another model version. Record cache status rather than averaging hits and misses into one unexplained number.

Reasoning settings add another variable. Some systems perform hidden or visible reasoning before a usable answer appears. The first network event may contain metadata rather than content the application can show. Define “first token” as the first usable answer chunk if that is what the user experiences.

Measure a distribution, not a demo

One request proves that the endpoint worked once.

For a useful test, create a fixed set of prompts at several input and output lengths. Warm the client connection separately from cold starts. Send requests at the concurrency the application expects, from the target region, through the same provider and API path that production will use.

Report at least:

• p50 TTFT for the typical request,
• p95 or p99 TTFT for the slow tail,
• output tokens per second after the first chunk,
• full completion time,
• input and output token counts,
• errors, retries, and rate-limit responses, and
• task success for the returned answer.

Keep failed requests in the operational report. Deleting timeouts can make the surviving latency distribution look excellent while the product is unusable.

Voice needs an end-to-end clock

Text TTFT is one segment of a voice turn. A cascaded system adds end-of-speech detection, transcription, model inference, speech synthesis, transport, buffering, and playback. A native audio model combines some stages but still has network, reasoning, synthesis, and playback delay.

Measure end of user speech to first audible agent sample. That is time to first audio, not TTFT. Track interruption recovery and false end-of-turn decisions beside it, because an aggressive endpoint detector can look fast by cutting the caller off.

The voice-agent model guide uses text first-answer rows to build a shortlist and then requires a call replay. That is the correct boundary between a model metric and a product decision.

Optimize after identifying the stage

If queueing dominates, change capacity, tier, region, concurrency, or provider. If prefill dominates, trim repeated instructions, retrieve fewer better documents, reduce tool schemas, or use a stable cacheable prefix. If decoding dominates, choose a faster model, request less output, or move long work off the interactive path.

Streaming improves perceived speed only when partial output is safe to show. A tool call cannot execute half an argument, and a structured response may be invalid until its closing token arrives. For those workloads, total time to a valid result matters more than the first byte.

TTFT is a precise metric with a narrow job: measure the opening wait. Keep it narrow, and it becomes useful.