Diffusion vs Autoregressive Language Models

Autoregressive language models generate one token after another.

Diffusion language models refine several masked or noisy positions across denoising steps.

We treat parallel decoding as an engineering opportunity, not proof of lower latency, higher quality, or cheaper serving.

The architecture creates a different optimization space.

Autoregressive decoding is sequential by design

An autoregressive model receives a prompt, predicts a probability distribution for the next token, selects one token, appends it, and repeats. Every new token depends on the generated prefix. That dependency makes the core decoding loop sequential.

Serving systems still exploit parallel hardware. They batch requests, process prompt tokens together during prefill, reuse cached prefixes, and sometimes use a smaller draft model to propose several tokens for verification. Those techniques can produce high throughput without changing the model's left-to-right factorization.

The practical strengths are maturity and predictability. Most model APIs, tool-call protocols, streaming interfaces, quantization methods, and inference engines were built around autoregressive decoding. The ecosystem knows how to operate it.

Diffusion can revise several positions

A text diffusion model starts from a sequence containing masked or noisy positions. Across multiple steps, it predicts replacements, keeps high-confidence tokens, and revises uncertain positions. Several tokens can change during one invocation.

The LLaDA paper describes a masked-diffusion language model trained to reconstruct randomly masked tokens. Dream publishes another diffusion reasoning implementation. DiffusionGemma uses a mixture-of-experts design and exposes open experimental weights.

These systems do not all use one algorithm. Masking schedules, sequence length, confidence thresholds, step counts, bidirectional context, and post-training differ. “Diffusion LLM” is a family label, not a shared service profile.

Parallel work does not remove serving bottlenecks

Updating multiple positions can reduce the number of sequential decoding steps. It can also do extra computation on positions that later change. A model that needs many denoising passes may give back part of the theoretical gain.

User-visible latency includes more than decoding:

Table 1 Download CSV

Stage	Autoregressive pressure	Diffusion pressure
Prompt prefill	Process the input before decoding	Process the input before denoising
Output generation	Sequential token dependency	Multiple refinement steps
Streaming	Natural token-by-token chunks	Requires a policy for stable, displayable text
Tool calls	Arguments grow left to right	Arguments may be revised before commitment
Serving	Mature optimized engines	Newer kernels and scheduling paths

A provider can publish a large sampling rate while the endpoint still has a noticeable first-answer delay. Mercury 2 demonstrates the distinction: its hosted row shows high output throughput and a slower initial response than the fastest comparable diffusion row.

Memory behavior can differ as well. Parallel refinement may change how serving systems allocate compute across a sequence, while autoregressive decoding maintains a growing cache of prior token state. The production effect depends on the implementation, batch, sequence length, and hardware, the architecture label alone does not predict capacity per accelerator.

The public numbers are not one benchmark

Inception reports 1,009 tokens per second for Mercury 2 on NVIDIA Blackwell. Google reports 1,479 tokens per second of sampling speed for Gemini Diffusion and lists overhead separately. DiffusionGemma's page reports more than 1,000 tokens per second on one H100.

Those are provider-published results with different models, prompts, hardware, serving paths, and timing boundaries. Dividing one by another does not yield a trustworthy speed advantage.

The diffusion LLM tracker ranks only models that share the current hosted runtime feed. Other rows remain visible and unranked. This preserves useful evidence without manufacturing precision.

Quality does not come from decoding alone

Diffusion does not turn an undertrained model into a strong reasoner. Autoregression does not prevent a model from serving an interactive product. Architecture interacts with scale, data, objectives, post-training, tool instruction, and inference budget.

Evaluate exact checkpoints on exact tasks. For a coding agent, run repository-level fixes and tool calls. For search, test citation support and abstention. For voice, measure first audible response and interruption recovery. A general benchmark can screen candidates, but it cannot substitute for the failure modes of the application.

Missing evidence should stay missing. A provider table can inform a trial while the model remains outside an independent quality rank.

Streaming is the clearest product difference

Autoregressive output is append-only: once a token is streamed, the model normally does not edit it. A diffusion decoder may revise positions until they are stable. The serving layer must decide when a span is safe to release.

That decision affects perceived latency and correctness. Commit too late and the system hides the architecture's speed. Commit too early and text may need corrections, which is awkward for users and dangerous for JSON, code, or tool arguments.

For a voice system, revisions are even harder because spoken audio cannot be silently edited. The implementation needs stable chunks before synthesis or a strategy that confines revision to unsaid text.

Pick a model, not a decoding ideology

Choose a hosted diffusion model when its measured first-answer time, output rate, price, task success, and API behavior beat the alternatives for the deployed workload. Choose an open diffusion model when research access or self-hosting justifies the newer serving path.

Choose an autoregressive model when it wins the same test or when mature tooling, broad provider access, stable streaming, and operational familiarity matter more than experimental throughput.

Then repeat the test after a provider update.

The useful distinction is measurable behavior. Architecture explains part of it, production evidence makes the decision.