# Inference is the part of AI you pay for

> Inference is the moment a trained model answers a prompt. Training built the weights once. Inference runs them on every request, is priced per token, and is where latency, throughput and your bill all come from.

- Published: 2026-09-17
- Last reviewed: 2026-09-15
- Data as of: 2026-09-14
- Article slug: what-is-inference-in-ai
- Author: [Glevd](https://x.com/glevd)
- Reading time: 9 minutes
- Topics: inference, basics, latency, tokens, runtime
- Canonical URL: https://benchlm.ai/blog/posts/what-is-inference-in-ai

Inference is the moment a trained model answers. Training built the weights once. Inference runs them on every prompt you send, and it is the part you pay for.

Most explanations stop there. What follows from that sentence is the useful part.

Why does a reply arrive in two phases? Why does output cost more than input? Why is one paragraph billed as a different number of tokens by different providers? And why does speed belong to the host as much as to the model?

## Inference means running the model, not building it

A language model is a large set of numbers, its weights, plus a procedure for turning text into more text using those numbers. Training is the expensive, one-off process that sets the weights. Inference is everything after: the weights stay fixed, your prompt goes in, and the model computes a probability for the next token, picks one, appends it, and repeats until it stops.

Nothing is learned during inference. Your request is not remembered afterwards, and your prompt does not change the weights. What looks like memory inside a chat is the application resending the conversation with every turn, which is why long conversations cost more with each message.

Two consequences follow. First, a provider's cost is per request, not per model, so a price per token is how inference is sold. Second, anything you can do to shorten the input, shorten the output, or reuse a repeated prefix reduces the bill directly, because the bill is the inference.

## Every reply is built in two phases

When a request lands, the model first reads the whole prompt. Engineers call that prefill. Prompt tokens are processed largely in parallel, and nothing comes back to you until prefill finishes. Then the model generates the reply one token at a time, a phase called decode. Each output token is a full pass through the model, and it cannot start until the previous token exists.

Two numbers describe those phases from the outside. Time to first token measures prefill plus queueing, which is how long you wait before anything appears. Long prompts and busy hosts stretch it. Output speed, in tokens per second, measures decode, which is how fast the reply streams once it starts.

One model can be quick on one and slow on the other. Our [time-to-first-token explainer](/blog/posts/time-to-first-token-explained) covers what moves the first number. The table below shows both for models that also clear a quality bar.

| Model | Latency (first answer) | Output speed | Type | Overall score |
| --- | --- | --- | --- | --- |
| Gemini 3 Flash | 0.94s | 193 t/s | Non-Reasoning | 62 |
| Claude Sonnet 4.6 | 1.17s | 42 t/s | Non-Reasoning | 63 |
| GPT-5 (high) | 1.21s | 74 t/s | Reasoning | 57 |
| Claude Opus 4.5 | 1.27s | 44 t/s | Non-Reasoning | 58 |
| GLM-4.5 | 1.45s | 51 t/s | Non-Reasoning | 56 |
| Claude Opus 4.6 | 2.06s | 37 t/s | Non-Reasoning | 69 |
| Kimi K2.6 | 3.05s | 36 t/s | Reasoning | 66 |
| GPT-5.4 nano | 3.64s | 191 t/s | Reasoning | 59 |

Median output speed and first-answer latency come from Artificial Analysis's hosted measurements, refreshed September 2026. The full list with filters is on [LLM speed](/llm-speed). One caution the table cannot show: one open-weight model runs at very different speeds on different hosts, because decode speed is a property of the hardware and serving stack, not only of the weights.

## Inference is priced per token, in and out

Providers quote inference in dollars per million tokens, with separate rates for input (your prompt) and output (the reply). Output costs more because each output token is its own forward pass. Input is cheaper because prefill runs in parallel. A third rate, cached input, applies when a repeated prefix such as a system prompt or a long document is read from a cache instead of being processed again.

Three current list prices, input and output per million tokens, read from our pricing registry as of September 18, 2026:

- Claude Sonnet 5: $2/$10
- GPT-5.6 Terra: $2/$12
- DeepSeek V4.1 Flash: $0.3/$1.2

Those are unit prices. What a request costs depends on how many tokens it moves, and a token is roughly four characters of English, about three-quarters of a word. [AI token cost](/ai-token-cost) works through what an email, a report and a book cost to read or write at today's rates, and the [pricing calculator](/llm-pricing#llm-pricing-calculator) applies any model's three rates to a monthly workload you type in.

## One text costs different tokens on different models

Each provider tokenizes text with its own vocabulary, and each injects its own scaffolding around your messages. So a price per million tokens does not predict cost per call until you know how many tokens the call will be billed as.

We measured this in the [structured-output evidence run](/blog/posts/test-structured-output) filed on September 14, 2026. Identical messages and an identical JSON schema were sent to Claude Sonnet 5 and GPT-5.6 Terra through one gateway. Sonnet was billed about 782 prompt tokens per call, Terra about 303. In the long-prompt profile, one 11,094-character handbook wrote 4,214 tokens to Sonnet's cache and 3,481 to Terra's. Same text, different token counts, and the difference is in the tokenizer and the request format, not in anything you wrote.

Compare cost per completed task, not price per token. Our [pricing hub](/llm-pricing) shows list rates. The run linked above shows how far they can drift once a real request is billed.

## Reasoning models pay for tokens you never see

Reasoning-capable models generate hidden tokens before the visible reply. Those tokens are inference too, and providers bill them as output. In the run above, one case failed on every repetition because 299 to 300 reasoning tokens consumed the whole 300-token completion budget and the JSON was cut off mid-string. The reply looked empty. The meter had run.

If you set a maximum output length for a reasoning model, budget for the thinking as well as the answer. If you compare a reasoning model's price with a non-reasoning model's, remember the reasoning model produces more output tokens per request than the reply shows. The type column in the speed table above marks which models think before answering.

## Where inference runs decides what you pay and how fast it is

One model can be served from several places: the provider's own API, a cloud platform such as Azure AI Foundry, Vertex AI or Amazon Bedrock, a gateway such as OpenRouter, a specialist host such as Groq or Cerebras, or your own hardware. Each route sets its own price, its own speed and its own retirement schedule.

Three things change with the route:

- **Price.**
  
  Cloud routes and gateways add or remove margin, and open-weight hosts compete on it.
  
  [Azure OpenAI pricing](/azure/llm-pricing)
  
  beside
  
  [OpenAI's own rates](/openai/api-pricing)
  
  shows one such gap.
- **Speed.**
  
  Decode speed follows the serving hardware. Our
  
  [OpenRouter alternatives audit](/openrouter-alternatives)
  
  ran one workload across seven gateways and hosts and scored the speed specialists separately, because they answered the same requests at a different speed, a different price and with a shorter model list.
- **Lifecycle.**
  
  A retirement date belongs to the route, not to the model, and we have traced
  
  [one model carrying three different retirement dates](/blog/posts/same-model-three-retirement-dates)
  
  . Our
  
  [deprecation calendar](/deprecations)
  
  keeps each date beside its source.

Running inference yourself removes the per-token price and replaces it with hardware, electricity and the engineering to serve the model well. [Best local LLMs](/best/local-llm) ranks the open-weight models by the memory they need.

## Four numbers keep a model decision honest

Inference is where every model decision becomes a bill, so record the assumptions that set it:

1. Tokens in and out per request, measured on your own prompts with the
   
   [token counter](/tools/token-counter)
   
   , not estimated from word counts.
2. Requests per month, and how many share a cacheable prefix.
3. Whether the model reasons, and the output budget that leaves for the reply.
4. Which route you will call, and its price, speed and retirement date on the day you decide.

Those four numbers are what the [pricing calculator](/llm-pricing#llm-pricing-calculator) needs, and they are what changes when a provider reprices or retires a model. Keep them beside the decision. [Radar](/radar) tracks price and lifecycle changes for the models you name and keeps a cost scenario with its sources attached, so the next review starts from the numbers you actually used.

## Glossary

| Term | Meaning |
| --- | --- |
| Inference | Running a trained model on a new input to get an output |
| Training | The one-off process that sets a model's weights |
| Token | The unit a model reads and writes, about four characters of English, and the unit prices are quoted in |
| Prefill | The phase where the model reads the prompt. It sets time to first token |
| Decode | The phase where the model writes the reply one token at a time. It sets output speed |
| Time to first token | The wait before the first output token arrives |
| Tokens per second | How fast the reply streams once it starts |
| Cached input | A repeated prompt prefix read from a cache at a lower rate |
| Reasoning tokens | Hidden tokens a reasoning model generates before the visible reply, billed as output |

## Frequently asked questions

### What is inference in AI?

Inference is running a trained model to produce an output from a new input. For a language model that means reading your prompt and generating the reply, one token at a time. Training happened once, before you arrived. Inference happens on every request, and it is what an API bills you for.

### What is the difference between training and inference?

Training adjusts the model's weights against a large dataset and happens once per model version, at the provider's expense. Inference keeps the weights fixed and runs them on your input. You never pay for training directly. You pay for inference per token, or through a subscription that meters it.

### What is an inference API?

An endpoint that accepts a prompt, runs a hosted model on it and returns the output, billed by the tokens in and out. OpenAI, Anthropic and Google run their own. Gateways and hosts such as OpenRouter, Together and Groq serve open-weight models the same way, at their own prices and speeds.

### Why does inference cost money?

Every request occupies accelerator memory and compute for as long as the model reads the prompt and writes the reply. Output tokens cost more than input tokens because each one is a separate forward pass through the model, while the prompt is processed largely in parallel.