LLM Inference Explained: How Models Generate Text (And What It Costs)

Training an LLM is a one-time compute event. Inference — running the trained model to generate responses — is the ongoing cost that scales with every user, every query, every token. It's where most of the money goes in production AI systems.

What is LLM inference?

Inference is the process of passing input tokens through a trained model's neural network layers to generate output tokens. Each forward pass through the model produces one token at a time — that's why LLMs generate text sequentially rather than all at once.

The key parameters that drive inference cost:

• Input tokens — everything in your prompt, system message, and context window
• Output tokens — every token the model generates in response
• Model size — larger models (more parameters) require more compute per token
• Context length — longer contexts require more memory and compute (quadratic attention cost)

Why output tokens cost more than input tokens

Most providers charge 3-5x more per output token than input token. The reason: input tokens can be processed in parallel (the whole prompt at once), but output tokens are generated sequentially — each token must be generated before the next one can start. This sequential decoding is more compute-intensive per token.

The three layers of inference cost

1. Prefill (input processing)

The model processes all input tokens in parallel. This is fast but memory-intensive — the entire key-value (KV) cache for the input must be held in GPU memory.

2. Decoding (output generation)

The model generates output tokens one at a time, each depending on all previous tokens. This is the slow, expensive phase — and it's why keeping outputs concise matters so much for cost.

3. KV cache

The key-value cache stores intermediate computations so the model doesn't reprocess the full context for every new token. It's the main bottleneck for long-context inference — a 128K context window can consume gigabytes of GPU memory just for the KV cache.

Why inference costs spike unexpectedly

• Verbose outputs — RLHF-trained models are biased toward longer responses. A model answering a 1-token question might output 50 tokens unprompted.
• Stuffed context windows — sending the full conversation history grows the input quadratically as conversations get longer
• Unnecessary tool calls — agentic workflows can trigger dozens of LLM calls per user task
• No caching — identical or near-identical prompts hitting the API every time instead of returning cached responses

How to reduce inference costs

1. Compress prompts — remove redundant instructions, filler phrases, and unnecessary context before sending
2. Cache aggressively — exact match for identical prompts, semantic caching for near-identical ones
3. Constrain output length — use max_tokens and explicit instructions like "answer in one sentence"
4. Right-size your model — gpt-4o-mini handles most tasks at 1/30th the cost of gpt-4o
5. Track per-endpoint usage — find which features consume the most tokens and optimize those first

The average production prompt is 40-70% waste tokens. Inference optimization starts with knowing which tokens are doing work and which aren't.