← Back to articles Blog

LLM Architecture Explained: Transformers, Attention, and What Happens When You Send a Prompt

You don't need to understand LLM architecture to use LLMs. But understanding it explains a lot about why they cost what they cost, why they have context limits, why they're slow at generating long outputs, and why some prompts work better than others.

The transformer: the architecture behind every major LLM

Virtually every major LLM today — GPT-4, Claude, Gemini, Llama, Mistral — is based on the transformer architecture, introduced in the 2017 paper "Attention Is All You Need." The key innovation was the attention mechanism, which lets the model look at all tokens in the context simultaneously rather than processing them sequentially.

How a prompt becomes a response

Step 1 — Tokenization

Your prompt is split into tokens — roughly 3-4 characters each for English text. "Hello world" becomes ["Hello", " world"]. The model never sees raw text, only token IDs.

This is why you're billed per token, not per character or word. See our guide to where token waste hides for how this affects costs.

Step 2 — Embedding

Each token ID is converted to a high-dimensional vector (typically 4,096-16,384 dimensions). These vectors encode semantic meaning — similar words have similar vectors.

Step 3 — Transformer layers

The embedded tokens pass through N transformer layers (GPT-4 has ~96 layers). Each layer has two components:

• Multi-head attention — lets every token attend to every other token in the context, learning relationships
• Feed-forward network — processes each token's representation independently

Step 4 — Output projection

The final layer outputs a probability distribution over all tokens in the vocabulary (~50,000-100,000 tokens). The model samples from this distribution to pick the next token.

Step 5 — Autoregressive decoding

The model generates one token at a time. Each new token is appended to the context, and the model runs another forward pass to generate the next one. This is why output tokens cost more than input tokens — they're generated sequentially, not in parallel.

Why context length is expensive

Attention is the bottleneck. Every token must attend to every other token — making attention complexity O(n²) with context length. Doubling the context window quadruples the attention compute. This is why:

• Long-context models (128K, 1M tokens) cost significantly more per token
• KV cache memory grows with context length
• Latency increases with longer inputs

Key architectural variants

Architecture	Examples	Key difference
Dense transformer	GPT-4, Claude, Llama 3	All parameters active for every token
Mixture of Experts (MoE)	Mixtral, DeepSeek, GPT-4 (rumored)	Only a subset of parameters active per token — more efficient
State space models	Mamba, RWKV	Linear complexity vs quadratic attention — better for very long sequences
Hybrid	Jamba, Zamba	Combines transformer attention with SSM layers

What architecture means for your costs

MoE models like DeepSeek get their cost efficiency from this architecture — they have many parameters but only activate a fraction per token, making inference cheaper than equally capable dense models. This is why DeepSeek can be 10-18x cheaper than GPT-4o while achieving similar benchmark scores.

The transformer's attention mechanism is both its greatest strength (it can relate any token to any other) and its greatest cost driver (attention scales quadratically with sequence length).