LLM Stats: Token Usage Benchmarks and Cost Data for 2026

Emmanuel Ekunsumi · 5 min read · 2026-07-14

There's plenty of benchmark data on LLM quality, but very little on how LLMs are actually used in production. Here's what the data from real applications looks like — token usage patterns, cost distributions, and the metrics that matter for teams building with LLMs.

Average token counts by use case

Use case	Avg input tokens	Avg output tokens	Total per call
Simple Q&A / chatbot	150-400	100-300	250-700
Document summarization	2,000-8,000	200-500	2,200-8,500
Code generation	500-2,000	300-1,000	800-3,000
RAG pipeline	1,500-5,000	200-400	1,700-5,400
Agentic task (per step)	2,000-10,000	500-2,000	2,500-12,000
Email drafting	200-600	100-400	300-1,000

Where the token budget goes

Based on analysis across production LLM applications, the average token breakdown looks like this:

System prompt: 15-40% of input tokens — often the biggest single chunk
Conversation history: 20-50% in multi-turn apps — grows with each turn
User input: 5-20% — the actual content from the user
RAG context: 30-60% in RAG apps — retrieved document chunks

The insight: in most apps, the user's actual question is a small fraction of what gets sent to the API. The rest is overhead — and much of it is token waste.

Cache hit rates in production

Semantic caching hit rates vary dramatically by application type:

Application type	Typical cache hit rate	Why
FAQ / support bot	40-70%	Users ask the same questions repeatedly
General chatbot	10-25%	More varied queries, less repetition
Code assistant	5-15%	Code context is highly unique per user
Document Q&A	20-40%	Common questions about shared docs
Data extraction	2-8%	Every document is different

This is why semantic caching matters most for support and FAQ applications — and why it's less impactful for creative or code-heavy workloads.

Cost per 1,000 API calls by provider (2026)

Provider + Model	At 500 avg tokens/call	At 2,000 avg tokens/call
OpenAI gpt-4o	$2.50	$10.00
OpenAI gpt-4o-mini	$0.075	$0.30
Anthropic claude-sonnet	$1.50	$6.00
Anthropic claude-haiku	$0.125	$0.50
Google gemini-2.5-flash	$0.05	$0.20
Mistral large	$1.00	$4.00

The compounding effect of optimization

Teams that implement all three core optimizations — prompt compression, semantic caching, and model right-sizing — typically see:

30-60% reduction in input tokens from compression
20-50% reduction in API calls from caching
10-30x cost reduction from model right-sizing

Applied together on a $10,000/month API bill, teams commonly land at $1,500-3,000/month for equivalent output quality.

See your own LLM stats in real time

Tokoscope tracks token usage, cache hit rates, and cost per endpoint across all providers. Free to start.

Get started free →