← Back to articles Blog

We analyzed thousands of LLM API calls. Here's where the waste is hiding.

When we started building Tokoscope, we had a hunch that most LLM applications were wasting tokens. What we didn't expect was how consistent the patterns would be — or how much money was being left on the table.

After analyzing thousands of real API calls across OpenAI, Anthropic, and Gemini, we found that 40–70% of tokens in the average production prompt are pure waste. Not because developers are careless. Because there's no visibility into what's actually happening.

The three biggest sources of waste

1. Bloated system prompts

System prompts are the most common source of token waste we see. Teams build them up over time — adding instructions, edge cases, and clarifications — without ever removing anything. The result is system prompts that repeat the same instruction three or four different ways, just to be safe.

Here's a real example of a bloated system prompt we compressed:

Please note that it is very important that you make sure to always
respond in a helpful and professional manner. As an AI assistant,
I want you to please make sure that you understand that you should
always be polite and respectful. Make sure to note that you should
answer questions clearly and concisely. It is important that you
make sure to stay on topic at all times.

That's 67 tokens. Here's the compressed version:

Be helpful, professional, and concise. Stay on topic.

8 tokens. Same behavior from the model. 88% reduction.

2. Missing semantic caching

Most production LLM apps send the same or very similar prompts repeatedly. A customer support bot gets asked "How do I reset my password?" dozens of times a day. A code assistant sees similar debugging requests constantly. Without caching, every one of those hits the API and costs tokens.

Exact match caching catches identical prompts. But what about prompts that mean the same thing worded differently? "How do I reset my password?" and "I forgot my password, what do I do?" should return the same cached response — but exact matching misses it completely.

Semantic caching, using embedding similarity, catches these near-duplicates and can reduce API calls by 30–40% in most production apps.

3. Stuffed context windows

The third pattern we see is developers defaulting to sending everything "just in case." RAG pipelines that return 20 chunks when 3 would suffice. Conversation histories that go back 50 turns when only the last 5 matter. Tool outputs that include full JSON payloads when only one field is needed.

The model processes all of it. You pay for all of it. Most of it doesn't change the output.

Why this happens

The core problem is visibility. When you can't see where your tokens are going, you can't optimize them. Most teams find out about token waste the same way — when the bill arrives.

"We had no idea our support bot was burning $4,000 a month on near-identical prompts until we started tracking it."

The good news: once you have visibility, the fixes are usually straightforward. Compress the system prompt. Add semantic caching. Trim the context window. Three changes, compounding savings.

What you can do today

1. Audit your system prompts. Read them out loud. If you're saying the same thing twice, cut one.
2. Add caching. At minimum, exact match. Ideally, semantic similarity at 85%+ threshold.
3. Track by endpoint. You can't optimize what you can't measure. Know which features cost the most.

Or drop in two lines of code and let Tokoscope handle all three automatically.