← Back to articles Blog

Semantic caching: why exact match isn't enough

Emmanuel Ekunsumi · Jun 25, 2026 · 4 min read

Most teams who add LLM caching start with exact match — store the response for a prompt hash, return it on an identical request. It's simple, effective, and catches maybe 10–15% of your API calls.

But here's the problem: users don't ask the same question the same way twice.

A customer support bot sees hundreds of variations of the same question every day. Your code assistant sees dozens of similar debugging requests. Exact match caching misses all of them — and you keep paying for every single call.

The gap exact match leaves behind

❌ Exact match misses

  • "How do I reset my password?"
  • "I forgot my password, what do I do?"
  • "Can you help me change my password?"
  • "Password reset instructions?"
  • "How to recover account access?"

✓ Semantic cache catches all of these

  • Same meaning, different wording
  • Same intent, different phrasing
  • All map to the same cached response
  • Zero API calls after the first
  • Full token savings on every hit

In a real customer support app, these five questions should all return the same cached answer. Exact match serves one of them from cache. Semantic caching serves all five.

Two-Layer Cache "What is the capital of France?" "Name the French capital city." "French capital?" Cache Layer 1: Exact hash match Layer 2: Semantic ⚡ Cache hit saved 21 tokens Cache miss → calls LLM API → stores response Similarity threshold ≥ 85% → hit All three prompts return the same cached response

Two-layer cache: exact match first, then semantic similarity at 85%+ threshold

How semantic caching works

Instead of hashing the raw prompt text, semantic caching converts prompts into embeddings — high-dimensional vectors that represent the meaning of the text, not just the characters.

When a new prompt comes in, we generate its embedding and compare it against all cached embeddings using cosine similarity. If the similarity score is above a threshold (we use 85%), we return the cached response instead of calling the LLM.

1. New prompt arrives
↓
2. Check exact match cache → miss
↓
3. Generate embedding via OpenAI text-embedding-3-small
↓
4. Compare against cached embeddings (cosine similarity)
↓
5a. Score ≥ 85% → return cached response ⚡
↓
5b. Score < 85% → call LLM, store response + embedding

Real results

Here's actual output from Tokoscope when a semantic cache hit fires:

⚡ Tokoscope cache hit [semantic (89.3% match)] — saved 93 tokens ($0.000049)

The two prompts — "What is the population of Japan?" and "How many people live in Japan?" — scored 89.3% similarity. Different words, same meaning, same cached response.

In production apps with repetitive use cases, semantic caching typically increases cache hit rates from 10–15% (exact match only) to 35–50%.

The cost of generating embeddings

Generating an embedding costs tokens too. OpenAI's text-embedding-3-small is $0.00002 per 1K tokens — essentially free for most prompt lengths. A 100-token prompt costs $0.000002 to embed.

Compare that to the cost of a GPT-4o call on the same prompt: roughly $0.0005–$0.005 depending on output length. Embedding is 100–1000x cheaper than the call it might save.

Threshold tuning: 85% is a safe default. Lower it to 80% for more hits (some risk of returning slightly off-topic responses). Raise it to 92% for stricter matching (fewer hits, always accurate).

How to enable it

In Tokoscope, semantic caching is enabled automatically when you integrate the SDK. No configuration needed. Every prompt gets checked against both the exact match cache and the semantic cache before hitting the API.

import { wrap } from 'tokoscope'

// Semantic caching is on by default
const client = wrap(new OpenAI(), {
  apiKey: 'ts_live_...'
})

Cache entries are stored for 7 days. You can clear the cache anytime from your dashboard settings.

Add semantic caching in 2 lines

Tokoscope handles exact match and semantic caching automatically. Free to start.

Get started free →