← Back to articles Blog

LLM Optimization: 8 Techniques to Cut Costs Without Cutting Quality

LLM costs can spiral fast. A feature that costs $200/month in development hits $20,000/month at production scale. LLM optimization is the practice of reducing that cost without sacrificing output quality. Here are the techniques that actually move the needle, ranked by ROI.

1. Semantic caching (highest ROI)

Cache responses for semantically similar prompts, not just identical ones. If 30% of your users ask variations of the same question, caching eliminates 30% of your API calls entirely.

Implementation: Embed each prompt, compare against cached embeddings at 85%+ cosine similarity, return cached response on hit.

Typical savings: 20-40% cost reduction

2. Prompt compression

Automatically rewrite bloated prompts to their minimum effective form. The average production prompt is 40-70% waste — redundant instructions, verbose preamble, unnecessary context.

Implementation: Score each prompt for waste, rewrite with a cheap model (Claude Haiku costs $0.00025/1K tokens) when waste exceeds 30%.

Typical savings: 30-60% on input tokens

3. Right-size your model

Most tasks don't need frontier models. GPT-4o-mini handles classification, summarization, and simple Q&A as well as GPT-4o at 1/30th the cost.

Rule of thumb: Start with the cheapest model. Only upgrade when quality is measurably insufficient.

Typical savings: 10-30x cost reduction for suitable tasks

4. Constrain output length

RLHF-trained models are verbose by default. Explicitly constraining output length via max_tokens and system prompt instructions reduces output token spend significantly.

// Add to every system prompt
"Answer concisely in 1-3 sentences unless more detail is explicitly requested."

// And in API params
{ max_tokens: 300 }

Typical savings: 20-40% on output tokens

5. Trim conversation history

Sending the full chat history with every message causes context to grow quadratically. Implement a rolling window or summarize older turns.

// Keep last N turns instead of full history
const messages = conversationHistory.slice(-6) // last 3 exchanges

Typical savings: 40-80% on multi-turn conversations

6. Batch requests

OpenAI's Batch API processes requests asynchronously at 50% discount. For non-real-time workloads (document processing, classification pipelines), batching is the easiest cost cut available.

// OpenAI Batch API
const batch = await openai.batches.create({
  input_file_id: fileId,
  endpoint: "/v1/chat/completions",
  completion_window: "24h"
})

Typical savings: 50% (OpenAI), 50% (Anthropic Message Batches)

7. Quantization for self-hosted models

Quantization reduces model precision (from FP16 to INT4/INT8), shrinking model size by 2-4x with minimal quality loss. A 70B model quantized to INT4 runs on hardware that wouldn't fit the full-precision version.

ollama pull llama3.3:70b-instruct-q4_K_M  # INT4 quantized

Typical savings: 50-75% hardware cost for equivalent throughput

8. Track cost per endpoint

You can't optimize what you don't measure. Most teams discover that 80% of costs come from 20% of features — often features no one notices or cares about.

Implementation: Tag every LLM call with the feature/endpoint it came from. Sort by cost descending. Optimize the top 3.

The most expensive LLM systems are the ones with no visibility. You can cut 40-70% of costs just by measuring where the money goes.