← Back to articles Blog

LiteLLM: The Open-Source LLM Proxy — When It Makes Sense

LiteLLM is an open-source Python library and proxy server that gives you a unified interface for 100+ LLM providers. Unlike OpenRouter which is a hosted service, LiteLLM is self-hosted — you run it on your own infrastructure.

What LiteLLM does

• Unified API — one interface for OpenAI, Anthropic, Gemini, Bedrock, Azure, Cohere, and more
• Provider fallbacks — if one provider fails, automatically retry with another
• Load balancing — distribute requests across multiple API keys or providers
• Spend tracking — per-model and per-user cost tracking via a SQLite or PostgreSQL database
• Rate limiting — set per-user or per-team token budgets
• Caching — basic exact-match caching via Redis

LiteLLM as a library vs proxy

LiteLLM has two modes:

Library mode (simplest)

import litellm

# Same interface for any provider
response = litellm.completion(
  model="gpt-4o",
  messages=[{"role": "user", "content": "Hello"}]
)

# Switch providers with one string change
response = litellm.completion(
  model="claude-sonnet-4-6",
  messages=[{"role": "user", "content": "Hello"}]
)

Proxy mode (full gateway)

# Start the proxy server
litellm --model gpt-4o --port 8000

# Now any OpenAI client points to localhost
client = OpenAI(base_url="http://localhost:8000")

LiteLLM vs OpenRouter

	LiteLLM	OpenRouter
Hosting	Self-hosted	Managed (their servers)
Data privacy	Full — stays on your infra	Passes through OpenRouter
Setup time	30-60 minutes	5 minutes
Markup	None	Small % on top of provider
Maintenance	You manage updates	Managed for you
Caching	Basic (Redis exact match)	None

The missing piece: semantic caching and compression

LiteLLM's built-in caching only does exact match — identical prompts hit the cache, everything else goes to the API. Semantic caching (catching near-identical prompts) requires additional tooling.

You can layer Tokoscope on top of LiteLLM's proxy since it exposes an OpenAI-compatible endpoint:

import { wrap } from 'tokoscope'
import OpenAI from 'openai'

// Point to your LiteLLM proxy
const client = wrap(new OpenAI({
  baseURL: 'http://localhost:8000',
  apiKey: 'your-litellm-key'
}), {
  apiKey: 'ts_live_...'
})

// Now you get LiteLLM routing + semantic caching + waste scoring

Who should use LiteLLM

LiteLLM makes sense if you have strict data privacy requirements (no data through third-party proxies), already run Kubernetes or Docker Compose, and have engineering bandwidth to maintain the proxy. For most early-stage teams, it's more infrastructure than the problem warrants.