RAG Explained: How Retrieval-Augmented Generation Works (And What It Costs)

Large language models have a knowledge cutoff — they can only answer based on what they learned during training. Retrieval-Augmented Generation (RAG) fixes this by giving the model access to external documents at query time, without retraining or fine-tuning.

How RAG works

RAG has two phases:

1. Indexing (done once)

1. Split your documents into chunks (typically 200-500 tokens each)
2. Embed each chunk using an embedding model (text-embedding-3-small is the standard)
3. Store the embeddings in a vector database (Pinecone, Weaviate, pgvector, Chroma)

2. Retrieval + generation (at query time)

1. Embed the user's query
2. Find the most similar chunks in the vector database (cosine similarity)
3. Inject the top 3-5 chunks into the LLM prompt as context
4. The LLM generates a response grounded in the retrieved documents

// Simplified RAG pipeline
const queryEmbedding = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: userQuery
})

const relevantChunks = await vectorDB.search({
  vector: queryEmbedding.data[0].embedding,
  topK: 5
})

const context = relevantChunks.map(c => c.text).join('\n\n')

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: `Answer using only this context:\n${context}` },
    { role: 'user', content: userQuery }
  ]
})

RAG vs fine-tuning

The token cost problem with RAG

RAG adds tokens to every query. If you inject 5 chunks of 400 tokens each, that's 2,000 extra input tokens per call. At scale, this compounds fast:

• 100K queries/day × 2,000 extra tokens = 200M extra tokens/day
• At gpt-4o pricing ($5/1M input tokens) = $1,000/day in RAG overhead alone

How to keep RAG token costs under control

• Smaller chunks — 200 tokens vs 500 tokens per chunk reduces context injection by 60%
• Fewer chunks — top-3 instead of top-5 reduces context by 40%
• Reranking — use a cheap reranker model to filter retrieved chunks before sending to the LLM
• Semantic caching — cache RAG responses for similar queries so the retrieval + generation pipeline only runs once
• Cheaper models — RAG with gpt-4o-mini instead of gpt-4o costs 30x less with minimal quality loss for retrieval tasks

The most expensive RAG systems are the ones that retrieve too many chunks and send them to the most expensive model. Right-sizing both cuts costs 80% with no quality loss.