Building a Production RAG System, End to End

Every stage of a real retrieval-augmented-generation pipeline - chunking, retrieval, re-ranking, evaluation, observability, caching - and the trade-offs that actually matter.
Author

Benedict Thekkel

Published

March 15, 2026

Retrieval-Augmented Generation is easy to demo and hard to ship. “Embed some docs, stuff the top-k into a prompt” gets you a toy. A production RAG system is a pipeline of ~14 stages, each with its own failure modes, metrics, and cost.

This post walks the whole pipeline the way I built it out across the DeepLearning repo’s LLM/RAG series, then points at the agentic / graph / multimodal extensions that came later.

The pipeline at a glance

flowchart LR
  D[Documents] --> P[Preprocess]
  P --> C[Chunk]
  C --> E[Embed]
  E --> I[(Vector Index)]
  Q[Query] --> QT[Query transform]
  QT --> R[Retrieve]
  I --> R
  R --> RR[Re-rank]
  RR --> CA[Context assembly]
  CA --> G[Generate]
  G --> A[Answer]
  A --> EV[Evaluate]
  EV -. feedback .-> QT
  G --> OB[Observability]
  R --> CH[(Cache)]

Each stage is a notebook. The table below is the map; the rest of the post drills into the decisions that move quality and cost the most.

# Stage What it decides Notebook
01 Overview Architecture + when RAG beats fine-tuning RAG
02 Chunking Recall ceiling and context waste Chunking
03 Embedding Semantic match quality, index cost Embedding
04 Indexing Latency vs recall (HNSW/IVF) Indexing
05 Retrieval Dense vs sparse vs hybrid Retrieval
06 Re-ranking Precision of the final top-k Re-ranking
07 Query transformation Recall on hard / vague queries Query transform
08 Context assembly Token budget, ordering, dedup Context assembly
09 Generation Grounding, citations, refusal Generation
10 Evaluation Whether any change helped Evaluation
11 Observability Debugging prod regressions Observability
12 Index lifecycle Freshness, re-embedding, drift Index lifecycle
13 Access control Per-user / per-tenant filtering Access control
14 Caching Cost and p50 latency Caching

Chunking: the cheapest lever with the biggest ceiling

Chunking sets the ceiling on retrieval quality: if the answer is split across two chunks and you only fetch one, no re-ranker saves you.

Strategy How Pros Cons
Fixed-size N tokens, hard cut Trivial, predictable Splits mid-idea
Recursive Split on separators, then size Respects structure Tuning per format
Semantic Break on embedding-distance shifts Coherent chunks Compute at ingest
Document-aware Headings / tables / code blocks Best for structured docs Format-specific parsers

See 02_Chunking. In practice: recursive as the default, document-aware where the corpus is structured (docs, wikis, code).

Retrieval: dense, sparse, or hybrid

Dense (embeddings) catches paraphrase; sparse (BM25) nails exact terms, codes, IDs. Hybrid + re-rank is the production default. The chart shows the pattern I keep seeing: hybrid lifts recall, and a cross-encoder re-rank lifts precision the most.

Method Strength Weakness
Dense (kNN) Semantic / paraphrase Misses rare exact tokens
Sparse (BM25) Exact terms, codes No semantics
Hybrid (RRF) Best recall Two indexes to run
Hybrid + re-rank Best precision@k Extra latency/cost

Details in 05_Retrieval and 06_Re-ranking.

Where the latency and the money go

Two charts I wish I’d drawn on day one: p50 latency by stage, and cost per 1k queries. Retrieval is rarely the bottleneck - generation and re-ranking are. Caching (stage 14) is the single biggest cost lever. See 11_Observability and 14_Caching.

Evaluation: you cannot improve what you do not score

RAG has two halves to grade separately: retrieval (did we fetch the right context?) and generation (did we use it faithfully?). Grading only end-answers hides which half broke.

Metric Half Question it answers
Context precision Retrieval Are the fetched chunks relevant?
Context recall Retrieval Did we get all needed chunks?
Faithfulness Generation Is the answer grounded in context?
Answer relevance Generation Does it address the query?

Full harness in 10_Evaluation.

Beyond the basics

Once the core pipeline is solid, the LLM/RAG series extends into the patterns that matter for real workloads:

RAG is also just one stage in a larger system - see the companion post on LLM pipeline patterns.

Back to top