Engineering Atlas › Building AI Systems

Retrieval-Augmented Generation

11 min read · last reviewed 2026-10-17

Open in the interactive app: practice, quiz, mock interview →

In one minute

How it works

Why RAG

Models know what was in their training data, up to a cutoff, and nothing about your company's documents, tickets or policies. Putting everything into the prompt does not scale and dilutes attention. RAG retrieves a small, relevant subset at query time, so answers can use current, private data and cite their sources. The 2020 paper that named the technique (Lewis et al.) combined a pre-trained neural retriever over a dense vector index of Wikipedia (the non-parametric memory) with a seq2seq generator (the parametric memory) (arXiv:2005.11401). Today the generator is usually a general-purpose LLM behind an API. A fixed pipeline like the one below always searches once; when a question needs several searches or a choice of index, newer designs let an agent call retrieval as a tool (Azure Architecture Center).

Indexing

  1. Load and clean. Extract text from PDFs, HTML, tickets and wikis; strip navigation, boilerplate and duplicates.
  2. Chunk. Split into passages of a few hundred tokens, usually with some overlap so sentences that span boundaries are not lost. Split on structure where possible (headings, paragraphs), and keep metadata: source, title, section, date, access permissions.
  3. Enrich. Optionally prepend context to each chunk, such as the document title and a one-line summary of where it sits. Anthropic's "contextual retrieval" write-up measured the top-20 retrieval failure rate falling from 5.7 percent to 3.7 percent with contextual embeddings, to 2.9 percent when contextual BM25 was added, and to 1.9 percent with reranking, which are reductions of 35, 49 and 67 percent (Anthropic). Those are one vendor's numbers on its own datasets, so re-measure on yours. Microsoft's guidance likewise treats chunk enrichment (title, summary, keywords) as its own phase to evaluate (Azure Architecture Center).
  4. Index. Embed each chunk into a vector and store it in a vector index, and also index the text for keyword search.

Querying

  1. Understand the question. Rewrite follow-ups into standalone queries ("what about for annual plans?" becomes "refund policy for annual plans"), and extract filters (product, region, date).
  2. Retrieve candidates. Run keyword search and vector search, each returning, say, the top 50, with metadata filters including the user's permissions.
  3. Fuse and rerank. Combine the lists, for example with reciprocal rank fusion, then optionally rerank the top candidates with a cross-encoder or a small model that scores query and passage together.
  4. Assemble. Put the top 5 to 10 passages into the prompt with their sources, ordered sensibly, within a token budget.
  5. Generate with citations. Instruct the model to answer only from the passages, cite them, and say when the answer is not there.

Hybrid search

Vector search finds passages with similar meaning even when words differ ("cancel my subscription" matches "terminate your plan"). Keyword search with BM25 finds exact terms vector search can miss: product codes, error messages, names, rare words. Reciprocal rank fusion merges ranked lists by summing 1 / (k + rank) for each document across lists, which needs no score calibration between systems. The original paper fixed k at 60 after a pilot and found it near-optimal, and RRF equalled or beat Condorcet Fuse and CombMNZ (Cormack et al., SIGIR 2009). Elasticsearch ships RRF as a retriever and defaults rank_constant to 60 (Elastic docs).

Evaluating RAG

Evaluate the two halves separately; Azure's RAG guide says to evaluate each step independently while remembering that users experience the end result, and it scores generation on groundedness, completeness, utilization and relevancy (Azure Architecture Center). Huyen covers retrieval algorithms and retrieval optimisation in the RAG chapter of AI Engineering (ToC). Retrieval: for a set of questions with known relevant passages, measure recall at k (did the right passage appear in the top k?) and mean reciprocal rank. Generation: given the right passages, is the answer correct, complete and faithful to them? A drop in answer quality with stable retrieval points at the prompt or model; a drop in retrieval points at chunking, embeddings or the index.

The same shapes as any system

RAG is the finding-things-fast shape (indexes, ranking) plus multi-step consistency (the index is a derived copy that must stay in sync with sources) plus trust but verify (citations and faithfulness checks).

Down the stack

Layer 1The contract

A RAG feature promises that answers are grounded in permitted, current sources, cite them, and say "I don't know" when the sources do not cover the question. Operationally: index freshness within a stated delay (for example, document edits searchable within 10 minutes), retrieval recall at 10 above a target on the eval set, and no passage returned to a user who lacks access to its source.

Layer 2The mechanism

Keyword search uses an inverted index: for each term, a posting list of documents containing it, scored with BM25, which rewards terms that are frequent in the passage but rare in the collection and normalises for passage length. Vector search uses an approximate nearest neighbour index such as HNSW (a layered graph you walk greedily toward the query vector) or IVF (clusters you probe), trading a little recall for large speedups. Metadata filters, including permission filters, are applied during or before the search so restricted chunks are never returned. Rerankers score the top candidates with a model that reads query and passage together, which is more accurate but too slow to apply to the whole corpus.

Layer 3The machine

An embedding is a vector of a few hundred to a few thousand floating-point numbers. A million chunks at 1,024 dimensions in 32-bit floats is about 4 GB of raw vectors, before index overhead, which is why vector indexes often quantise vectors to 8-bit or smaller codes: int8 is exactly 4 times smaller and binary 32 times, and in one benchmark a 41-million-text index needed about 5.2 GB in binary form against roughly 200 GB in float32 (Hugging Face). Each similarity is a dot product, a few hundred to a few thousand multiply-adds that CPUs vectorise with SIMD instructions; HNSW keeps the number of comparisons per query in the low thousands instead of millions, so search takes milliseconds. Inverted index lookups are dominated by reading and intersecting compressed posting lists from memory or disk.

Why it's built this way

  1. Models lack private and current knowledge, and long prompts dilute attention.
  2. Retrieving a small relevant subset solves both, if retrieval is good.
  3. Meaning-based and exact-term matching fail differently, so hybrid search covers both.
  4. Fast first-stage retrieval plus slower reranking balances recall, precision and latency.
  5. The index is derived data, so freshness, permissions and evaluation are ongoing engineering work.

Diagram

flowchart LR
  subgraph Index
    D["Documents"] --> CH["Chunk and enrich"]
    CH --> EM["Embed"]
    CH --> KW["Keyword index (BM25)"]
    EM --> VI["Vector index (HNSW)"]
  end
  Q["Question"] --> RW["Rewrite and filters"]
  RW --> KW
  RW --> VI
  KW --> F["Fuse (RRF)"]
  VI --> F
  F --> RR["Rerank top 50"]
  RR --> P["Prompt with top passages"]
  P --> G["Answer with citations"]

Worked example

A SaaS company answers support questions from 12,000 help articles and 400,000 resolved tickets. The first version embeds whole articles, retrieves the top 3 by vector similarity, and answers. On a 300-question eval, answer accuracy is 64 percent. Looking at failures, 70 percent of wrong answers had no relevant passage in the context.

Changes, measured one at a time on retrieval recall at 10: chunking articles by section at about 400 tokens with 50 tokens of overlap (recall 58 to 71 percent); prepending article title and section path to each chunk (71 to 78); adding BM25 keyword search fused with RRF, which fixed questions containing error codes like E-4012 (78 to 86); a reranker over the top 50 (86 to 91). Answer accuracy rises to 85 percent with the same model and prompt. Then permission filters are added so internal-only articles are never retrieved for customers, and an indexing pipeline listens for article edits so changes are searchable within 5 minutes.

In the wild

Trade-offs

ChoiceStrengthsWeaknessesChoose when
Vector search onlyHandles paraphraseMisses exact codes and namesConversational text without identifiers
Keyword (BM25) onlyExact matches, cheap, explainableMisses paraphraseLogs, code, IDs, small corpora
Hybrid with RRFRobust across query typesTwo indexes to runMost production RAG
RerankerBetter precision at the topAdded latency and costWhen top-k quality matters
Long context instead of RAGSimpleCost, latency, distraction, size limitsSmall, fixed document sets

Pitfalls in production

Leadership lens

Check yourself

  1. Why measure retrieval recall separately from answer accuracy?

    Show answer

    If the right passage is not retrieved, no prompt or model can answer correctly; separating them shows whether to fix retrieval or generation.

  2. Why combine keyword and vector search?

    Show answer

    Vector search handles paraphrase but misses exact identifiers and rare terms; keyword search does the opposite; fusing them covers both kinds of queries.

  3. How does reciprocal rank fusion combine two ranked lists?

    Show answer

    Each document gets the sum of 1/(k + rank) over the lists it appears in, so documents ranked high in either or both lists rise, without needing comparable scores.

  4. Where must permission filtering happen in RAG?

    Show answer

    At retrieval time, using the requesting user's access, so restricted passages never enter the prompt.

Sources and further study

Primary

Systems docs

Books

Other curricula

Free resources

Open in the interactive app: practice, quiz, mock interview →