Gen AI

Retrieval-Augmented Generation (RAG)

Giving a language model an open book: before it answers, it looks up relevant facts and reads them into its prompt, so replies stay grounded and current.

A question feeds a retriever that pulls relevant chunks from a document store into the prompt, which the language model uses to produce a grounded answer

Why Does This Exist?

A language model only knows what was in its training data, frozen at some cutoff date. Ask about your company's internal docs, last week's news, or a niche API, and it will either admit ignorance or, worse, confidently make something up. That confident fabrication is called a hallucination, and it's the single biggest obstacle to trusting these models in real products.

Retrieval-augmented generation is the fix that won. Instead of retraining the model every time facts change, you keep a searchable knowledge base on the side. When a question comes in, you first retrieve the most relevant snippets, then paste them into the prompt and ask the model to answer using them. The model's language skill stays; the facts come from a source you control and can update any time.

Think of It Like This

Open-book exam vs. closed-book

A closed-book exam tests what you memorized, and under pressure you'll bluff the answers you half-remember. An open-book exam is different: you still need to understand the material to write a good answer, but you look up the exact figures instead of guessing. RAG turns the model's closed-book exam into an open-book one. It hands the model the right pages first, so it reasons over real text instead of hazy memory.

How It Actually Works

RAG has two phases. The first happens ahead of time, the second at query time.

Index (offline). Split your documents into chunks, convert each chunk into an embedding vector, and store those vectors in a vector database built for fast similarity search.
Retrieve (at query time). Embed the user's question the same way, then find the chunks whose vectors are closest to it. Closeness in embedding space means "about the same thing", so this is search by meaning, not keyword matching.
Augment. Stitch the top few chunks into the prompt, usually with an instruction like "answer using only the context below."
Generate. The transformer reads the question plus the retrieved context and writes an answer grounded in it, ideally citing which chunk it used.

A RAG request, end to end

Step 1 of 4

Embed the question

Turn the user's query into a vector using the same embedding model as the documents.

Show all steps

Embed the question: Turn the user's query into a vector using the same embedding model as the documents.
Retrieve top-k: Search the vector store for the handful of chunks closest in meaning to the query.
Build the prompt: Insert those chunks into the prompt as context alongside the original question.
Generate grounded: The model answers from the supplied context, so the reply reflects your data, not stale memory.

Show Me the Code

1import numpy as np2
3
4def retrieve(query: np.ndarray, chunks: np.ndarray, k: int = 2) -> np.ndarray:5    # Score every chunk by dot-product similarity to the query, take the top k.6    scores = chunks @ query7    return np.argsort(scores)[::-1][:k]8
9
10query = np.array([0.2, 0.9, 0.1])11chunks = np.array([[0.1, 0.8, 0.0], [0.9, 0.1, 0.2], [0.2, 0.85, 0.15]])12print(retrieve(query, chunks))  # -> [2 0], the two most relevant chunks

Swap the toy vectors for real embeddings and a vector index, and this is the heart of every RAG system: rank by similarity, keep the best few.

Watch Out For

Garbage retrieval means garbage answers

RAG is only as good as what it retrieves. If chunking splits a table mid-row, or the search returns off-topic passages, the model faithfully answers from bad context. Most RAG failures are retrieval failures, so tune your chunk size and search quality before blaming the model.

Context still gets stuffed into a finite window

Retrieved chunks compete for the same limited token budget as everything else in the prompt. Cram in too many and you blow the context window or bury the useful passage among distractions. Retrieve few, retrieve well.

The Quick Version

RAG lets a model answer from an external, updatable knowledge base instead of frozen training data.
Documents are chunked, embedded, and stored; a query retrieves the closest chunks by meaning.
The chunks are added to the prompt so the model generates a grounded answer.
It's the standard cure for hallucination and stale knowledge, with no retraining needed.
Retrieval quality and context-window limits are where RAG systems live or die.