RAG Architecture

Why Does This Exist?

Your model's memory ends on the day its training data stopped. A handbook updated yesterday, a customer's private records, a price that changed this morning: none of it is in the weights, and retraining the model every time a fact changes is slow and expensive. Context windows keep growing, but pasting a whole knowledge base into every prompt still breaks down once the corpus is large, and you pay for those tokens on every single request.

Retrieval-augmented generation (RAG) keeps the weights frozen and changes what the model reads instead. For each question, the system fetches a few relevant passages from an external store and puts them in the prompt, so the model answers from material handed to it now rather than from memory of training. Closed book versus open book: that shift is the whole idea.

The one idea this page assumes is small. An embedding turns any passage into a list of numbers positioned so that closeness means relatedness, and cosine similarity scores that closeness from 0 to 1. If that sentence already clicks, you have everything you need; the full treatment with its own worked example is embeddings.

Scope, so you know where you are: this page is the pipeline itself. Slicing documents into passages lives in chunking strategies, measuring the system lives in RAG evaluation, and debugging it lives in RAG failure diagnosis.

Think of It Like This

How It Actually Works

The two lanes: index once, retrieve per query

Indexing runs offline, whenever documents change. Each document is split into chunks, short passages sized so one embedding holds one idea, every chunk is embedded into a vector, and those vectors sit in an index. Once per document, no matter how many questions arrive later.

Retrieval runs per question. The question is embedded with the same model, that query vector is scored against every stored chunk vector, the top few chunks are pasted into the prompt next to the question, and the model generates from the combined text. The weights never move. That property should feel familiar: it is the same trick in-context learning uses, pointed at retrieved knowledge instead of examples.

How the ranking is scored

Write the embedding model as $E$. It maps the question to a query vector $q = E(\text{question})$, where $q$ is just a list of numbers standing in for what you asked. It maps chunk $i$ to a chunk vector $d_i = E(\text{chunk}_i)$, the same kind of list standing in for that passage. Same $E$ on both sides, or the comparison is meaningless.

The score for chunk $i$ is the cosine of the angle between the two lists:

Here $q \cdot d_i$ is the dot product (multiply the lists element by element, then add), $\lVert q \rVert$ and $\lVert d_i \rVert$ are the lengths of the two lists (square each entry, add, take the square root), and $s_i$ is the resulting score: near 1 when the two point almost the same way, near 0 when they are unrelated. Keep the top $k$, where $k$ is simply how many chunks survive into the prompt. Small $k$ keeps the prompt cheap and focused; larger $k$ covers more ground at the cost of tokens and noise.

Worked example: three chunks, one shipping question

Three chunks sit in the store:

• C1: "Refunds are issued within 30 days." with $d_1 = [0.9, 0.1]$
• C2: "Orders ship in 3 to 5 business days." with $d_2 = [0.6, 0.8]$
• C3: "Support is open 9 to 5 on weekdays." with $d_3 = [0.1, 0.9]$

The question is "How long until my order ships?" and its embedding is $q = [0.8, 0.6]$. Its length is $\lVert q \rVert = \sqrt{0.8^2 + 0.6^2} = \sqrt{0.64 + 0.36} = \sqrt{1.0} = 1$, which keeps the arithmetic honest and short.

Score each chunk:

• C1: $q \cdot d_1 = 0.8 \times 0.9 + 0.6 \times 0.1 = 0.72 + 0.06 = 0.78$. Length $\lVert d_1 \rVert = \sqrt{0.81 + 0.01} = \sqrt{0.82} \approx 0.906$. So $s_1 \approx 0.78 / 0.906 \approx 0.86$.
• C2: $q \cdot d_2 = 0.8 \times 0.6 + 0.6 \times 0.8 = 0.48 + 0.48 = 0.96$. Length $\lVert d_2 \rVert = \sqrt{0.36 + 0.64} = 1$. So $s_2 = 0.96$.
• C3: $q \cdot d_3 = 0.8 \times 0.1 + 0.6 \times 0.9 = 0.08 + 0.54 = 0.62$. Length $\lVert d_3 \rVert = \sqrt{0.01 + 0.81} = \sqrt{0.82} \approx 0.906$. So $s_3 \approx 0.62 / 0.906 \approx 0.68$.

Ranking: C2 at 0.96, then C1 at 0.86, then C3 at 0.68. With $k = 2$ the prompt carries C2 and C1, and C3 stays in the store. The model answers "Orders ship in 3 to 5 business days [C2]." The bracket names the chunk the answer actually used, which is the thread you pull when an answer looks wrong.

What augmentation does to the prompt

Augmentation is prompt construction, nothing fancier. Instruction, then the surviving passages with their ids, then the question. No weights change, no retraining, nothing carries over to the next query. Keep the chunk ids visible in the prompt and ask the model to cite them, because those ids tell you exactly which evidence each answer rested on.

The price, and when to skip RAG

RAG buys freshness and grounding: new documents answer questions minutes after indexing, and answers can point at their sources. It pays in three currencies: latency from the extra lookup, tokens from stuffing passages into every prompt, and upkeep of the index itself, since everything must be re-embedded whenever the embedding model changes.

So skip it when the knowledge is stable and already in the model, or when the whole corpus fits in context and you can paste it directly (that tradeoff is long context versus RAG). Skip it too when the question needs math over the entire corpus, like counting or averaging across all documents: retrieval returns the most similar few passages, never a full table, so you can't build aggregates on them. The method assumes the answer sits inside a few passages and that similarity tracks relevance. Paraphrase gaps, multi-step questions, and contradictory documents break that assumption, which is where upgrades like hybrid search help.

Code

Two functions, two lanes. Indexing runs once; retrieval runs per question over the same three vectors from the worked example.

1import numpy as np2
3
4def cosine_sim(a: np.ndarray, b: np.ndarray) -> float:5    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))6
7
8chunks = {9    "C1": np.array([0.9, 0.1]),10    "C2": np.array([0.6, 0.8]),11    "C3": np.array([0.1, 0.9]),12}13query = np.array([0.8, 0.6])  # question: how long until my order ships14
15sims = {cid: round(cosine_sim(query, vec), 2) for cid, vec in chunks.items()}16print(sims)  # -> {'C1': 0.86, 'C2': 0.96, 'C3': 0.68}17
18ranked = sorted(sims, key=lambda c: sims[c], reverse=True)19print(ranked[:2])  # -> ['C2', 'C1']

Both # -> lines print what the worked example derived by hand: C2 first at 0.96, C1 second at 0.86, C3 last at 0.68, and the top two are C2 and C1.

Watch Out For

The Quick Version

• RAG freezes the weights and answers from passages fetched at request time, not from training memory.
• Indexing runs once per document. Retrieval, ranking, and generation run on every question.
• Cosine ranking decides what the model sees: 0.96 beats 0.68, and the loser never reaches the prompt.
• The prompt carries the winning passages plus the question, so the answer can cite the chunk it used.
• Retrieval caps generation. A missing or misranked chunk fails no matter how good the prompt is.
• Skip RAG for stable knowledge, tiny corpora that fit in context, or questions needing math over everything.

What to Read Next

• Chunking Strategies
• Embeddings
• RAG Evaluation
• RAG Failure Diagnosis
• Graph RAG