RAG Architecture

Why Does This Exist?

A language model's knowledge is frozen at whatever point its training data was collected, and it has no way to consult a specific company's internal documents, a customer's private records, or anything published after training finished. Fine-tuning a model on new information is expensive, slow, and — as the fine-tuning band covers directly — poorly suited to information that changes frequently, since every update requires retraining. Meanwhile, context windows are large but still finite, so simply pasting an entire knowledge base into every prompt doesn't scale past a certain corpus size, and it's wasteful even when it technically fits.

Retrieval-augmented generation (RAG) answers this by keeping the model's weights entirely unchanged and instead giving it access to relevant information at the moment it's needed, retrieved fresh from an external source and inserted directly into the prompt. The model isn't asked to remember everything; it's asked to read whatever's handed to it right now and answer based on that — the closed-book-versus-open-book distinction that runs through this entire band.

Think of It Like This

How It Actually Works

Two lanes: indexing once, retrieving per query

The diagram above splits the architecture into exactly the two lanes that matter. Indexing happens ahead of time, offline, whenever documents are added or updated: each document is split into chunks (per chunking strategies), each chunk is embedded, and the resulting vectors are stored in an index. This happens once per document, not once per query — indexing cost is amortized across every future query that might retrieve from it.

Retrieval happens fresh for every incoming query, at request time: the query itself gets embedded using the same model that embedded the documents, that query vector gets matched against the stored index to find the most relevant chunks, those chunks get inserted directly into the prompt alongside the original question (augmentation), and the model generates its answer conditioned on both the question and the retrieved material.

Why each stage's failure looks different

Every stage in this pipeline can fail independently, and each failure has a genuinely different signature. A document that was never properly parsed or indexed can never be retrieved, no matter how good the retrieval or generation stages are — the information simply isn't in the system. A document that's indexed correctly but retrieved poorly (the right chunk exists, but the search didn't surface it) produces an answer that's missing information the system technically has. And a document that's retrieved correctly but ignored or misused by the model during generation produces a wrong answer despite the system having done everything right up to that point. These are three distinct failure modes with three distinct fixes, and conflating them — treating every bad answer as a "prompt problem," say — is the single most common mistake in debugging a RAG system, covered directly on RAG failure diagnosis.

What "augmentation" actually does to the prompt

The augmentation step is mechanically simple: retrieved chunks get formatted into the prompt, typically with some structure indicating they're reference material rather than part of the conversation itself, and an instruction telling the model to base its answer on that material specifically. The model itself is never modified — no weights change, nothing is retrained. This is the same "everything happens in the prompt, nothing happens to the weights" property that in-context learning relies on, applied here to inject retrieved knowledge rather than task examples.

Show Me the Code

A minimal, end-to-end RAG pass, with the indexing and retrieval lanes kept explicit as separate functions.

1import numpy as np2
3
4def index_documents(docs: list[str], embed_fn) -> tuple[list[str], np.ndarray]:5    """Runs ONCE, offline, whenever documents change."""6    chunks = docs                                    # assume pre-chunked for this minimal example7    vectors = np.array([embed_fn(c) for c in chunks])8    return chunks, vectors9
10
11def retrieve(query: str, chunks: list[str], vectors: np.ndarray, embed_fn, k: int = 2) -> list[str]:12    """Runs PER QUERY, at request time."""13    q_vec = embed_fn(query)14    sims = vectors @ q_vec / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q_vec))15    top_k = np.argsort(-sims)[:k]16    return [chunks[i] for i in top_k]17
18
19def toy_embed(text: str) -> np.ndarray:20    return np.random.default_rng(abs(hash(text)) % (2**32)).normal(size=16)21
22
23docs = ["Refund policy: 30 days.", "Shipping takes 3-5 business days.", "Support hours are 9-5 ET."]24chunks, vectors = index_documents(docs, toy_embed)                # indexing lane, once25retrieved = retrieve("how long until my order ships", chunks, vectors, toy_embed, k=1)26augmented_prompt = f"Context: {retrieved}\n\nQuestion: how long until my order ships"27print(augmented_prompt)

index_documents runs once regardless of how many queries follow; retrieve runs fresh every single time a question arrives — that separation is the entire architectural shape this page describes.

Watch Out For

The Quick Version

• RAG keeps the model's weights unchanged and instead retrieves relevant information at query time, inserting it directly into the prompt.
• Indexing happens once, offline, whenever documents change; retrieval happens fresh for every incoming query.
• The pipeline has multiple independent stages — indexing, retrieval, augmentation, generation — and each can fail with a distinct signature.
• Augmentation is purely a prompt-construction step; nothing about the model itself changes.
• Conflating retrieval failures with generation failures is the most common mistake in diagnosing a RAG system that returns wrong answers.

What to Read Next

• Chunking Strategies is the first real preprocessing decision inside the indexing lane this page describes.
• Reranking is a common refinement stage inserted between retrieval and generation.
• RAG Evaluation separates retrieval quality from generation quality, the distinction this page's failure-mode section leans on.
• RAG Failure Diagnosis is the ordered process for finding which of this page's stages actually caused a bad answer.