Generative AI

Retrieval-Augmented Generation (RAG)

See how injecting retrieved knowledge into a prompt grounds a model’s generation, saving it from hallucinating.

How a model augments its parametric memory with an external knowledge base by embedding a query, matching it against document embeddings, and injecting the nearest context into its prompt before generation.

Query
Document chunk
Retrieved

User Query

The user query is mapped into the same vector space as the document chunks.

Challenge

Solved

Adjust the threshold to retrieve only the most relevant document for a query about "Paris".

Type "capital of france" and raise the threshold until exactly one document is retrieved.

A higher threshold increases precision but lowers recall. If it is too high, you might retrieve nothing!

Check your understanding

2 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.

Large Language Models (LLMs) are trained on massive amounts of text, but their parametric memory is static. Once training finishes, they cannot learn new facts without being retrained or fine-tuned. Furthermore, if you ask an LLM for specific or obscure information, it often "hallucinates" a plausible-sounding but incorrect answer because it lacks the actual knowledge.

Retrieval-Augmented Generation (RAG) solves this by separating memory from reasoning. Instead of forcing the model to memorize every fact, RAG provides the model with a search engine.

How RAG works

Embed the query: When you ask a question, it is first converted into a dense vector (an embedding).
Vector search: We search a vector database containing document chunks (also embedded) to find the nearest neighbors to your query in the vector space. High cosine similarity means the text is semantically related.
Context injection: The most relevant document chunks are retrieved and prepended to your original query, forming an "augmented prompt".
Generation: The LLM reads the context and your query, and generates an answer strictly based on the provided text.

By explicitly providing the context, the LLM acts more like a reading comprehension engine than a memorization engine.

Experiment

Try typing different queries in the interactive lab. Watch how the query's location in vector space changes, pulling in different nearest neighbors. If you set the similarity threshold too high, the system will fail to retrieve anything, leaving the model with no context to answer the question!

Reference

Cosine Similarity: (A · B) / (||A|| ||B||)
Augmented Prompt: Context + User Query

Break it on purpose

When the similarity threshold is too high, no documents are retrieved, forcing the model to fallback or hallucinate.