Paper breakdown

Retrieval-Augmented Generation (RAG)

The 2020 paper that introduced a general-purpose fine-tuning recipe for combining pre-trained parametric and non-parametric memory.

Paper: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Authors: Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela · 2020

Read the paper

RAG combines a retriever that searches a document index with a generator that produces the final answer based on those documents.

The Problem

Large language models memorize an impressive amount of factual knowledge in their weights, but this "parametric" memory has severe limitations. It cannot easily be updated or expanded without retraining, it is prone to hallucination when asked for specific facts, and it cannot provide citations for its claims. Knowledge-intensive NLP tasks (like open-domain question answering) required a way to access external, up-to-date information.

The Idea

Instead of relying solely on the model's internal weights, the authors proposed a hybrid architecture: Retrieval-Augmented Generation (RAG). Given an input query, RAG first uses a dense neural retriever to search a massive external corpus (like Wikipedia) for relevant documents. It then feeds both the original query and the retrieved documents into a sequence-to-sequence generator (like BART) to produce the final answer. The entire system can be fine-tuned end-to-end.

How It Works

RAG consists of two main components:

The Retriever (DPR): A Dense Passage Retriever uses a pre-trained bi-encoder (based on BERT) to embed both the query and the documents into the same vector space. It calculates the similarity between the query vector and document vectors to retrieve the top-K most relevant passages.

The Generator (BART): A sequence-to-sequence model receives the input query concatenated with the retrieved passages. It attends to this combined context to generate the output token-by-token.

The authors proposed two variants:

RAG-Sequence: Uses the same retrieved documents to generate the entire sequence.
RAG-Token: Can draw on different retrieved documents to generate different tokens in the sequence.

Why It Mattered

RAG proved that LLMs do not need to memorize the entire internet. By decoupling knowledge storage (the document index) from language understanding and generation (the neural network), RAG systems can be updated instantly simply by swapping or updating the document index, without retraining the model. This drastically reduced hallucinations and allowed the model to provide provenance (citations) for its answers.

What Came After

RAG became the standard architectural pattern for building enterprise AI applications. While the original paper focused on fine-tuning the retriever and generator end-to-end, the industry largely shifted towards "zero-shot RAG," where a frozen LLM (like GPT-4) is prompted with context retrieved from a vector database. Innovations like advanced chunking, re-ranking (e.g., Cohere Rerank), and graph-based retrieval (GraphRAG) have since built upon this foundation.