Retrieval-Augmented Generation (RAG) System
AdvancedOverview
A Retrieval-Augmented Generation (RAG) system grounds LLM outputs in an external knowledge base. Instead of relying purely on the model's parametric memory, the system fetches relevant documents in real-time and injects them into the prompt. This prevents hallucinations, allows the model to answer questions about proprietary or private data, and removes the need for frequent fine-tuning.
Functional Requirements
- Ingest and index large collections of documents (PDFs, HTML, text).
- Users can query the system in natural language.
- Retrieve the top-K most relevant document chunks based on semantic similarity.
- Generate an accurate, source-cited response using an LLM.
- Update the knowledge base without retraining the model.
Non-Functional Requirements
- Retrieval latency should be < 150ms.
- End-to-end latency (including LLM generation) should be acceptable for conversational UX (< 1-2s for first token).
- High accuracy/recall in retrieval to prevent garbage-in, garbage-out.
- Scalable vector indexing to handle hundreds of millions of chunks.
Capacity Estimation
Assume an enterprise knowledge base of 10 Million documents, averaging 5 chunks per document (50M chunks total).
- Storage: Each chunk text ≈ 1KB. Embeddings (e.g., 1536 dimensions float32) ≈ 6KB per vector. 50M * 7KB ≈ 350 GB of vector DB storage.
- Indexing Throughput: If 100k documents are updated daily, the system must compute 500k embeddings/day via an Embedding Model API (≈ 6 embeddings/sec, easily handled).
- Query QPS: At 100 queries/sec, the Vector DB must perform Approximate Nearest Neighbor (ANN) search across 50M vectors in < 50ms, requiring mostly in-memory index serving.
High-Level Architecture
The architecture is split into an Offline Ingestion pipeline and an Online Retrieval pipeline. Offline: Documents are parsed, split into smaller chunks, embedded into high-dimensional vectors by an Embedding Model, and stored in a Vector Database. Online: A user's query is routed to the Orchestrator (e.g., LangChain/LlamaIndex), which embeds the query and searches the Vector DB for the top-K chunks. A Re-ranker optionally scores these chunks for exact relevance. The Orchestrator constructs a prompt containing the chunks and the user query, sending it to the LLM for generation.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| document_metadata | doc_id (PK), source_url, title, author, created_at, access_level | Relational DB (PostgreSQL) |
| vector_chunk | chunk_id (PK), doc_id (FK), chunk_text, embedding_vector (e.g., vector(1536)) | Vector DB (Pinecone, Milvus, pgvector) |
Detailed Design
Chunking Strategy
Splitting documents optimally is crucial. If chunks are too small, they lose context; if too large, they dilute semantic similarity and consume too much of the LLM context window. Strategies include fixed-size windowing with overlap, sentence-boundary splitting, and semantic/hierarchical chunking (embedding the title alongside the paragraph).
Approximate Nearest Neighbor (ANN) Search
Exact KNN search is O(N) and too slow for 50M vectors. The Vector Database uses ANN algorithms, primarily HNSW (Hierarchical Navigable Small World) graphs. HNSW builds a multi-layered graph where the top layer has long connections (like a highway) and bottom layers have local connections. It provides sub-millisecond search at the cost of high memory usage (the graph must reside in RAM).
Hybrid Search & Re-ranking
Pure vector search struggles with exact keyword matching (e.g., "Error Code 404"). Modern RAG uses Hybrid Search: combining Dense Vector Search (semantic) with Sparse Keyword Search (BM25) using a weighting formula (Reciprocal Rank Fusion). Because retrieval often returns partially relevant chunks, a Cross-Encoder Re-ranker model (like Cohere Rerank) scores the top 50 chunks against the query and returns the absolute best 5 to the LLM.
Prompt Assembly and LLM Generation
The Orchestrator injects the retrieved text into a system prompt template: "Answer the query using ONLY the following context. If you don't know, say so. [Context Blocks]". The LLM then generates the answer, ideally citing the source doc_id provided in the context blocks.
Bottlenecks & Solutions
A major bottleneck is Vector DB Memory Requirements. HNSW indexes are RAM-intensive. Solutions include using Product Quantization (PQ) or Scalar Quantization to compress vectors (e.g., float32 to int8) at a slight cost to recall, or tiering data (keeping the HNSW graph in RAM but offloading the actual vectors to SSD via memory mapping like DiskANN).
Interview Follow-up Questions
Q: How do you handle document access control (permissions) during retrieval?
allowed_roles: ['engineering', 'admin']) to each chunk's metadata. At query time, the Vector DB executes a pre-filter or post-filter step to only search/return vectors that the user has permission to view, ensuring the LLM never sees unauthorized data.