RAG in Production
A demo answers one person. A production system answers hundreds at once without leaking private files or paying twice for the same question.
Why Does This Exist?
A demo RAG answers you. A production RAG answers a thousand yous at 9 a.m. on Monday, without showing salaries to interns, without paying for the same holiday question fifty times, and without going blind when it fails.
That gap is the whole page. A notebook prototype joins a vector store to a model and stops. Real traffic adds four jobs around that core: checking who may see each chunk, skipping repeat work, blocking bad input and output, and writing down what happened so you can fix it. You'll feel each one the first week you serve a team.
The one idea you need first: core RAG retrieves text chunks and then generates with them in context. If that loop is fuzzy, read RAG architecture first, because everything here assumes you know what top k retrieval means. Splitting docs is taught in chunking strategies, and measuring answer quality is taught in RAG evaluation. This page covers only the wrapper that makes that loop safe, fast, and visible.
Think of It Like This
A library with a locked archive and a photocopy shelf
You run a busy public library. Anyone may ask for anything, but back rooms need a badge. The local archive needs a member card. Staff files need a staff badge. The librarian checks your badge before fetching, not after reading the pages aloud.
The most asked forms live as photocopies by the desk. Holiday hours. Card applications. Ninety of your 200 morning visitors take a copy and leave. They never wait for a fetch, and you don't reprint the same page twice.
A bouncer turns away hecklers at the door, and a second checker skims each reply before it goes out. Every request leaves a slip: who asked, what was fetched, what was said. A small box by the exit collects thumbs up and thumbs down.
Where it stops: photocopies never change meaning, but cached answers go stale when source docs change. And a librarian says plainly when a book is missing, while a model will invent a smooth answer from nothing. You still need an explicit rule for saying you don't know.
How It Actually Works
A production request moves through six valves in order: input check, cache lookup, filtered retrieval, generation, output check, logging. You'll follow one query through them, then see what 200 do together.
Call the incoming text , the query text you receive. Its embedding , a list of numbers standing in for meaning, is what you compare. Call the similarity score , the cosine between and a stored chunk, a number from 0 to 1. Call the cutoff , the similarity threshold you demand before trusting a cached answer. Call , the count of chunks you pass to the model. Call , the hit rate, the fraction of queries served from cache.
The request path in order
- Input check screens for injection, toxicity, and off topic requests. A lightweight classifier or pattern list says yes or no before anything costly runs. See prompt injection for the attack this blocks.
- Cache lookup embeds and searches past queries. When the best match scores above , you return the stored answer. No retrieval. No generation.
- Filtered retrieval runs only on a miss. You search the vector store with a role filter attached, keep the top allowed chunks, and drop the rest before the model ever sees them.
- Generation writes the answer from plus those chunks, with citations to chunk ids so the trace stays honest.
- Output check screens the draft for leaked secrets, banned content, and broken format. Bad drafts get blocked or rewritten.
- Logging writes the triple of query, chunk ids, and answer, plus user feedback. That log is how you debug next week.
Secure retrieval: filter first, then rank
Metadata filtering means each stored chunk carries access tags, such as allowed_roles: ["staff", "payroll"], and each query carries the asker's role. The database applies the filter during search, not after. The model never sees what the user may not see.
Take one miss for the query about the refund window. The store returns six candidates with scores . Two are blocked by role: the 0.87 chunk needs payroll clearance and the 0.81 chunk needs staff clearance, and this asker has neither. The survivors rank as . With you keep and cite the 0.91 chunk first. How those scores are computed is taught in similarity metrics, and where the tags come from is taught in metadata filtering.
The cache math that pays for the wrapper
Semantic cache means near duplicate queries share one answer when their embeddings sit close together. You pick here: above it you reuse, below it you rebuild. A higher is picky and safe. A lower reuses more and risks wrong reuse.
Call the queries in the window, the hits, the misses, so and . Call the lookup time and the full retrieve plus generate time, both in milliseconds. Call the mean time your users feel:
In plain words: weight each path by how often you take it. Cost follows the same shape. When each miss costs cents and each hit costs almost nothing, spend is and savings are .
Worked example: 200 queries through the pipe
Take an illustrative hour with queries. Ninety match a cached answer above , so and . Then . Suppose and , both stated assumptions for this example rather than measured vendor figures:
Mean latency drops from 2400 ms to 1347 ms, a cut of 1053 ms per query, about 44 percent. When each miss costs 4 cents, you still spend cents and avoid cents. The diagram above draws this exact split: 200 in, 90 shortcutting to the answer in 60 ms, 110 walking the long path in 2400 ms.
The trade is plain. You buy lower bills, faster medians, fewer leaks, and a trace for every failure. You pay in moving parts: a cache to invalidate, tags to keep correct at ingest, two classifiers to tune, and logs to store. Skip the cache when answers must be fresh on every ask, such as live prices or live inventory, because a stale hit costs more than a slow miss. Skip role filters only when every doc is public. The method assumes paraphrases land near each other in embedding space, tags are right, and guardrails catch the common attacks. When paraphrase drifts, you'll get a safe miss and pay full price. When tags are wrong, you leak or you over block. When a clever injection slips past, the output check and the trace are your second and third nets. Detail on those nets is in guardrails.
Code
This snippet checks the headline numbers from the worked example. It takes the same counts and times and returns the mean your users would feel.
def mean_latency(hits: int, total: int, hit_ms: float, miss_ms: float) -> float: misses = total - hits # queries that reach retrieval plus generation return (hits * hit_ms + misses * miss_ms) / total
total = 200hits = 90mean = mean_latency(hits, total, 60.0, 2400.0)print(mean) # -> 1347.0print(hits / total) # -> 0.45print(hits * 4) # -> 360The prints match the prose: 1347.0 ms mean, 0.45 hit rate, 360 cents avoided when each miss costs 4 cents. Change in your head and watch move: a pickier cutoff lowers and pushes the mean back toward 2400.
Watch Out For
Stale cache answers after a doc update
Your refund window changes from 30 days to 14, yet users still see 30. The symptom is specific: fresh ingest looks right in a direct probe, but common phrasings keep returning the old number. The cache keyed on query similarity and never learned the source changed.
Version the key by what was cited. Store the source doc hash with each cached answer, give repeated answers a short time to live, and delete entries whose hash no longer matches on ingest. Show the source date in the answer so a stale hit looks stale.
Filtering only in the prompt still leaks
You tell the model to hide payroll docs, yet a trace shows the salary chunk in context and the answer quotes it softly. The symptom is a retrieved secret, not a rude model. Never filter only in the prompt. Instructions don't unsee context.
Attach the role filter to the vector query itself, deny by default, and prove it with a canary doc that no test user may retrieve. Read the trace for chunk ids on every sensitive ask, not just the final text.
The Quick Version
- Production RAG is core retrieval plus four jobs: access filtering, caching, guardrails, and tracing.
- Filter at retrieval time by user role, so private chunks never reach the model.
- With 90 of 200 queries cached, mean latency falls from 2400 ms to 1347 ms in the worked example.
- Guard both ends: screen input for injection and off topic asks, then check output for leaks and bad format.
- Log the query, the chunk ids used, and the answer for every request, and collect thumbs feedback to find failures.
- Skip caching when answers must be fresh on every ask, because a stale fast answer costs more than a slow right one.