Skip to content
AI360XpertBeta

RAG in Production

A demo answers one person. A production system answers hundreds at once without leaking private files or paying twice for the same question.

A production RAG request passes guardrails, a semantic cache, filtered retrieval and output checks, with 90 of 200 queries answered from cache in 60 ms instead of 2400 ms.
A production RAG request passes guardrails, a semantic cache, filtered retrieval and output checks, with 90 of 200 queries answered from cache in 60 ms instead of 2400 ms.

Why Does This Exist?

A demo RAG answers you. A production RAG answers a thousand yous at 9 a.m. on Monday, without showing salaries to interns, without paying for the same holiday question fifty times, and without going blind when it fails.

That gap is the whole page. A notebook prototype joins a vector store to a model and stops. Real traffic adds four jobs around that core: checking who may see each chunk, skipping repeat work, blocking bad input and output, and writing down what happened so you can fix it. You'll feel each one the first week you serve a team.

The one idea you need first: core RAG retrieves text chunks and then generates with them in context. If that loop is fuzzy, read RAG architecture first, because everything here assumes you know what top k retrieval means. Splitting docs is taught in chunking strategies, and measuring answer quality is taught in RAG evaluation. This page covers only the wrapper that makes that loop safe, fast, and visible.

Think of It Like This

A library with a locked archive and a photocopy shelf

You run a busy public library. Anyone may ask for anything, but back rooms need a badge. The local archive needs a member card. Staff files need a staff badge. The librarian checks your badge before fetching, not after reading the pages aloud.

The most asked forms live as photocopies by the desk. Holiday hours. Card applications. Ninety of your 200 morning visitors take a copy and leave. They never wait for a fetch, and you don't reprint the same page twice.

A bouncer turns away hecklers at the door, and a second checker skims each reply before it goes out. Every request leaves a slip: who asked, what was fetched, what was said. A small box by the exit collects thumbs up and thumbs down.

Where it stops: photocopies never change meaning, but cached answers go stale when source docs change. And a librarian says plainly when a book is missing, while a model will invent a smooth answer from nothing. You still need an explicit rule for saying you don't know.

How It Actually Works

A production request moves through six valves in order: input check, cache lookup, filtered retrieval, generation, output check, logging. You'll follow one query through them, then see what 200 do together.

Call the incoming text qq, the query text you receive. Its embedding e(q)e(q), a list of numbers standing in for meaning, is what you compare. Call the similarity score ss, the cosine between e(q)e(q) and a stored chunk, a number from 0 to 1. Call the cutoff τ\tau, the similarity threshold you demand before trusting a cached answer. Call kk, the count of chunks you pass to the model. Call hh, the hit rate, the fraction of queries served from cache.

The request path in order

  1. Input check screens qq for injection, toxicity, and off topic requests. A lightweight classifier or pattern list says yes or no before anything costly runs. See prompt injection for the attack this blocks.
  2. Cache lookup embeds qq and searches past queries. When the best match scores above τ\tau, you return the stored answer. No retrieval. No generation.
  3. Filtered retrieval runs only on a miss. You search the vector store with a role filter attached, keep the top kk allowed chunks, and drop the rest before the model ever sees them.
  4. Generation writes the answer from qq plus those kk chunks, with citations to chunk ids so the trace stays honest.
  5. Output check screens the draft for leaked secrets, banned content, and broken format. Bad drafts get blocked or rewritten.
  6. Logging writes the triple of query, chunk ids, and answer, plus user feedback. That log is how you debug next week.

Secure retrieval: filter first, then rank

Metadata filtering means each stored chunk carries access tags, such as allowed_roles: ["staff", "payroll"], and each query carries the asker's role. The database applies the filter during search, not after. The model never sees what the user may not see.

Take one miss for the query about the refund window. The store returns six candidates with scores [0.91,0.87,0.84,0.81,0.78,0.62][0.91, 0.87, 0.84, 0.81, 0.78, 0.62]. Two are blocked by role: the 0.87 chunk needs payroll clearance and the 0.81 chunk needs staff clearance, and this asker has neither. The survivors rank as [0.91,0.84,0.78,0.62][0.91, 0.84, 0.78, 0.62]. With k=3k = 3 you keep [0.91,0.84,0.78][0.91, 0.84, 0.78] and cite the 0.91 chunk first. How those scores are computed is taught in similarity metrics, and where the tags come from is taught in metadata filtering.

The cache math that pays for the wrapper

Semantic cache means near duplicate queries share one answer when their embeddings sit close together. You pick τ=0.92\tau = 0.92 here: above it you reuse, below it you rebuild. A higher τ\tau is picky and safe. A lower τ\tau reuses more and risks wrong reuse.

Call NN the queries in the window, HH the hits, MM the misses, so M=N−HM = N - H and h=H/Nh = H / N. Call LhitL_{hit} the lookup time and LmissL_{miss} the full retrieve plus generate time, both in milliseconds. Call LmeanL_{mean} the mean time your users feel:

Lmean=h⋅Lhit+(1−h)⋅LmissL_{mean} = h \cdot L_{hit} + (1 - h) \cdot L_{miss}

In plain words: weight each path by how often you take it. Cost follows the same shape. When each miss costs cc cents and each hit costs almost nothing, spend is M⋅cM \cdot c and savings are H⋅cH \cdot c.

Worked example: 200 queries through the pipe

Take an illustrative hour with N=200N = 200 queries. Ninety match a cached answer above τ\tau, so H=90H = 90 and M=110M = 110. Then h=90/200=0.45h = 90 / 200 = 0.45. Suppose Lhit=60L_{hit} = 60 and Lmiss=2400L_{miss} = 2400, both stated assumptions for this example rather than measured vendor figures:

Lmean=(90×60+110×2400)/200=269400/200=1347L_{mean} = (90 \times 60 + 110 \times 2400) / 200 = 269400 / 200 = 1347

Mean latency drops from 2400 ms to 1347 ms, a cut of 1053 ms per query, about 44 percent. When each miss costs 4 cents, you still spend 110×4=440110 \times 4 = 440 cents and avoid 90×4=36090 \times 4 = 360 cents. The diagram above draws this exact split: 200 in, 90 shortcutting to the answer in 60 ms, 110 walking the long path in 2400 ms.

The trade is plain. You buy lower bills, faster medians, fewer leaks, and a trace for every failure. You pay in moving parts: a cache to invalidate, tags to keep correct at ingest, two classifiers to tune, and logs to store. Skip the cache when answers must be fresh on every ask, such as live prices or live inventory, because a stale hit costs more than a slow miss. Skip role filters only when every doc is public. The method assumes paraphrases land near each other in embedding space, tags are right, and guardrails catch the common attacks. When paraphrase drifts, you'll get a safe miss and pay full price. When tags are wrong, you leak or you over block. When a clever injection slips past, the output check and the trace are your second and third nets. Detail on those nets is in guardrails.

Code

This snippet checks the headline numbers from the worked example. It takes the same counts and times and returns the mean your users would feel.

def mean_latency(hits: int, total: int, hit_ms: float, miss_ms: float) -> float:    misses = total - hits  # queries that reach retrieval plus generation    return (hits * hit_ms + misses * miss_ms) / total

total = 200hits = 90mean = mean_latency(hits, total, 60.0, 2400.0)print(mean)  # -> 1347.0print(hits / total)  # -> 0.45print(hits * 4)  # -> 360

The prints match the prose: 1347.0 ms mean, 0.45 hit rate, 360 cents avoided when each miss costs 4 cents. Change τ\tau in your head and watch hh move: a pickier cutoff lowers hh and pushes the mean back toward 2400.

Watch Out For

Stale cache answers after a doc update

Your refund window changes from 30 days to 14, yet users still see 30. The symptom is specific: fresh ingest looks right in a direct probe, but common phrasings keep returning the old number. The cache keyed on query similarity and never learned the source changed.

Version the key by what was cited. Store the source doc hash with each cached answer, give repeated answers a short time to live, and delete entries whose hash no longer matches on ingest. Show the source date in the answer so a stale hit looks stale.

Filtering only in the prompt still leaks

You tell the model to hide payroll docs, yet a trace shows the salary chunk in context and the answer quotes it softly. The symptom is a retrieved secret, not a rude model. Never filter only in the prompt. Instructions don't unsee context.

Attach the role filter to the vector query itself, deny by default, and prove it with a canary doc that no test user may retrieve. Read the trace for chunk ids on every sensitive ask, not just the final text.

The Quick Version

  • Production RAG is core retrieval plus four jobs: access filtering, caching, guardrails, and tracing.
  • Filter at retrieval time by user role, so private chunks never reach the model.
  • With 90 of 200 queries cached, mean latency falls from 2400 ms to 1347 ms in the worked example.
  • Guard both ends: screen input for injection and off topic asks, then check output for leaks and bad format.
  • Log the query, the chunk ids used, and the answer for every request, and collect thumbs feedback to find failures.
  • Skip caching when answers must be fresh on every ask, because a stale fast answer costs more than a slow right one.