RAG Failure Diagnosis

A wrong RAG answer lives in one of four stages. You check ingestion, chunking, retrieval, then generation in that order, cheapest first, so you fix what actually broke.

A refund question fails at the chunking gate because no single chunk holds 30 days whole, so diagnosis stops before retrieval or generation are blamed

Why Does This Exist?

You shipped a RAG demo. Someone asks what the refund window is, and the bot says it cannot find the policy. You know the policy file is in the folder you indexed. Your first instinct is to rewrite the system prompt. Do not start there.

That edit helps in exactly one of four cases: the model received the right passage and ignored it. If the file never made it into the index, if a chunk boundary sliced the answer in half, or if search never surfaced the intact chunk, no wording change moves the needle. You will iterate for an afternoon and the answer will stay wrong in exactly the same way.

The one idea you need first is the pipeline itself. RAG architecture indexes documents once, retrieves per query, then generates from what was retrieved. RAG evaluation scores retrieval and generation separately so a headline number stops hiding which half failed. This page does the single-answer version of that split: given one wrong answer, you walk four gates in cheapest-first order and stop at the first one that fails.

Think of It Like This

Tracing a missing parcel

You order a part and it never arrives. Four different things could have gone wrong. The warehouse never stocked it. It was stocked but shelved across two bins so neither bin holds a complete part. Staff picked the wrong box. Or the right box arrived and you never opened it.

You would not start by rewriting the delivery instructions. You check the stock record first, then the shelf, then the picking log, and only then the doorstep. Each check is cheaper than the next, and each one makes the later checks pointless if it fails. RAG diagnosis runs the same way, from the index up to the model.

Where it stops: parcels are either on the shelf or they are not, while retrieval is ranked. A chunk can be indexed yet score too low to reach the top results. So the retrieval question is not just is it shelved, but did it rank high enough to be handed over.

How It Actually Works

A wrong answer comes from one of four gates. You test them upstream to downstream, cheapest first, and you stop at the first failure. Ingestion asks whether the fact is in the index at all. Chunking asks whether one stored chunk holds it whole. Retrieval asks whether search returned that chunk for this query. Generation asks whether the model used what it was handed.

Notation you will see below: $q$ is the query text you send, $C$ is the set of stored chunks in the index, $k$ is the number of passages you retrieve and hand the model, and recall@ $k$ is the share of truly relevant passages that show up in those top $k$ . In plain words, recall@ $k$ tells you whether search found the right material before the model ever saw it.

The four gates in cheapest order

Start with ingestion because it is a direct text search over raw indexed content. Search the stored source for the expected fragment, bypassing retrieval and the model entirely. If it is absent, nothing downstream can succeed. Fix the loader, the parser, or the refresh before you touch anything else.

If ingestion passes, inspect the stored chunks around the answer. A boundary that cuts between words leaves neither half sufficient, and the symptom is a partially right answer built from half a fact. The fix lives in semantic chunking: larger windows, overlap, or boundaries that follow topic shifts instead of character counts.

If one chunk holds the answer whole, run $q$ against the index on its own and look at the top $k$ . This isolates search from generation. When the intact chunk exists but misses the top $k$ , the fix is on the retrieval side, often reranking or a query rewrite, not a prompt rewrite. Only when the right chunk reached the prompt and the answer still contradicts it have you earned the right to edit the prompt.

Worked example: 30 days split across two chunks

Take $q$ as what is the refund window and the expected fragment as 30 days. The source document holds ten words:

Our(1) refund(2) policy(3) allows(4) returns(5) within(6) 30(7) days(8) of(9) purchase(10).

Ingestion check: search the one indexed source for 30 days. It appears across positions 7 to 8, so 1 of 1 sources holds it. Pass.

Chunking used a fixed window of seven words with no overlap. So $C$ has two members. $C_1$ holds words 1 to 7, Our refund policy allows returns within 30, and $C_2$ holds words 8 to 10, days of purchase. Search each chunk for the whole two-word fragment 30 days. $C_1$ has 30 without days, $C_2$ has days without 30, so 0 of 2 chunks holds it whole. Fail. Stop there.

You never reach retrieval or generation in this trace. You would have asked for the top $k = 2$ passages, and recall@ $2$ is beside the point here because there is no intact relevant chunk for search to find. The diagram shows exactly this state: ingestion marked pass, chunking marked fail with the 30 and days halves drawn apart, and the two downstream gates dimmed as not reached.

Cost, limits, and when a single trace misleads

The trade-off is minutes against hours. Two literal searches over stored text cost you minutes and rule out the two cheapest causes. Skipping them to tune prompts or embeddings first feels faster and burns far longer when the cause sat upstream.

Do not use a single-answer trace when the problem is aggregate. If your eval set shows low recall@ $k$ across dozens of queries, go to RAG evaluation and fix retrieval as a system instead of chasing one query. And do not use this trace for refusals or policy blocks. A model that withholds an answer on purpose did not fail to retrieve it.

This procedure assumes the expected answer is a contiguous string you can grep for, and it fails when the answer needs paraphrase or synthesis across chunks. If the doc says thirty-day window and you grep for 30 days, the literal check reports a miss that is not real. Normalize obvious variants before you call ingestion failed, and treat multi-hop questions as a separate problem.

Code

This snippet runs the same four gates in order and stops at the first failure. Types are explicit and the printed label is the diagnosis you act on.

1def diagnose(2    expected: str,3    raw_documents: list[str],4    chunks: list[str],5    retrieved: list[str],6    answer: str,7) -> str:8    if not any(expected in doc for doc in raw_documents):9        return "INGESTION"10    if not any(expected in chunk for chunk in chunks):11        return "CHUNKING"12    if not any(expected in chunk for chunk in retrieved):13        return "RETRIEVAL"14    if expected not in answer:15        return "GENERATION"16    return "OK"17
18
19docs = ["Our refund policy allows returns within 30 days of purchase."]20stored = [21    "Our refund policy allows returns within 30",22    "days of purchase.",23]24pulled = ["Our refund policy allows returns within 30"]25draft = "I do not see the refund window."26
27print(diagnose("30 days", docs, stored, pulled, draft))  # -> CHUNKING

The call returns CHUNKING because 30 days sits in the source yet in neither stored chunk whole. That matches the hand trace above: 1 of 1 sources pass, 0 of 2 chunks fail, and the later stages never decide anything.

Watch Out For

Rewriting the prompt when the chunk never arrived

Symptom: you try three prompt wordings and the answer stays wrong in the same way, still missing the same fact. The model cannot cite a passage it never received, so each rewrite tests nothing.

Fix: before any prompt edit, grep the index for the expected fragment and then grep the stored chunks for the whole fragment. If either misses, fix ingestion or chunking first. Never retune a prompt on a missing-fact answer until you have seen the intact chunk inside the retrieved context.

Calling a split chunk a retriever bug

Symptom: recall@ $k$ sits at zero, so you retune embeddings or add a reranker, and the score does not move. The retriever looks broken, but there was no whole chunk to rank. No scorer rescues a fragment that was cut in half at index time.

Fix: check chunk-intact before retrieval quality. If 0 of your stored chunks holds the expected fragment whole, widen the window, add overlap, or move to topic-following splits, then re-measure. Only tune search once a whole chunk exists.

The Quick Version

A wrong RAG answer comes from ingestion, chunking, retrieval, or generation, and each stage has its own fix.
Check them cheapest first and stop at the first failure, so a missing document never triggers a week of prompt edits.
A chunk boundary that splits 30 and days leaves 0 of 2 chunks sufficient, which no retriever or prompt can rescue.
Retrieval gets blamed only after you prove one stored chunk holds the answer whole yet misses the top results.
Generation is at fault only when the right chunk reached the prompt and the answer still contradicts it.