RAG Evaluation

Why Does This Exist?

You ship a RAG assistant, collect fifty test questions, and grade the final answers. Say it gets thirty right. That headline number feels like progress, until someone asks the only question that matters: why did the other twenty fail? Did the search step never surface the right chunk, or did the model hold the right chunk and write past it? One end-to-end score answers neither. It blends two different failures into one shrug.

That blend costs real weeks. A team reading a low answer score as a writing problem will tune prompts, swap system instructions, and plead with the model to stick to context, while the actual cause sits upstream: the index never returned the one chunk with the answer. No prompt edit can summon text the model never received. You have to measure the two stages apart before you fix either.

This page is about that measurement. The pipeline itself, how chunks get indexed and stuffed into the prompt, lives in RAG architecture. The cheapest-first debugging order for a bad answer lives in RAG failure diagnosis. Here you learn the two score families, how to compute them on one small example, and what each result tells you to do next. If you came from model testing, note the scope difference: post-training evaluation grades the model in isolation, while this page grades the whole system, search step included.

The one idea you need first

Every score here needs a labeled eval set: a list of representative questions, each paired with the ids of the chunks that actually answer it, plus a short reference answer. Retrieval metrics read the chunk ids. Generation checks read the reference answer and the retrieved text. Without those labels you can stare at outputs all day, but you cannot compute a single number on this page. Building that set is unglamorous work, and it is the whole foundation.

Think of It Like This

How It Actually Works

A RAG answer passes through two gates. First, the retriever picks the top chunks for the query. Then the model writes an answer from those chunks plus its own weights. Evaluation scores each gate on its own terms, using the same eval set, and reports two numbers side by side. Retrieval gets graded against the labeled chunk ids. Generation gets graded against the retrieved text and the reference answer.

Retrieval first: did the right chunks show up?

Call the labeled relevant chunks for a query the set $R$, and call the retriever's top results the list $D_k$, where $k$ is just your cutoff, how many chunks you asked for. Two ratios tell the story:

In plain words: recall@k is the share of the needed chunks that showed up anywhere in the top $k$. Precision@k is the share of the retrieved chunks that were actually needed. The vertical bars mean set size, and $R \cap D_k$ is the overlap, chunks that are both relevant and retrieved.

Recall is the one you watch most, because a missing chunk is unrecoverable downstream. If recall@3 is 0.50, half the needed material never reached the prompt, and the model starts every answer half blind. Precision matters too, mostly as context budget: stuffing the prompt with irrelevant chunks wastes tokens and distracts the model. When recall is low, the levers sit on the retrieval side, a fuller index in your vector database, better chunk boundaries, or a reranking pass over the candidates. No prompt edit belongs in that list.

Generation next: did the answer stay loyal?

Assume retrieval did its job and grade only the writing. Split the answer into its checkable claims, call that count $C$, and count how many are supported by the retrieved passages, call that $S$:

Faithfulness is that share. A claim counts as supported only if some retrieved passage actually states it, not if it sounds plausible or matches the model's training memory. An answer can also fail a second, separate check: answer relevance, whether it addresses the question asked at all. A perfectly grounded summary of the wrong topic is faithful and useless. Keep the two apart, because they fail independently and get fixed differently.

Worked example: one query, two scores

Take the query "refund window on sale items?" Your labels say two chunks answer it, so $R = \{\text{C2}, \text{C4}\}$: C2 states the 30-day refund policy, C4 states that sale items are final sale. The retriever returns $D_3 = [\text{C1}, \text{C2}, \text{C3}]$, shipping times, the refund policy, and support hours. The overlap $R \cap D_3$ is just $\{\text{C2}\}$, size 1:

Half the needed material arrived. The sale exclusion, the crux of the question, never reached the prompt.

Now the model answers in four claims: refunds take 30 days (backed by C2), contact support through the portal (backed by C3), replies come within 2 business days (backed by C3), and sale items qualify for refunds (backed by nothing, and wrong, since the missing C4 says the opposite). So $S = 3$, $C = 4$:

Read the pair the way the diagram does. Recall 0.50 says the search step dropped C4, so the first fix is retrieval, not wording. Faithfulness 0.75 says the model additionally invented the sale-items claim instead of saying the passages were silent. One end-to-end grade, wrong answer, would hide both facts. The split hands you an ordered todo: fetch C4 first, then tighten grounding.

Why one final score cannot do this job

Four combinations produce the same wrong answer: bad retrieval with good writing, good retrieval with sloppy writing, both bad, and the sneaky one, bad retrieval papered over by a fluent invention. A single correctness number collapses all four into identical failure, so every fix becomes a guess. Two scores separate them, and the separation is the entire technique.

It costs something honest. Labels take real effort: representative questions, chunk ids per question, reference answers, all maintained as your docs change. That is the trade-off, labeling labor in exchange for knowing which half of the system to touch. The standing assumption is that your eval questions mirror real traffic; when they drift, every number on this page quietly goes stale. And one boundary: skip the full harness while the product is still a toy with no users and no direction, but keep a tiny golden set from day one, because the week you need it is the week you will not have time to build it. When the scores point at a failure, the debugging order continues in RAG failure diagnosis.

Code

Two tiny scorers, run on the worked example above. Retrieval and generation stay in separate functions on purpose, so the two numbers can never collapse into one.

1def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:2    top_k = set(retrieved[:k])3    return len(top_k & relevant) / len(relevant)4
5
6def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:7    top_k = retrieved[:k]8    return sum(1 for doc in top_k if doc in relevant) / k9
10
11def faithfulness(backed: list[bool]) -> float:12    return sum(backed) / len(backed)13
14
15retrieved = ["C1", "C2", "C3"]16relevant = {"C2", "C4"}17
18print(f"{recall_at_k(retrieved, relevant, k=3):.2f}")      # -> 0.5019print(f"{precision_at_k(retrieved, relevant, k=3):.2f}")   # -> 0.3320print(f"{faithfulness([True, True, True, False]):.2f}")    # -> 0.75

Retrieval reports 0.50 while generation reports 0.75, the exact split verdict from the worked example. One blended score would have printed a single failure and told you nothing about where to look.

Watch Out For

The Quick Version

• RAG evaluation reports two numbers, a retrieval score and a generation score, because one final-answer grade cannot say which stage failed.
• Recall@k is the share of needed chunks that arrived in the top results, and low recall means the fix sits in search, never in the prompt.
• Faithfulness is the share of answer claims supported by retrieved passages, and it catches fluent inventions a correctness grade lumps together with retrieval misses.
• In the worked example, recall@3 of 0.50 plus faithfulness of 0.75 orders the todo: fetch the missing chunk first, then tighten grounding.
• Every metric here rests on a labeled eval set of representative questions, chunk ids, and reference answers, and stale questions make honest numbers lie.

What to Read Next

• RAG Architecture
• RAG Failure Diagnosis
• RAG in Production