Gen AI

RAG Evaluation

Retrieval quality and generation quality are two separate scores measuring two separate stages, and a single end-to-end answer-quality number can't tell you which of the two actually failed.

Retrieval quality and generation quality are two separate scores measuring two separate stages, and scoring only the final answer conflates two different failure causes into one number

Why Does This Exist?

It's tempting to evaluate a RAG system the way you'd evaluate any question-answering system: take a set of test questions, generate an answer for each, and score how good the final answers are. That single number is genuinely useful as a headline metric, but it hides a critical ambiguity: a low score doesn't tell you why the answers were bad. Retrieval might have found the right passage every time while the model ignored it during generation — or retrieval might never have found the right passage at all, in which case no amount of generation quality could have produced a correct answer.

Conflating these two very different failure causes into one end-to-end score is the single most common process error in evaluating a RAG system, and it directly costs debugging time: a team chasing a low answer-quality score by tweaking prompts, when the actual problem is that retrieval never surfaces the right document, will iterate for a long time without improving anything.

Think of It Like This

Grading a research paper on two separate rubrics

Grading a student's research paper on a single overall score hides whether a weak grade came from citing the wrong sources or from writing a poor analysis of the right ones. A more useful grading approach splits the rubric in two: one score for whether the student found and cited relevant, correct sources, and a separate score for whether their actual analysis and writing, given those sources, was any good.

A student who found excellent sources but wrote a sloppy analysis needs different feedback than one who wrote brilliantly about entirely the wrong sources. RAG evaluation makes exactly this same split, on purpose, because the two failure modes call for genuinely different fixes.

How It Actually Works

Retrieval metrics: was the right material even found?

Retrieval metrics measure the search stage in isolation, independent of anything the model later does with what was retrieved. Recall@k measures, out of the truly relevant passages for a query, what fraction actually appear somewhere in the top- $k$ retrieved results. Precision@k measures, out of the passages actually retrieved, what fraction were genuinely relevant. Both require a labeled set of query-to-relevant-passage pairs to compute against, established ahead of time — without that ground truth, there's no way to check whether retrieval found the right thing, only whether it found something.

Generation metrics: was the retrieved material used correctly?

Generation metrics assume retrieval already succeeded and evaluate only what the model did with the material it was handed. Faithfulness (sometimes called groundedness) checks whether every claim in the generated answer is actually supported by the retrieved passages, rather than the model adding unsupported claims from its own general knowledge — a faithfulness failure is a specific, checkable form of hallucination, distinguishable from an answer that's simply wrong because retrieval failed. Answer relevance checks whether the generated answer actually addresses the question that was asked, independent of whether it's grounded correctly — an answer can be perfectly faithful to the retrieved material while still failing to actually answer the specific question posed.

Why measuring both separately, not just the final answer, matters

The diagram above's core claim is structural: a bad final answer can originate in either box, or both, and an end-to-end score alone can't distinguish which. Measuring retrieval and generation as two separate scores turns "the system got 60% of test questions wrong" into an actionable diagnosis — if recall@k is high but faithfulness is low, the fix is on the generation side (better prompting, a model more reliable at staying grounded); if recall@k itself is low, no amount of generation-side tuning will help, because the correct material was never available to generate from in the first place.

The dependency this evaluation has on labeled data

Both retrieval and generation metrics, done rigorously, need some form of ground truth: a set of representative queries paired with which passages actually answer them (for retrieval metrics), and reference answers or a reliable way to judge faithfulness (for generation metrics). Building this evaluation set is real, upfront work, and skipping it in favor of purely eyeballing outputs is a common shortcut that leaves a team unable to say, with any confidence, whether a change to the system actually helped or hurt.

Show Me the Code

Computing recall@k for retrieval and a simple faithfulness check for generation, kept as two distinct, separately-reported numbers.

1def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:2    top_k = set(retrieved[:k])3    return len(top_k & relevant) / len(relevant) if relevant else 0.04
5
6def faithfulness(answer_claims: list[str], retrieved_passages: list[str]) -> float:7    """Fraction of claims that appear supported by at least one retrieved passage."""8    supported = sum(any(claim in passage for passage in retrieved_passages) for claim in answer_claims)9    return supported / len(answer_claims) if answer_claims else 0.010
11
12retrieved_docs = ["doc_refund_policy", "doc_shipping_times", "doc_support_hours"]13relevant_docs = {"doc_refund_policy"}14
15print(recall_at_k(retrieved_docs, relevant_docs, k=3))    # -> 1.0 -- the relevant doc was retrieved16
17claims = ["refunds take 30 days", "shipping is free worldwide"]18passages = ["Our refund policy allows refunds take 30 days from purchase."]19print(faithfulness(claims, passages))    # -> 0.5 -- only the first claim is actually supported

recall_at_k reports a perfect score for retrieval here, while faithfulness catches that the generated answer includes an unsupported claim about free worldwide shipping — exactly the kind of split verdict a single end-to-end score would have hidden entirely.

Watch Out For

Scoring only the final answer and skipping retrieval metrics entirely

Measuring only whether generated answers match a reference answer, without ever checking recall or precision on retrieval independently, leaves a team with no way to tell whether a low score comes from bad retrieval or bad generation. This is the exact process error this page opens with, and it's easy to fall into simply because reference-answer scoring is more familiar from other NLP evaluation contexts, even though it hides the diagnosis a RAG system specifically needs.

Building a retrieval evaluation set that doesn't reflect real query patterns

Retrieval metrics are only as trustworthy as the labeled query-to-relevant-passage pairs they're computed against, and a labeled set built from easy, clearly-phrased test queries can report excellent recall while real users' messier, more ambiguous queries perform far worse in production. An evaluation set needs to genuinely reflect the query patterns a system actually receives, not just whatever queries were easiest to label.

The Quick Version

Retrieval metrics (recall@k, precision@k) measure whether the search stage found the right material, independent of generation.
Generation metrics (faithfulness, answer relevance) measure whether the model used retrieved material correctly, assuming retrieval already succeeded.
A single end-to-end answer-quality score can't distinguish a retrieval failure from a generation failure — measuring both separately can.
Faithfulness specifically checks whether every claim in an answer is supported by retrieved passages, catching a distinct form of hallucination.
Both kinds of metrics need real, upfront labeled data to compute rigorously — skipping that leaves a team unable to measure whether changes actually help.