Skip to content
AI360XpertBeta

Relevance Tuning

One query looks fine while a thousand others quietly break. Relevance tuning means grading a test set of queries once, scoring every change against it, and only shipping the change that lifts the whole set.

For one query, tuning the retrieval mix moves the perfect document from rank 3 to rank 1, lifting NDCG from 0.59 to 1.00
For one query, tuning the retrieval mix moves the perfect document from rank 3 to rank 1, lifting NDCG from 0.59 to 1.00

Why Does This Exist?

Someone types login page error. Your search returns a tutorial for building a login page at rank 1, and the actual troubleshooting guide sits at rank 3. You nudge the settings until that query looks right, ship it, and never notice you pushed five other queries down. That's the trap: judging a whole engine by the one query in front of you.

Relevance tuning is the discipline that replaces eyeballing with measurement. You grade a fixed set of test queries once, turn every change into a single score, and ship only the change that lifts the whole set. The tuning itself is cheap. The test set is the work, and it's what makes every later decision safe.

This page covers that discipline: the test set, the score, and the levers. The levers themselves have their own pages. Reranking is the strongest one, a slower model that rescores your shortlist. Hybrid search is the mix of word matching and meaning matching whose balance you'll tune. And RAG evaluation zooms out further, grading the generated answer on top of the retrieval this page tunes.

Think of It Like This

A chili cook-off, not your own lunch

You are perfecting a chili recipe for a cook-off with fifty judges. Tasting it yourself tells you almost nothing, because your own bowl is one palate and the judges are fifty. So you hand out scorecards, every judge grades every batch, and you change one ingredient at a time and watch the average.

The scorecards are your judgment set. The average is your metric. The one-ingredient rule is the whole game: change the beans and the heat together and you'll never know which one the judges liked.

Where it stops: a judge scores each bowl on its own, but search has an order. A perfect document at rank 3 is a worse result than the same document at rank 1, and no scorecard captures that. That's why the metric discounts by position, which no cook-off needs.

How It Actually Works

A retrieval pipeline turns a query into a ranked list, and every list it produces is a bet about which document deserves each position. Relevance tuning checks those bets against human grades, one number per query, then averages across the set. Three pieces, in order.

Grades first: the judgment set

Pick a set of queries that looks like your real traffic, and for each one, grade a handful of documents on a fixed scale. Most teams use 0 for useless through 3 for perfect, with 1 and 2 for partial matches. So for the query reset password, the step-by-step reset guide gets a 3, the login FAQ that mentions passwords in passing gets a 1, and the pricing page gets a 0.

Those grades are the ground truth everything else is scored against. Small teams grade by hand. Larger ones draft grades with LLM-as-a-judge setups and spot-check them. Either way, the set has to cover the queries your users actually ask. A test set of only easy queries will bless an engine that fails on the hard ones.

One ranking becomes one number

The standard score is NDCG, normalized discounted cumulative gain. The name is three ideas stacked:

  • Gain. Each document at each position contributes (2rel−1)(2^{rel} - 1), where relrel is its grade. The exponent means a perfect document is worth far more than a partial one: grade 3 contributes 7, grade 1 contributes just 1.
  • Discounted. That gain is divided by log⁡2(i+1)\log_2(i + 1), where ii is the 1-based position. A perfect document at rank 1 counts in full. The same document at rank 3 is divided by log⁡2(4)=2\log_2(4) = 2, so it counts half.
  • Normalized. Divide by the score of the ideal ordering, so a perfect ranking always scores 1.00 no matter how many relevant documents exist.

Add up the discounted gains down the ranking and you get the DCG. Divide by the ideal ranking's DCG and you get the NDCG, always between 0 and 1.

Worked example

Four documents carry grades A = 3, B = 1, C = 0, D = 0. Your engine returns them in the order [B,C,A,D][B, C, A, D], with the perfect guide buried at rank 3:

DCG=11+0log⁡23+72+0=1+0+3.5=4.5\text{DCG} = \frac{1}{1} + \frac{0}{\log_2 3} + \frac{7}{2} + 0 = 1 + 0 + 3.5 = 4.5

The ideal order is [A,B,C,D][A, B, C, D], and since log⁡23≈1.585\log_2 3 \approx 1.585:

IDCG=71+11.585+0+0≈7+0.63=7.63\text{IDCG} = \frac{7}{1} + \frac{1}{1.585} + 0 + 0 \approx 7 + 0.63 = 7.63

So NDCG=4.5/7.63≈0.59\text{NDCG} = 4.5 / 7.63 \approx 0.59. One number, and it points straight at the problem: the 7 points sitting at rank 3 got halved by the discount. Tune the pipeline until it returns [A,B,C,D][A, B, C, D], and the NDCG is 1.00. That 0.59 to 1.00 climb is exactly what the diagram above draws.

Turn one lever at a time

With the score in place, the levers are ordinary retrieval choices. The blend weight between word matching and vector matching. Whether a reranker rescores the top results, and how deep that shortlist goes. Filters, freshness boosts, and how the documents were sliced up in the first place (see chunking strategies). Change one, rescore the whole set, keep the change only if the mean NDCG rises.

Notice the trade-off. Tuning buys ranking quality you can defend with a number, and it costs judgment labor plus a full rescore for every experiment. Skip it on a tiny prototype corpus you can still read end to end. Start it the week eyeballing stops working, which is earlier than most teams admit.

And know the assumption it rests on: your judgment set stands in for live traffic. When the documents change or the questions drift, the grades go stale and the metric starts blessing yesterday's engine. Re-grade on whatever cadence your corpus churns at.

Code

This snippet is the check, not the formula retyped. It scores two orderings of the worked example against the ideal, in plain standard-library Python:

import math

def dcg(grades: list[int]) -> float:    # position i is 1-based, so the discount uses log2(i + 1)    return sum((2**g - 1) / math.log2(i + 2) for i, g in enumerate(grades))

def ndcg(ranked: list[int], ideal: list[int]) -> float:    return dcg(ranked) / dcg(ideal)  # 1.0 means the order matches the ideal

ideal = [3, 1, 0, 0]print(round(ndcg([1, 0, 3, 0], ideal), 2))  # -> 0.59print(round(ndcg(ideal, ideal), 2))  # -> 1.0

The first line reproduces the hand computation from the worked example: the perfect document buried at rank 3 scores 0.59. The second confirms the ideal order scores exactly 1.0. When your engine's real rankings disagree with this function, suspect your grading data, not the arithmetic.

Watch Out For

Your NDCG climbs while your users get unhappier

This is Goodhart's law wearing a search costume. The judgment set is a proxy for what users want, and tuning against it long enough teaches the engine the proxy instead. You'll see it as a slow split: offline NDCG keeps rising across experiments while live clicks, copy-pastes, or support tickets sit flat or get worse.

Hold out a slice of judged queries you never tune against and report both numbers. Small gains on the tuned slice that vanish on the held-out slice are overfitting, full stop. And confirm any large change with a live A/B test before calling it a win. When the two disagree, trust the users.

Judgments that quietly expire

Your corpus keeps growing but your grades do not. New documents arrive ungraded, so the metric treats them as irrelevant by default and keeps rewarding the old winners. The symptom is a flat NDCG beside fresh pages your users clearly love, or new topics that never surface no matter how you tune.

Tie re-grading to corpus churn, not the calendar. Every refresh should add recent real queries, including ones the engine currently gets wrong, and grade the new documents against them. A judgment set is a living test suite. Unmaintained, it becomes a monument.

The Quick Version

  • Relevance tuning means grading a fixed set of test queries once, then scoring every retrieval change against those grades instead of eyeballing one query.
  • A judgment set pairs each test query with documents graded from 0 for useless through 3 for perfect.
  • NDCG divides a ranking's position-discounted gain by the ideal ranking's gain, so 1.00 means perfect order and 0.59 means a good answer is buried.
  • Change one lever at a time, the hybrid mix, the reranker, the filters, and keep only what lifts the whole set's mean score.
  • Judgment sets mirror today's traffic, not tomorrow's, so re-grade whenever the documents or the questions move.