Recommendation Evaluation
Standard metrics like accuracy do not care where the right answer sits in a list. Ranking metrics do, and diversity checks stop the list going stale.
Why Does This Exist?
A recommender does not output one answer. It outputs a ranked list, usually ten items, and position is the product. A liked film at rank 1 means the user taps immediately. The same film at rank 10 means scrolling past nine misses first. Any metric blind to order calls those two outcomes equal, and they are not.
So offline evaluation, scoring a model before it ever faces an A/B test, needs metrics that punish burying the good items. That is what this page builds: the rank-aware scores, plus the beyond-accuracy checks that stop a high-scoring list from boring everyone.
The one idea you need first. Precision is the share of shown items the user wanted, and recall is the share of wanted items the model showed. Both ignore order entirely. The full treatment is in classification metrics. This page starts where those stop: same precision and recall, different ranking, different user. Its neighbours are the system being scored, the recommender funnel, and the model family trained against these scores, learning to rank.
Think of It Like This
Two librarians, one book
You ask two librarians for a book you want. The first hands you that exact book. The second wheels over a cart of a hundred books with yours buried at the bottom. By a metric that only asks whether you got the book, both score full marks. By any metric that values your afternoon, the first librarian wins by a mile, because the second made you dig through ninety-nine wrong answers first.
The book is the relevant item, the cart is the top-K list, and your digging is scroll cost. That is the whole intuition behind rank-aware metrics.
It stops here: you wanted one specific book, while recommender users want several items with shades of liking, a three-star film and a two-star one. The librarian story cannot grade that. NDCG below can, because it takes graded relevance as its input.
How It Actually Works
Recall at K and precision at K
Look only at the top items, usually the ten the user sees. is the cutoff rank, fixed before scoring. Recall at is the fraction of everything the user wanted that appears in those top . Precision at is the fraction of those top the user wanted. Both are simple and both share one blindness: rank 1 and rank count exactly the same.
Mean reciprocal rank
Mean reciprocal rank, MRR, cares about order under one assumption: the user wants a single thing, like the answer to a search query. Find the rank of the first relevant item, take its reciprocal , and average over users. First hit at rank 1 scores , at rank 2 scores , at rank 10 scores . Clean, but it stops reading after the first success, so it undervalues lists meant to fill an evening with several picks.
NDCG
NDCG is the standard because it drops both limitations. It takes graded relevance, discounts by position, and normalizes against perfection. For a list with relevance grades at ranks , where is how much the user liked the item sitting at rank :
The gain rewards highly relevant items exponentially, a 3-star pick earns far more than a 1-star one. The discount shrinks that gain the deeper the item sits. DCG is then divided by the DCG of the ideal ordering of the same items, so NDCG always lands between 0 and 1, with 1.00 meaning perfectly ordered.
Worked example
One user, four movies, relevance grades known. List A orders them , best first. List B reverses that to , best last. The user liked three of the four.
List A term by term: rank 1 gives . Rank 2 gives . Rank 3 gives . Rank 4 gives zero. DCG is , and since the order is ideal, NDCG is .
List B: rank 1 gives zero. Rank 2 gives . Rank 3 gives . Rank 4 gives . DCG is , so NDCG is , about .
Now the punchline. Recall at 4 is for both lists, and precision at 4 is for both. Those metrics cannot tell the lists apart. MRR can: for A against for B. NDCG sees the full picture: against .
Beyond accuracy: coverage and diversity
A list can ace NDCG and still fail users. Three checks catch that:
- Catalog coverage is the share of the catalog shown to at least one user. Recommending the same fifty blockbusters from ten thousand titles is a coverage of half a percent and a wasted catalog.
- Diversity asks whether the ten items differ from each other. Ten near-identical picks score well item by item and bore the user as a set.
- Novelty asks whether the picks include things the user would like but never find alone. Milk next to cereal is accurate and teaches nothing.
Watch Out For
Trusting offline NDCG further than it reaches
NDCG is computed on history, so it only scores items the user already touched. A model that surfaces an obscure film the user would love gets a zero for that pick, because the logs show no interaction. The symptom is a model that climbs offline and goes flat in the live test. Treat offline scores as a filter for bad ideas, then prove the survivors with an A/B test, which measures what users do rather than what they did.
Picking a K that flatters the model
Recall at 50 almost always looks better than recall at 5, and neither number means anything until matches the surface. The symptom is a metric that climbs while the product feels unchanged: the model improved at rank 40, where nobody scrolls. Set to the number of items the user actually sees, the carousel length or the first screen, and report that number next to the score.
The Quick Version
- Rank 1 and rank 10 are different products, so recommenders need metrics that punish burying good items.
- Recall at K and precision at K ignore order: both lists in the example score 1.00 and 0.75 despite reversed quality.
- MRR scores of the first hit, which suits single-answer needs like search.
- NDCG discounts graded gain by and normalizes against the ideal order: 1.00 against 0.55 in the example.
- High NDCG with low coverage, diversity or novelty is a boring recommender. Measure all four.