Skip to content
AI360XpertBeta

Recommender System Architecture

A multi-stage funnel that takes a catalog of millions of items and filters it down to the ten picks one user sees, cheap and fast first, careful and slow last.

A cheap retrieval stage cuts a million items to a few hundred in milliseconds, so a heavy ranker that could never score the whole catalog scores only the survivors within budget
A cheap retrieval stage cuts a million items to a few hundred in milliseconds, so a heavy ranker that could never score the whole catalog scores only the survivors within budget

Why Does This Exist?

Open a video app and it shows ten picks from a catalog running into the millions. Scoring every item with the best available model, per user, per visit, is not expensive. It is impossible inside a latency budget: a heavy ranker that spends a fraction of a millisecond per item still needs minutes to cover millions. So production recommenders split the job in two. A fast, rough stage discards the obvious misses, and a slow, careful stage judges the survivors.

That split is the whole architecture. Everything else is detail about where each stage spends its milliseconds.

The one idea you need first. Retrieval means fetching a few hundred plausible candidates from the full catalog, and ranking means ordering those candidates precisely. Retrieval optimizes recall, never dropping the winner. Ranking optimizes order, putting the winner on top. The classic retrieval signal, users who liked this also liked that, is collaborative filtering. This page wires the stages together; learning to rank trains the ranking model itself, and recommendation evaluation scores the funnel once it runs.

Think of It Like This

Hiring from ten thousand resumes

You need one new hire and ten thousand people apply. Reading every application carefully takes months you do not have, so you run the process as a funnel. First a quick keyword scan discards anyone without the required experience, cutting ten thousand to a hundred in an afternoon. Then you spend the real effort, long interviews and reference checks, on those hundred, and offer the job to one.

The keyword scan is retrieval: cheap, fast, and allowed to keep some mediocre candidates, but never allowed to discard the best one. The interviews are ranking: slow, careful, and affordable only because a hundred people are left. The final offer is the top-10 carousel.

It stops here: hiring ends with a single winner, while a recommender serves millions of users with different tastes every day, and interviews happen one after another while retrieval is a parallel vector search. The funnel shape transfers. The scale does not.

How It Actually Works

Stage 1: retrieval

The goal is brutal and narrow: cut roughly a million items to a few hundred in tens of milliseconds. Speed wins over accuracy, so this stage uses two-tower neural networks, where user and item vectors are computed separately and compared with fast approximate nearest-neighbour search, or classic signals like collaborative filtering. The contract is recall, not precision. Keeping a few mediocre candidates costs nothing, because the ranker will sink them. Dropping the true best item is unforgivable, because no later stage can recover it.

Stage 2: filtering

Business rules remove what the user must not see: already purchased, out of stock, seen yesterday, failing a safety policy. This stage is usually hardcoded logic rather than learning, and it typically trims the candidate set a little further, from a few hundred to slightly fewer. Small stage, real consequences when a rule misfires.

Stage 3: ranking

Now the expensive model earns its keep. With only hundreds of survivors, the system can afford hundreds of features per candidate: time of day, interactions from minutes ago, the text of the description, long-term taste. A heavy model scores each survivor, often as a predicted probability of a tap, and sorts. Accuracy over speed, the mirror image of retrieval.

Stage 4: re-ranking

The top of the ranked list is often ten near-identical items, and identical bores. Re-ranking penalizes similarity so the final ten mix genres, creators or price points. It buys satisfaction at a small cost in raw relevance, which is a trade worth naming rather than hiding.

Worked example: the latency budget

Suppose the page must render within 100 milliseconds, and the heavy ranker costs 0.25 milliseconds per item. Scoring a million items with it takes 1,000,000×0.25 ms=250,000 ms1{,}000{,}000 \times 0.25\ \text{ms} = 250{,}000\ \text{ms}, about four minutes. Dead on arrival.

Run the funnel instead. Retrieval scans the catalog with approximate search in 10 milliseconds and returns 400 candidates. Filtering applies rules and keeps 350. Ranking scores those at full cost: 350×0.25 ms=87.5 ms350 \times 0.25\ \text{ms} = 87.5\ \text{ms}. Total: 10+87.5=97.5 ms10 + 87.5 = 97.5\ \text{ms}, inside budget. The numbers are assumed, but the shape is the point: the ranker touches 0.035 percent of the catalog, and that fraction is what makes the budget close.

Watch Out For

Tuning the ranker while retrieval drops the winner

When ranking metrics stall despite a bigger model, the instinct is a still bigger model. Often the ranker never saw the item it should have picked: retrieval discarded it with the other 999,600. The symptom is a sophisticated ranker whose offline gains never reach users. Measure retrieval recall directly, the share of eventually-clicked items present in the candidate set, before touching the ranker.

A filter rule that silently eats the catalog

Filter rules look harmless one by one, and their combination can starve entire slices of users: an over-strict freshness rule plus an over-strict quality rule leaves new users with near-empty shelves. The symptom is coverage collapsing for some cohort after an innocent rule change. Log how many candidates each rule drops per request, and alert on the drop rate, not just the final list length.

The Quick Version

  • A heavy ranker cannot score millions of items per visit, so recommenders work as a funnel: cheap retrieval first, careful ranking last.
  • Retrieval buys recall in milliseconds with two-tower models and approximate search. Its only unforgivable sin is dropping the winner.
  • Filtering enforces business rules, ranking scores the survivors with hundreds of features, and re-ranking trades a little relevance for variety.
  • At 0.25 milliseconds per item, ranking a million items takes about four minutes, while ranking 350 survivors takes 87.5 milliseconds.
  • When the ranker stalls, check retrieval recall first. When coverage collapses, check the filter rules.