Two-Tower Retrieval
Encode the user with one network and the item with another, then score with a dot product. The split lets you precompute item vectors and search millions in milliseconds.
Why Does This Exist?
A catalog with 10 million videos cannot be ranked one by one at login. If a joint network needs 5 milliseconds per user plus item pair, scoring the catalog costs 50 million milliseconds, which is 50,000 seconds, so a budget of tens of milliseconds forces a split.
This page covers the retrieval step that cuts millions to a few hundred. The careful ordering of those survivors lives in learning to rank, and the full funnel lives in recommender system architecture.
One idea first. An embedding is a short dense vector that stands in for a user or an item, so similarity becomes arithmetic. The classic way to learn such vectors from ratings is matrix factorization, and the neighbour counting way is collaborative filtering.
Think of It Like This
Separate questionnaires for daters and dates
You run a dating service with 1,000 daters and 1,000 profiles. You do not lock each dater in a room with each profile for an hour. That is 1,000,000 meetings.
Instead each dater fills one questionnaire about tastes, and each profile owner fills one questionnaire about traits, independently. Then you overlay the two sheets and score compatibility in seconds. The user tower is the dater questionnaire, the item tower is the profile questionnaire, and the dot product is the overlay. Item questionnaires can be filled the night before and filed, which is the precompute.
Where it stops. Real towers learn their vectors from clicks and watches, not fixed answers, and they keep updating. Questionnaires freeze what people say about themselves.
How It Actually Works
The two towers and the meeting point
Let be user features and be item features. The user tower maps to in , and the item tower maps to in . Here is the embedding size, and are the two networks, and and are the two vectors. The score is their dot product:
In words, multiply matching slots and add. Vectors pointing the same way score high. The towers never see each other until this sum, which is why item vectors can be filed away.
Worked example
Take , and . Then
Against three more items the same user scores , , and , so the top two are the item and the item. The diagram above shows this meeting point, and the code below retrieves the same pair.
Why precompute makes it fast
Because never reads , is identical for every user. Score all 10 million items once overnight and store the vectors. At login run once, about 5 milliseconds, then ask the store for the closest vectors, about 20 milliseconds with approximate search. Total about 25 milliseconds.
The trade is late interaction. The model cannot use a cross feature such as the user only liking one genre with one actor, because the two sides meet only in the sum. When ordering matters more than speed, hand the few hundred survivors to learning to rank. The method assumes a match factorizes into two sides, and it fails for cold start users or items with no history.
Code
A dot product plus top two retrieval on the worked vectors, with no tower code needed for the point.
def dot(a: list[float], b: list[float]) -> float: return sum(x * y for x, y in zip(a, b)) # the towers meet only here
def top_k(user: list[float], items: list[list[float]], k: int) -> list[int]: ranked = sorted(range(len(items)), key=lambda i: dot(user, items[i]), reverse=True) return ranked[:k] # approximate search does this fast at scale
user = [0.4, -0.2, 0.8]items = [[0.5, -0.1, 0.7], [0.1, 0.9, -0.3], [-0.4, 0.2, 0.5], [0.6, 0.1, 0.2]]print(round(dot(user, items[0]), 2)) # -> 0.78print(top_k(user, items, 2)) # -> [0, 3]The printout is for the worked pair and for the top two, which matches the arithmetic in the section above.
Watch Out For
Expecting cross feature reasoning from a late meeting
Symptom: the team adds an actor plus genre interaction, such as liking action only with one star, and retrieval keeps returning the wrong genre with the right star. Offline recall looks flat no matter how wide the towers get.
Fix: keep the two tower model for retrieval only. Pass its few hundred candidates to a ranker where user and item features concatenate early, which is the learning to rank setup. Do not widen the towers to fix a structural blind spot.
Training on random negatives and shipping popularity bias
Symptom: the top results are the same global popular items for every user. Personalization is invisible in the logs, and niche items never surface even for their fans.
Fix: replace uniform random negatives with in batch negatives plus hard negatives drawn from the same popularity band as the positive. Check the share of popular items before and after, and keep the band until that share moves.
The Quick Version
- Two towers encode the user and the item separately and meet only in a dot product.
- Item vectors never depend on the user, so 10 million of them can be precomputed overnight.
- At login one user vector plus approximate search returns candidates in about 25 milliseconds.
- The worked dot is plus plus , which equals , and the top two of four score and .
- A late meeting cannot learn cross features, so ranking still needs a heavier model.
- New users or items with no history need a cold start path alongside retrieval.