Skip to content
AI360XpertBeta

Two-Tower Retrieval

Encode the user with one network and the item with another, then score with a dot product. The split lets you precompute item vectors and search millions in milliseconds.

Two separate towers encode the user and the item, and a single dot product at the top scores the match, so item vectors can be precomputed and searched fast
Two separate towers encode the user and the item, and a single dot product at the top scores the match, so item vectors can be precomputed and searched fast

Why Does This Exist?

A catalog with 10 million videos cannot be ranked one by one at login. If a joint network needs 5 milliseconds per user plus item pair, scoring the catalog costs 50 million milliseconds, which is 50,000 seconds, so a budget of tens of milliseconds forces a split.

This page covers the retrieval step that cuts millions to a few hundred. The careful ordering of those survivors lives in learning to rank, and the full funnel lives in recommender system architecture.

One idea first. An embedding is a short dense vector that stands in for a user or an item, so similarity becomes arithmetic. The classic way to learn such vectors from ratings is matrix factorization, and the neighbour counting way is collaborative filtering.

Think of It Like This

Separate questionnaires for daters and dates

You run a dating service with 1,000 daters and 1,000 profiles. You do not lock each dater in a room with each profile for an hour. That is 1,000,000 meetings.

Instead each dater fills one questionnaire about tastes, and each profile owner fills one questionnaire about traits, independently. Then you overlay the two sheets and score compatibility in seconds. The user tower is the dater questionnaire, the item tower is the profile questionnaire, and the dot product is the overlay. Item questionnaires can be filled the night before and filed, which is the precompute.

Where it stops. Real towers learn their vectors from clicks and watches, not fixed answers, and they keep updating. Questionnaires freeze what people say about themselves.

How It Actually Works

The two towers and the meeting point

Let uu be user features and ii be item features. The user tower fuf_u maps uu to eue_u in RdR^d, and the item tower fif_i maps ii to eie_i in RdR^d. Here dd is the embedding size, fuf_u and fif_i are the two networks, and eue_u and eie_i are the two vectors. The score is their dot product:

s=eu⋅ei=∑k=1deu,kei,ks = e_u \cdot e_i = \sum_{k=1}^{d} e_{u,k} e_{i,k}

In words, multiply matching slots and add. Vectors pointing the same way score high. The towers never see each other until this sum, which is why item vectors can be filed away.

Worked example

Take d=3d = 3, eu=[0.4,−0.2,0.8]e_u = [0.4, -0.2, 0.8] and ei=[0.5,−0.1,0.7]e_i = [0.5, -0.1, 0.7]. Then

s=0.4×0.5+(−0.2)×(−0.1)+0.8×0.7=0.20+0.02+0.56=0.78s = 0.4 \times 0.5 + (-0.2) \times (-0.1) + 0.8 \times 0.7 = 0.20 + 0.02 + 0.56 = 0.78

Against three more items the same user scores −0.38-0.38, 0.200.20, and 0.380.38, so the top two are the 0.780.78 item and the 0.380.38 item. The diagram above shows this 0.780.78 meeting point, and the code below retrieves the same pair.

Why precompute makes it fast

Because fif_i never reads uu, eie_i is identical for every user. Score all 10 million items once overnight and store the vectors. At login run fuf_u once, about 5 milliseconds, then ask the store for the closest vectors, about 20 milliseconds with approximate search. Total about 25 milliseconds.

The trade is late interaction. The model cannot use a cross feature such as the user only liking one genre with one actor, because the two sides meet only in the sum. When ordering matters more than speed, hand the few hundred survivors to learning to rank. The method assumes a match factorizes into two sides, and it fails for cold start users or items with no history.

Code

A dot product plus top two retrieval on the worked vectors, with no tower code needed for the point.

def dot(a: list[float], b: list[float]) -> float:    return sum(x * y for x, y in zip(a, b))  # the towers meet only here

def top_k(user: list[float], items: list[list[float]], k: int) -> list[int]:    ranked = sorted(range(len(items)), key=lambda i: dot(user, items[i]), reverse=True)    return ranked[:k]  # approximate search does this fast at scale

user = [0.4, -0.2, 0.8]items = [[0.5, -0.1, 0.7], [0.1, 0.9, -0.3], [-0.4, 0.2, 0.5], [0.6, 0.1, 0.2]]print(round(dot(user, items[0]), 2))  # -> 0.78print(top_k(user, items, 2))  # -> [0, 3]

The printout is 0.780.78 for the worked pair and [0,3][0, 3] for the top two, which matches the arithmetic in the section above.

Watch Out For

Expecting cross feature reasoning from a late meeting

Symptom: the team adds an actor plus genre interaction, such as liking action only with one star, and retrieval keeps returning the wrong genre with the right star. Offline recall looks flat no matter how wide the towers get.

Fix: keep the two tower model for retrieval only. Pass its few hundred candidates to a ranker where user and item features concatenate early, which is the learning to rank setup. Do not widen the towers to fix a structural blind spot.

Training on random negatives and shipping popularity bias

Symptom: the top results are the same global popular items for every user. Personalization is invisible in the logs, and niche items never surface even for their fans.

Fix: replace uniform random negatives with in batch negatives plus hard negatives drawn from the same popularity band as the positive. Check the share of popular items before and after, and keep the band until that share moves.

The Quick Version

  • Two towers encode the user and the item separately and meet only in a dot product.
  • Item vectors never depend on the user, so 10 million of them can be precomputed overnight.
  • At login one user vector plus approximate search returns candidates in about 25 milliseconds.
  • The worked dot is 0.200.20 plus 0.020.02 plus 0.560.56, which equals 0.780.78, and the top two of four score 0.780.78 and 0.380.38.
  • A late meeting cannot learn cross features, so ranking still needs a heavier model.
  • New users or items with no history need a cold start path alongside retrieval.