Late Interaction Retrieval (ColBERT)
You keep one vector per token instead of squeezing a document into one, then score each query token against its best match in the document. The detail a single pooled vector throws away is exactly what this keeps.
Why Does This Exist?
Single-vector search is fast because everything is precomputed. You embed each document once into one vector, embed the query into one vector, and rank with a dot product. That works until a document says too much for one vector to hold. A long passage about refunds, late delivery, and warranty claims gets squeezed into a single point, and the query "late refund" matches the blur instead of the clause it needed. This is the bargain embeddings force on you: one vector, one meaning, and trouble whenever a text carries several.
The accurate fix looks nothing like that. A cross-encoder reads the query and the document together, token by token, and judges relevance directly. It catches the clause-level matches that pooled vectors miss. But it runs a full transformer pass per query-document pair, so you can't run it over a whole corpus at search time. Speed on one side, precision on the other, and nothing in between.
Late interaction is the middle. You still encode documents ahead of time, so search stays cheap, but you skip the pooling step that crushes each text into one vector. Every token keeps its own vector, and the query meets the document token to token at the very last step. This page covers that scoring rule and what it costs. For the single-vector baseline, start with semantic search. For the pipeline that consumes these scores, see RAG architecture.
The one idea you need first: a dot product multiplies two vectors element by element and adds the results, so . A large value means the arrows point the same way, zero means they sit at right angles. When both vectors have length 1, the dot product equals the cosine similarity, which is why toy examples like the one below can treat them as the same thing. The full treatment is dot product.
Think of It Like This
Three mystery tastes, one shelf of jars
You are judging a cooking contest with your eyes closed. The mystery dish has three flavors you must identify: smoke, citrus, and heat. The pantry shelf holds three labeled jars.
A single-vector judge blends the whole dish into one spoonful and tastes once. Smoke plus citrus plus heat becomes one muddy impression, and you guess from that. A cross-encoder judge cooks every jar into the dish, one full remake per jar, and tastes each remake. Accurate, and dinner is ruined.
You do neither. You taste smoke against each jar and keep the best match. Then citrus against each jar, keeping its best. Then heat. Three tastings, three best matches, one summed score. The flavors never got blended, and you never cooked a full meal.
Where it stops: pantry jars taste the same no matter who their neighbors are. Token vectors do not. A transformer reads each token alongside its neighbors, so the vector for "refund" inside "late refund" differs from "refund" standing alone. The matching rule is the jars. The vectors themselves are contextual.
How It Actually Works
From one vector to one vector per token
A bi-encoder pools. It runs the tokens through the encoder, then averages them (or takes one special token) into a single vector per text. Late interaction deletes that step. A query with tokens becomes a matrix with rows, one vector per row. A document with tokens becomes a matrix with rows. In plain words: is the query length, is the document length, and each row is that token's contextual vector.
Documents are still encoded offline and stored, which is what keeps search fast. Queries are encoded when they arrive. The two sides never pass through the encoder together. That separation is the "late" in late interaction: the query and the document first meet inside the scoring rule, nowhere earlier.
MaxSim: one best match per query token, then a sum
Write for the dot product of query token with document token . In plain words, is how well query word matches document word . The MaxSim score is:
Read it inside out. For each query token, take its best match anywhere in the document (the max over ). Then add those best matches up (the sum over ). Each query token gets one vote, and each vote is the strongest evidence for that token alone. Two query tokens may point at the same document token. Nothing forbids sharing, since each max is computed independently.
This one rule explains where the method sits. Against a bi-encoder, MaxSim never pools, so clause-level evidence survives. Against a cross-encoder, the encoder still runs once per text instead of once per pair, so the document side stays precomputable. The price is storage and arithmetic: you keep vectors per document instead of one, and scoring touches dot products per candidate. That is why late interaction usually reranks a shortlist from a cheaper first stage such as BM25 or dense search, the pattern reranking describes and hybrid search generalizes. If your corpus is small, or your queries are mostly exact codes, a lexical index with a cross-encoder rerank gives you the precision for less machinery.
One assumption to carry with you: MaxSim treats a large dot product as relevance. Under domain shift, say legal or biomedical text the encoder rarely saw, that assumption cracks and the ranking drifts. The fix is the same as for any embedding model. Fine-tune the encoder on your own query-document pairs before you trust the order it produces.
Worked example
Toy vectors, every one of length 1, so each dot product is a cosine. Query tokens: , , . Document tokens: , , . (Check one length: . The rest check the same way.)
First the full similarity grid, one dot product per pair:
- : against ; against ; against . Best: 1.00 ().
- : against ; against ; against . Best: 0.80 ().
- : against ; against ; against . Best: 1.00 ().
Then the sum:
Notice that and both chose . That sharing is legal and common: a document token that answers two query tokens earns both votes. A competing document with no exact partner for would top out lower, which is exactly how the right clause wins here without any keyword rule. The diagram above draws this same grid, with the three winning cells outlined.
Code
The point of this snippet is the rule, not the encoder. It replays the worked example in plain Python: build the grid, take each row's max, sum them. If your implementation can't reproduce 2.80 here, the bug is in your scoring, not your vectors.
def dot(a: list[float], b: list[float]) -> float: return sum(x * y for x, y in zip(a, b))
query: list[list[float]] = [[1.0, 0.0], [0.0, 1.0], [0.6, 0.8]]doc: list[list[float]] = [[0.8, 0.6], [0.6, 0.8], [1.0, 0.0]]
best: list[float] = [max(dot(q, d) for d in doc) for q in query]score: float = sum(best)
print([round(v, 2) for v in best]) # -> [1.0, 0.8, 1.0]print(round(score, 2)) # -> 2.8Both prints agree with the hand computation: best matches of 1.00, 0.80 and 1.00, summing to 2.80. Keep this snippet around. It is the fastest way to check a new MaxSim implementation before you point it at real embeddings.
Watch Out For
Padding tokens win the max
Real batches pad short texts to equal length, and those padding vectors sit in the document matrix alongside real tokens. The max doesn't know that. If a padding vector happens to score well against a query token, that query token's vote goes to nothing, and short documents can outrank good ones for no visible reason.
The symptom is eerie uniformity: for many queries, several best matches come from the same trailing positions, and trimming the padding changes the ranking. The fix is to exclude padding before the max, not after. Set padded positions to a large negative value (or mask them out of the grid entirely), and log which document index each max came from while debugging. If you ever see an index past the document's true length, the mask leaked.
An index several times fatter than your dense one
One vector per token means a document with hundreds of tokens costs hundreds of vectors. Teams that swap a dense index for token-level storage without planning hit memory pressure and slow cold search, then blame the scoring rule for what is really a capacity problem.
The symptom shows up at index time: storage several times larger than the dense baseline for the same corpus, and latency that climbs with document length. The fix is the two-stage pattern. Let a cheap first stage (BM25, dense search, or their hybrid search combination) pull a shortlist, and run MaxSim only there. Compress the stored token vectors if your library supports it. Never score the whole corpus token to token per query.
The Quick Version
- Late interaction skips pooling and stores one vector per token on each side, so clause-level evidence survives to scoring time.
- MaxSim scores a document by giving each query token its best document match, then summing those best matches.
- On this page's toy grid, best matches of 1.00, 0.80 and 1.00 sum to 2.80, and two query tokens may share a single document token.
- The price is one vector per token instead of one per document, so production setups retrieve a cheap shortlist and run MaxSim only on that.
- MaxSim assumes a large dot product means relevance. Under domain shift, fine-tune the encoder on your own pairs before trusting the ranking.