Gen AI

Reranking

A fast retriever pulls a wide shortlist of candidates cheaply, then a slower, more accurate model rescores only that shortlist, reordering it before the top few results ever reach the language model.

A fast retriever pulls a wide shortlist of candidates cheaply, then a slower cross-encoder rescoring only that shortlist reorders it by a more accurate relevance judgment before the top few reach the model

Why Does This Exist?

Approximate nearest neighbor search over embeddings is fast precisely because it compares a query vector against document vectors that were computed independently, ahead of time — a query never influences how a document was embedded, and vice versa. That independence is what makes the whole approach scale to millions of documents, but it comes at a real cost to accuracy: a model that never actually looks at the query and document together, only compares two separately-computed vectors, can miss relevance signals that only become obvious when the two are read side by side.

A model that does read query and document together — scoring their combination directly rather than comparing two independent vectors — can be noticeably more accurate at judging true relevance. The problem is cost: that kind of joint scoring is far too slow to run against every document in a large corpus for every query. Reranking resolves this tension by using the fast, independent approach to narrow the field first, then applying the slow, accurate approach only to the small shortlist that survives.

Think of It Like This

A wide net, then a careful second look

Fishing with a wide net first catches a large, rough haul quickly — mostly the right kind of fish, some things that don't belong, but fast and covering a lot of water. Nobody inspects each fish individually while the net is still in the water; that would take far too long over the whole area being fished.

Once the net is pulled in, though, a careful second pass over just that catch — checking each one properly — is entirely practical, because the pool being inspected has shrunk from "the whole ocean" to "whatever's actually in the net." Reranking works the same way: a fast, wide-net retrieval pass narrows a huge corpus down to a manageable shortlist, and only then does a slower, more careful pass get applied — checking each candidate properly, precisely because there are now few enough of them to afford it.

How It Actually Works

Two encoder types, two different jobs

A bi-encoder — the architecture behind ordinary embedding-based retrieval — encodes the query and each document separately, each into its own fixed vector, and similarity is computed afterward between those two independent vectors. This is fast because document vectors can be precomputed once, entirely independent of any future query, and a query only needs to be compared against already-computed vectors at search time.

A cross-encoder takes the query and a specific candidate document together, as one combined input, and outputs a single relevance score for that specific pair. Because the model actually attends across both texts jointly, rather than compressing each into an independent vector first, it can pick up on interactions between the query and document that a bi-encoder's separately-computed vectors simply can't represent. The cost is that a cross-encoder score has to be computed fresh for every single query-document pair — there's no way to precompute it ahead of time the way a bi-encoder's document vectors can be, since the score genuinely depends on both inputs being present together.

Why the shortlist has to come first

Running a cross-encoder against every document in a large corpus, for every query, is computationally infeasible at real scale — the diagram's left side shows a retriever narrowing a large corpus to perhaps a hundred candidates cheaply, and only then does the cross-encoder, on the right, rescore that smaller set. The final ranking a user actually sees comes from the cross-encoder's scores on the shortlist, not the original bi-encoder ordering — the bi-encoder's only job was making sure the right candidates survived into that shortlist at all.

Why this specific upgrade tends to pay off disproportionately

Adding reranking to an existing bi-encoder-only pipeline is frequently one of the highest-return single changes available in RAG, because it directly targets an accuracy gap bi-encoders structurally can't close on their own. It's a separate lever from a better embedding model or chunking strategy, and it can be added on top of an existing pipeline without changing anything upstream of it.

Show Me the Code

A minimal two-stage pipeline: a cheap similarity-based shortlist, followed by a stand-in cross-encoder rescoring only that shortlist.

1import numpy as np2
3
4def bi_encoder_shortlist(query_vec: np.ndarray, doc_vecs: np.ndarray, k: int) -> np.ndarray:5    sims = doc_vecs @ query_vec / (np.linalg.norm(doc_vecs, axis=1) * np.linalg.norm(query_vec))6    return np.argsort(-sims)[:k]                          # cheap: precomputed doc_vecs, one pass7
8
9def cross_encoder_score(query: str, doc: str) -> float:10    """Stand-in: a real cross-encoder reads query and doc TOGETHER through one model."""11    overlap = len(set(query.split()) & set(doc.split()))12    return overlap / max(len(doc.split()), 1)13
14
15rng = np.random.default_rng(0)16doc_vecs = rng.normal(size=(50, 16))17query_vec = rng.normal(size=16)18docs = [f"document {i} about various topics" for i in range(50)]19
20shortlist_ids = bi_encoder_shortlist(query_vec, doc_vecs, k=5)      # fast, wide net21reranked = sorted(shortlist_ids, key=lambda i: cross_encoder_score("various topics", docs[i]), reverse=True)22print(list(shortlist_ids), "->", reranked)   # -> the bi-encoder's order, reordered by the cross-encoder

The cross-encoder function only ever runs against the five candidates the bi-encoder already shortlisted — never against all fifty documents — which is exactly the cost-control mechanism that makes reranking practical at real scale.

Watch Out For

Running a cross-encoder over the full corpus instead of a shortlist

A cross-encoder's accuracy is genuinely appealing, and it's tempting to wonder why not just use it for the entire search instead of a two-stage pipeline. The answer is pure cost: a cross-encoder score requires a full model forward pass for every single query-document pair, with no way to precompute anything ahead of time, which makes it computationally infeasible against a corpus of any real size. The two-stage design isn't a compromise chosen out of caution — it's the only practical way to get cross-encoder-level accuracy at all, on anything beyond a tiny corpus.

Setting the shortlist size too small for the reranker to help

If the initial bi-encoder shortlist is too narrow, the true best result may not even be present among the candidates the reranker gets to see, and no amount of accurate rescoring recovers a document that was never shortlisted in the first place. The shortlist size is a real tuning parameter — wide enough that the reranker has genuine room to improve on the initial ordering, narrow enough that the reranking stage's added latency stays acceptable.

The Quick Version

Bi-encoders embed queries and documents independently, which is fast but misses interactions only visible when both are read together.
Cross-encoders score a query and a specific document jointly, more accurately, but can't be precomputed and are too slow to run against a full corpus.
Reranking uses a fast bi-encoder to shortlist candidates, then a slower cross-encoder to rescore only that shortlist.
The final ranking a system uses comes from the reranker's scores, not the original bi-encoder ordering that produced the shortlist.
Adding reranking to an existing retrieval pipeline is frequently one of the highest-return upgrades available, since it targets a gap bi-encoders can't close alone.