Reranking

Retrieve cheaply and widely first, then read the shortlist carefully. A fast search pulls a hundred candidates and a slower judge reorders just those, so only the best few reach the model.

A cheap bi-encoder shortlists five passages by cosine, then a cross-encoder that reads query and passage together rescores them so D2 at 0.92 beats D1 at 0.31 and the prompt keeps D2 and D3

Why Does This Exist?

You ask a question and the retriever has milliseconds to search millions of passages. Embeddings make that possible: each passage becomes a vector once, ahead of time, and your query becomes one vector at search time, so matching is arithmetic over precomputed lists. Approximate nearest neighbor search makes that arithmetic fast enough to ship.

Speed has a price. A bi-encoder scores your query and each passage separately, then compares the two vectors. It never reads your words next to the passage's words, so a paraphrased answer with none of your terms can look distant when it is exactly right. Short sentences, negations, and fine conditions suffer first.

A cross-encoder is the better judge. It reads the query and one passage together through a single model and returns one relevance number for that pair. Attention runs across both texts, so the paraphrase finally gets credit. But each score costs a full forward pass, nothing comes precomputed, and you pay it per pair. Across millions of passages on every query, that bill never clears.

So you split the work. Pull a wide shortlist with the cheap scorer, rescore only those survivors with the careful one, and hand the model the winners in the new order. That split is reranking.

This page is about the rescoring step. Casting a wider shortlist from two signals is hybrid search; the pipeline that consumes the winners is RAG architecture; the pass that trades a little relevance for variety afterward is maximal marginal relevance.

The one idea you need first: cosine similarity is a number from minus one to one that says how alike two vectors point, with one meaning same direction. Bi-encoder retrieval ranks by it. The full picture lives in embeddings; that one sentence carries you through the rest here.

Think of It Like This

Hiring for one role

You have 200 applications for one job. You don't interview 200 people. A quick keyword scan pulls 20 resumes that look plausible in seconds each, then you sit down with 5 finalists for half an hour each and make offers to 2. The scan is cheap and wide; the interview is slow and careful; the offers go to interview winners, not scan winners.

Reranking maps onto that directly. The keyword scan is the bi-encoder cosine. The interview is the cross-encoder reading query and passage together. The 2 offers are the top passages that join the prompt.

Where it stops: your interview can surface things no resume contained, like how someone thinks aloud. A cross-encoder can't ask follow-ups. It re-reads the same passage more carefully, and its verdict is only as good as the queries and passages it trained on.

How It Actually Works

Here is the shape of it: index once, retrieve cheaply, rescore narrowly, and keep a handful. Two scorers, one handoff, and the final order comes from the second scorer alone.

Two scorers, two different views

Call your question $q$ (plain text, what you typed) and each candidate passage $d_i$ (the $i$ -th chunk in the store, where $i$ just numbers them).

A bi-encoder maps $q$ and each $d_i$ to vectors separately, then scores the pair with cosine. Write that $s_{\mathrm{bi}}(q, d_i)$ (the bi-encoder score, from minus one to one). Because document vectors don't depend on your query, you compute them once at index time and reuse them forever. Fast. Forgetful.

A cross-encoder skips vectors entirely. It feeds the pair $(q, d_i)$ (your question glued to one passage) through a single transformer and returns $s_{\mathrm{ce}}(q, d_i)$ (the cross-encoder relevance, from zero to one). No side can be precomputed, because the score exists only when both texts are present together. Slow. Perceptive.

The four steps

Say the store holds millions of chunks. Pick two depths: $k_1$ (how many survive the cheap pass, picture a hundred) and $k_2$ (how many reach the prompt, picture a handful like five). In plain words, $k_1$ sets your rescoring bill, since cost grows linearly with it, and $k_2$ sets the model's reading load.

First, index. Encode every chunk once with the bi-encoder and store the vectors. This happens before any query arrives, which is the whole reason the first pass is cheap.

Next, retrieve. Encode $q$ , score it against the store with $s_{\mathrm{bi}}$ , and keep the top $k_1$ . Order here doesn't matter much. Survival matters: anything outside this shortlist is gone for good.

Then, rescore. Run the cross-encoder over just those $k_1$ pairs and replace every score with $s_{\mathrm{ce}}$ . This is the only ranking the reader sees.

Finally, keep the top $k_2$ by $s_{\mathrm{ce}}$ and join them to the prompt in that order, best first. If your shortlist came from two signals, rerank the fused list from hybrid search the same way.

The trade is explicit: a deeper $k_1$ buys the reranker more chances to find truth and charges one forward pass per extra candidate. Skip reranking when the corpus is tiny enough to score directly, when latency leaves no room for a second pass, or when your misses come from recall rather than ordering. The method assumes two things: the shortlist contains the answer, and the cross-encoder behaves on your kind of text. When the first fails, no rescoring recovers the miss. When the second fails, usually on unfamiliar phrasing or a new domain, the careful judge misfires with confidence.

Worked example: five passages, one flip

Take a toy query about refunds for a cancelled flight and five candidates. The bi-encoder's cosine values arrive in this order:

D1 at 0.81, D2 at 0.77, D3 at 0.74, D4 at 0.69, D5 at 0.62. So the shortlist order is D1, D2, D3, D4, D5. D1 leads because it shares the words flight and booking, though it answers booking, not refunds.

Set $k_1 = 5$ (keep all five for the toy) and rescore each pair jointly. The cross-encoder returns D1 at 0.31, D2 at 0.92, D3 at 0.68, D4 at 0.45, D5 at 0.12. Sorting those gives D2, D3, D4, D1, D5. Check the chain: 0.92 beats 0.68, which beats 0.45, which beats 0.31, which beats 0.12.

With $k_2 = 2$ , the prompt keeps D2 and D3. D1 falls from first to fourth, which is the whole lesson: separate vectors liked the wording, joint reading liked the answer. The diagram above draws these same ten numbers, and the snippet below sorts them.

Code

The point of this snippet is the handoff, not the models. It sorts one fixed shortlist twice: once by the cheap score, once by the joint score, and keeps two. Run it and you'll see D1 lead first, then fall.

1def top_k(scores: dict[str, float], k: int) -> list[str]:2    # order by the score you pass in; the caller decides which score counts3    return sorted(scores, key=lambda d: scores[d], reverse=True)[:k]4
5
6bi_scores: dict[str, float] = {"D1": 0.81, "D2": 0.77, "D3": 0.74, "D4": 0.69, "D5": 0.62}7ce_scores: dict[str, float] = {"D1": 0.31, "D2": 0.92, "D3": 0.68, "D4": 0.45, "D5": 0.12}8
9print(top_k(bi_scores, 5))  # -> ['D1', 'D2', 'D3', 'D4', 'D5']10print(top_k(ce_scores, 2))  # -> ['D2', 'D3']

The prints match the hand sort above: the cheap order starts D1, D2, D3, D4, D5, and the joint order keeps D2 and D3. Change $k_2$ to 3 and the third kept passage is D4 at 0.45.

Watch Out For

A shortlist that missed cannot be rescored into a hit

Your reranker looks broken: you tune it for a week and the answers never improve. Check recall at $k_1$ before you touch the reranker again. If the true passage isn't inside the shortlist, the cross-encoder never sees it, and no threshold or prompt trick recovers what was never scored.

Widen $k_1$ until recall at that depth stops climbing, then fix the first stage itself with hybrid search or query rewriting. Only then does tuning the reranker pay.

Averaging scores from two different scales

Two numbers that both claim to be relevance still can't be averaged when they live on different scales. Cosine runs from minus one to one and shifts with each query; the cross-encoder returns zero to one per pair. Blend them raw and a strong joint pick like D2 at 0.92 can sink beneath a wording match the first stage loved.

Rank by the joint score alone for the final order. If you must combine signals, fuse by rank position and let the cross-encoder cast the last vote on the shortlist.

The Quick Version

Reranking splits search into a cheap shortlist and a careful rescore: a bi-encoder pulls candidates with precomputed vectors, and a cross-encoder reorders only those by reading query and passage together.
The final order comes from the cross-encoder alone. The bi-encoder decides survival into the shortlist, never the ranking the model reads.
In the toy query, D2 at 0.92 beats D1 at 0.31 after rescoring, so a prompt that keeps two takes D2 and D3 while the wording match D1 falls to fourth.
Cost grows with shortlist depth because each extra candidate costs one joint forward pass. Depth is the latency knob.
A reranker cannot recover a passage the shortlist missed. When gains stall, measure recall at shortlist depth before tuning the reranker.
Never average cosine scores with cross-encoder scores. The two live on different scales, so rank by the joint score or fuse by rank.