RAG Fusion
You ask the same question a few different ways, run each version, and trust the passages that keep showing up. One awkward phrasing stops deciding everything.
Why Does This Exist?
You ask a help center how climate change affects your suppliers and get back generic fluff. The passage you needed talks about extreme weather disruption to logistics networks. It never shares a word with your phrasing except change. One query. One embedding. One miss. The model then answers from nothing, fluently.
Query rewriting tries to fix this with one better query. Sometimes that single rewrite lands. Sometimes it swaps one blind spot for another, and you are back where you started with extra latency to show for it. You cannot know which case you are in until you have already paid.
RAG Fusion stops betting on one phrasing. A small model writes three or four variants that attack from different angles, you run each one as its own search, then you fuse the ranked lists by position. A passage that places well across several variants outranks a passage that tops only one. You pay for extra searches. You stop losing to a single synonym gap.
This page covers that generate, search, and fuse loop. The base retrieve and read loop lives in RAG architecture. The same rank math applied to meaning plus exact words lives in hybrid search. The second pass that reads query and passage together lives in reranking.
You already know the base contract, so this stays short. You index chunks once, ahead of time. At query time you embed the question, pull the closest chunks, paste them into the prompt, and generate. Fusion slots between retrieval and generation: same contract, more shots on goal, one principled vote at the end.
Think of It Like This
Three neighbours and one plumber
You need a plumber fast. You ask three neighbours, but you phrase it a little differently each time. One hears leaking tap. One hears burst pipe upstairs. One hears reliable emergency callout. Each gives you a short ranked list of names.
One name shows up second on the first list, first on the second, second on the third. Another name tops the first list and appears nowhere else. You call the repeat name. That is the whole bet here: consistency across independent asks beats one loud recommendation.
Where it stops: your neighbours understand plumbing and can vouch. Reciprocal rank fusion understands nothing. It counts positions, not truth. If all three variants drift toward the same wrong reading, the wrong passage wins by the same math, and no vote fixes a pool with no good candidate in it.
How It Actually Works
One question goes in. Several ranked lists come out. One fused ordering decides what reaches the prompt. Three steps, always in this order.
1. Write the variants
Call the original user text . A fast model reads and writes variants, so the full query set holds queries in total. In plain words: is the list of questions you will actually search, is how many there are, and three or four is the usual count. Not ten. Each extra variant is another full search you pay for.
Good variants change the angle, not just the wording. For impact of climate change on business, you might get financial risks of global warming for companies, corporate adaptation to extreme weather, and supply chain disruption from climate events. Synonyms in one, a sub question in another, a broader frame in the third. Keep itself in . It is often still the best query you have.
2. Search each one on its own
Run the same retriever once per query in . Each run returns its own top few, best first. The runs do not share scores. They only share the corpus.
Scores stay separated for a reason. Two searches can return numbers on unrelated scales, and the scale shifts with the query terms. So the merge below never looks at the raw similarity. It looks only at where each document landed.
3. Fuse by rank, not score
Write for one candidate document. Write for the position of in the results for query , counting from 1. If does not appear for , that query contributes zero for . Let be a smoothing constant that softens the gap between the very top ranks, with 60 as the common default. Then the fused score for document is:
In plain words: each list contributes one fraction, large when the rank is small. Add the fractions and sort by the total, largest first. A document placed well twice collects two large fractions and passes a document that tops one list but misses the rest.
The trade is concrete. You add one model call plus retrievals to every query, so tail latency rises. Do not pay it when the question is already precise, like an exact error string or a quoted clause, or when your budget has no room for parallel search. The method assumes the variants add coverage without drifting off topic. When they drift, fusion amplifies the drift and a reranker on top cannot recover candidates no variant retrieved.
Worked example: Doc A wins by showing up
Take and three queries, each returning its top three:
- Q0, the original: 1st Doc B, 2nd Doc A, 3rd Doc C.
- Q1, variant one: 1st Doc A, 2nd Doc D, 3rd Doc E.
- Q2, variant two: 1st Doc F, 2nd Doc A, 3rd Doc G.
You only need three fractions: for a first place, for a second, for a third.
- Doc A: .
- Doc B: .
- Doc F: .
- Doc D: .
- Doc C, Doc E, Doc G: one third place each, .
Final order: Doc A first by a wide margin, Doc B and Doc F tied next, then Doc D, then the three single third places. Doc A never needed the most similar vector anywhere. It placed second, first, and second, and that consistency is exactly what the diagram above draws and the snippet below prints.
Code
The point here is the merge itself on the worked fixture: three orderings in, one fused ordering out.
def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> dict[str, float]: scores: dict[str, float] = {} for ranked_list in ranked_lists: for rank, doc_id in enumerate(ranked_list, start=1): # ranks count from 1 scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank) return dict(sorted(scores.items(), key=lambda kv: kv[1], reverse=True))
original: list[str] = ["Doc_B", "Doc_A", "Doc_C"]variant_1: list[str] = ["Doc_A", "Doc_D", "Doc_E"]variant_2: list[str] = ["Doc_F", "Doc_A", "Doc_G"]
fused = reciprocal_rank_fusion([original, variant_1, variant_2], k=60)print(list(fused)[:3]) # -> ['Doc_A', 'Doc_B', 'Doc_F']print(round(fused["Doc_A"], 5)) # -> 0.04865print(round(fused["Doc_B"], 5)) # -> 0.01639Doc A collects plus plus for 0.04865. Doc B keeps only its single first place at 0.01639, which is why the repeat name beats the one hit wonder.
Watch Out For
Variants that all say the same thing
A lazy prompt returns four copies of the original with one word swapped. You then pay for four searches that return near identical lists, and fusion just restates the single query order at four times the cost.
The symptom is specific: the fused top three match the original top three, and your eval sits flat while latency climbs. Ask for distinct angles in the prompt, keep the original in the set, cap the count at three or four, and drop near duplicate variants before you search. If fusion rarely changes the order, your variants are the bug.
Averaging similarity scores across variants
It feels tidy to average the cosine numbers from each search. Do that and whichever query happens to run hot wins every time for no reason about relevance. The fused list then mirrors one variant, and tuning a weight just moves the bias around.
Never average raw scores across different queries. Fuse by rank position with reciprocal rank fusion, exactly as above. If you need comparable numbers again, let a reranker rescore the short fused list on one shared scale before anything reaches the model.
The Quick Version
- RAG Fusion runs the same retriever on an original query plus three or four model written variants instead of betting on one phrasing.
- Reciprocal rank fusion scores each document as the sum of 1 over k plus its rank in each list, with k commonly 60, so only positions matter.
- A document placed second, first, and second across three lists scores 0.04865 and tops single list winners tied at 0.01639.
- The price is one extra model call plus one search per variant, so skip the loop for exact lookups or tight latency budgets.
- A reranker can reorder the fused shortlist but cannot recover a document that no variant retrieved.