Skip to content
AI360XpertBeta

RAG Fusion

You ask the same question a few different ways, run each version, and trust the passages that keep showing up. One awkward phrasing stops deciding everything.

One question becomes three searches, and reciprocal rank fusion at k=60 puts Doc A first at 0.04865 because it places second, first, and second across the three lists
One question becomes three searches, and reciprocal rank fusion at k=60 puts Doc A first at 0.04865 because it places second, first, and second across the three lists

Why Does This Exist?

You ask a help center how climate change affects your suppliers and get back generic fluff. The passage you needed talks about extreme weather disruption to logistics networks. It never shares a word with your phrasing except change. One query. One embedding. One miss. The model then answers from nothing, fluently.

Query rewriting tries to fix this with one better query. Sometimes that single rewrite lands. Sometimes it swaps one blind spot for another, and you are back where you started with extra latency to show for it. You cannot know which case you are in until you have already paid.

RAG Fusion stops betting on one phrasing. A small model writes three or four variants that attack from different angles, you run each one as its own search, then you fuse the ranked lists by position. A passage that places well across several variants outranks a passage that tops only one. You pay for extra searches. You stop losing to a single synonym gap.

This page covers that generate, search, and fuse loop. The base retrieve and read loop lives in RAG architecture. The same rank math applied to meaning plus exact words lives in hybrid search. The second pass that reads query and passage together lives in reranking.

You already know the base contract, so this stays short. You index chunks once, ahead of time. At query time you embed the question, pull the closest chunks, paste them into the prompt, and generate. Fusion slots between retrieval and generation: same contract, more shots on goal, one principled vote at the end.

Think of It Like This

Three neighbours and one plumber

You need a plumber fast. You ask three neighbours, but you phrase it a little differently each time. One hears leaking tap. One hears burst pipe upstairs. One hears reliable emergency callout. Each gives you a short ranked list of names.

One name shows up second on the first list, first on the second, second on the third. Another name tops the first list and appears nowhere else. You call the repeat name. That is the whole bet here: consistency across independent asks beats one loud recommendation.

Where it stops: your neighbours understand plumbing and can vouch. Reciprocal rank fusion understands nothing. It counts positions, not truth. If all three variants drift toward the same wrong reading, the wrong passage wins by the same math, and no vote fixes a pool with no good candidate in it.

How It Actually Works

One question goes in. Several ranked lists come out. One fused ordering decides what reaches the prompt. Three steps, always in this order.

1. Write the variants

Call the original user text q0q_0. A fast model reads q0q_0 and writes N−1N-1 variants, so the full query set QQ holds NN queries in total. In plain words: QQ is the list of questions you will actually search, NN is how many there are, and three or four is the usual count. Not ten. Each extra variant is another full search you pay for.

Good variants change the angle, not just the wording. For q0=q_0 = impact of climate change on business, you might get financial risks of global warming for companies, corporate adaptation to extreme weather, and supply chain disruption from climate events. Synonyms in one, a sub question in another, a broader frame in the third. Keep q0q_0 itself in QQ. It is often still the best query you have.

2. Search each one on its own

Run the same retriever once per query in QQ. Each run returns its own top few, best first. The runs do not share scores. They only share the corpus.

Scores stay separated for a reason. Two searches can return numbers on unrelated scales, and the scale shifts with the query terms. So the merge below never looks at the raw similarity. It looks only at where each document landed.

3. Fuse by rank, not score

Write DD for one candidate document. Write rankq(D)\text{rank}_q(D) for the position of DD in the results for query qq, counting from 1. If DD does not appear for qq, that query contributes zero for DD. Let kk be a smoothing constant that softens the gap between the very top ranks, with 60 as the common default. Then the fused score RRF(D)\text{RRF}(D) for document DD is:

RRF(D)=∑q∈Q1k+rankq(D)\text{RRF}(D) = \sum_{q \in Q} \frac{1}{k + \text{rank}_q(D)}

In plain words: each list contributes one fraction, large when the rank is small. Add the fractions and sort by the total, largest first. A document placed well twice collects two large fractions and passes a document that tops one list but misses the rest.

The trade is concrete. You add one model call plus NN retrievals to every query, so tail latency rises. Do not pay it when the question is already precise, like an exact error string or a quoted clause, or when your budget has no room for parallel search. The method assumes the variants add coverage without drifting off topic. When they drift, fusion amplifies the drift and a reranker on top cannot recover candidates no variant retrieved.

Worked example: Doc A wins by showing up

Take k=60k = 60 and three queries, each returning its top three:

  • Q0, the original: 1st Doc B, 2nd Doc A, 3rd Doc C.
  • Q1, variant one: 1st Doc A, 2nd Doc D, 3rd Doc E.
  • Q2, variant two: 1st Doc F, 2nd Doc A, 3rd Doc G.

You only need three fractions: 1/61≈0.016391/61 \approx 0.01639 for a first place, 1/62≈0.016131/62 \approx 0.01613 for a second, 1/63≈0.015871/63 \approx 0.01587 for a third.

  • Doc A: 1/62+1/61+1/62≈0.01613+0.01639+0.01613=0.048651/62 + 1/61 + 1/62 \approx 0.01613 + 0.01639 + 0.01613 = 0.04865.
  • Doc B: 1/61+0+0=0.016391/61 + 0 + 0 = 0.01639.
  • Doc F: 0+0+1/61=0.016390 + 0 + 1/61 = 0.01639.
  • Doc D: 0+1/62+0=0.016130 + 1/62 + 0 = 0.01613.
  • Doc C, Doc E, Doc G: one third place each, 1/63≈0.015871/63 \approx 0.01587.

Final order: Doc A first by a wide margin, Doc B and Doc F tied next, then Doc D, then the three single third places. Doc A never needed the most similar vector anywhere. It placed second, first, and second, and that consistency is exactly what the diagram above draws and the snippet below prints.

Code

The point here is the merge itself on the worked fixture: three orderings in, one fused ordering out.

def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> dict[str, float]:    scores: dict[str, float] = {}    for ranked_list in ranked_lists:        for rank, doc_id in enumerate(ranked_list, start=1):  # ranks count from 1            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)    return dict(sorted(scores.items(), key=lambda kv: kv[1], reverse=True))

original: list[str] = ["Doc_B", "Doc_A", "Doc_C"]variant_1: list[str] = ["Doc_A", "Doc_D", "Doc_E"]variant_2: list[str] = ["Doc_F", "Doc_A", "Doc_G"]
fused = reciprocal_rank_fusion([original, variant_1, variant_2], k=60)print(list(fused)[:3])  # -> ['Doc_A', 'Doc_B', 'Doc_F']print(round(fused["Doc_A"], 5))  # -> 0.04865print(round(fused["Doc_B"], 5))  # -> 0.01639

Doc A collects 1/621/62 plus 1/611/61 plus 1/621/62 for 0.04865. Doc B keeps only its single first place at 0.01639, which is why the repeat name beats the one hit wonder.

Watch Out For

Variants that all say the same thing

A lazy prompt returns four copies of the original with one word swapped. You then pay for four searches that return near identical lists, and fusion just restates the single query order at four times the cost.

The symptom is specific: the fused top three match the original top three, and your eval sits flat while latency climbs. Ask for distinct angles in the prompt, keep the original in the set, cap the count at three or four, and drop near duplicate variants before you search. If fusion rarely changes the order, your variants are the bug.

Averaging similarity scores across variants

It feels tidy to average the cosine numbers from each search. Do that and whichever query happens to run hot wins every time for no reason about relevance. The fused list then mirrors one variant, and tuning a weight just moves the bias around.

Never average raw scores across different queries. Fuse by rank position with reciprocal rank fusion, exactly as above. If you need comparable numbers again, let a reranker rescore the short fused list on one shared scale before anything reaches the model.

The Quick Version

  • RAG Fusion runs the same retriever on an original query plus three or four model written variants instead of betting on one phrasing.
  • Reciprocal rank fusion scores each document as the sum of 1 over k plus its rank in each list, with k commonly 60, so only positions matter.
  • A document placed second, first, and second across three lists scores 0.04865 and tops single list winners tied at 0.01639.
  • The price is one extra model call plus one search per variant, so skip the loop for exact lookups or tight latency budgets.
  • A reranker can reorder the fused shortlist but cannot recover a document that no variant retrieved.