Skip to content
AI360XpertBeta

Query Decomposition

One tangled question rarely matches one stored passage. Query decomposition splits it into small focused searches that run side by side, then combines what they find.

Query decomposition splits one tangled question into two focused sub-queries retrieved in parallel, then fuses the hits so one answer can compare them
Query decomposition splits one tangled question into two focused sub-queries retrieved in parallel, then fuses the hits so one answer can compare them

Why Does This Exist?

You ask one question with two halves: "Did Q3 revenue grow faster in Europe or North America?" A vector store hears the blended meaning of the whole sentence and returns whichever single chunk sits nearest to it in embedding space. That is often a chunk that mentions both regions in passing, not the two precise figures you need. The Europe number lives in one chunk, the North America number in another, and no single passage holds both.

So the search fails before the model ever gets to think. It receives almost-right context and writes a confident answer from it. That's the failure this technique removes.

Query decomposition puts a small planning step in front of retrieval. A model reads your question and splits off one focused search per fact, then the system runs those searches side by side. Each arm retrieves its own hits, the hits get fused and trimmed, and only then does a model write the answer. This page covers the split itself: when to do it, how the arms merge, and what it costs. If your problem is one vague query rather than several missing facts, start with query rewriting instead. If you already split and want the sibling trick that asks the same question several ways, read RAG Fusion. The pipeline all of this plugs into is RAG architecture.

Think of It Like This

One shopping list, two aisles, two shoppers

You send two friends to the store with one list: apples and motor oil. You don't ask each of them to find everything. You split the list by aisle. One heads to produce, the other to auto care, and they shop at the same time. At checkout you combine both baskets and compare prices.

Your tangled question is the list. Each sub-query is one aisle. Parallel retrieval is the two shoppers. The final answer is you at checkout, holding both receipts and deciding which region grew faster.

Where it stops: groceries never contradict each other, but retrieved chunks do. Two passages can state different numbers, and the split itself can drop the word that mattered, like "faster" or "not." The checkout step still needs judgment, which is why the original question travels with the sub-queries to the final model.

How It Actually Works

Call the user's original question QQ. The planner splits it into nn sub-queries, written q1,q2,…,qnq_1, q_2, \ldots, q_n, where nn is just the count of arms. Each arm retrieves its top kk chunks, where kk is the hits kept per arm. The fused set DD is everything retrieved, minus duplicates. Those four symbols carry the whole mechanism.

Decide: split or keep whole

The planner reads QQ and either returns a list of sub-queries or returns QQ unchanged. That second outcome matters. Single-fact questions should never reach the fan-out below.

Ask whether the arms can run on their own. "Europe Q3 growth" and "North America Q3 growth" are independent, so they run in parallel. "The capital of the country where Einstein was born" is dependent: the second arm needs the first arm's answer before it can even be phrased. Dependent questions run in sequence, one retrieval feeding the next, or skip decomposition entirely.

Use a small fast model for this call and save your strong model for synthesis. And keep QQ itself as one more retrieval arm. The comparison wording lives in QQ, in no arm, and you'll want it back at the end.

Retrieve each piece on its own

Each qiq_i is embedded separately and searched on its own, which is the entire point. A focused query matches its fact cleanly, where the blended QQ matched nothing well. Take the top kk per arm with a small kk, union the lists into DD, drop duplicates by chunk id, and trim with reranking until the set fits your token budget. Small budgets win here. Two arms at top 2 give 4 hits, and that's usually plenty.

Worked example

Take QQ as "Did Q3 revenue grow faster in Europe or North America?" The planner sets n=2n = 2: q1q_1 is "Europe Q3 revenue growth" and q2q_2 is "North America Q3 revenue growth." With k=2k = 2 you pull 2×2=42 \times 2 = 4 hits:

  • E1: Europe Q3 revenue 8.2m, up on Q2 revenue of 7.7m. The rise is 0.5/7.70.5 / 7.7, about 6.5 percent, so call it about 6 percent.
  • T1: global Q3 total 17.3m across both regions. This chunk matches both arms, so it arrives twice.
  • N1: North America Q3 revenue 9.1m, up on Q2 revenue of 8.75m. The rise is 0.35/8.750.35 / 8.75, exactly 4 percent.

Fusing gives 4 hits, one twice, hence 3 unique chunks: E1, T1, N1. At about 45 tokens a chunk, that is 3×45=1353 \times 45 = 135 tokens of context. The final model sees QQ plus DD, compares about 6 percent against 4 percent, and answers Europe by about 2 points. The total is the cross-check: 8.2+9.1=17.38.2 + 9.1 = 17.3, which matches T1, so neither arm misfired.

Fuse, then answer once

The final prompt holds the original QQ plus the fused DD, never the arms alone. Name the trade before you pay it. Decomposition costs a planner call, nn searches instead of one, and a wider context that slows the answer and risks burying the best chunk in the middle. What it buys is recall on questions no single query can cover.

So don't use it by default. A single-fact question gets query rewriting, which is one call and one search. A sprawling question on a tiny budget gets a smaller kk first. And check each arm's hits before synthesis: decomposition multiplies good retrieval, but it multiplies silence too. If an arm comes back empty, its fact is missing from DD, and the final answer will guess. That is the assumption underneath the whole trick, that every arm is answerable from the store, and an empty arm is how you watch it fail.

Code

The snippet below is the bookkeeping at the center of the diagram: fuse per-arm hit lists, drop the duplicate, check the token budget, and record the verdict. It uses only the standard library, and every # -> line is the real output.

from dataclasses import dataclass

@dataclass(frozen=True)class Chunk:    cid: str    text: str

def fuse(ranked_lists: list[list[Chunk]]) -> list[Chunk]:    seen: set[str] = set()    fused: list[Chunk] = []    for hits in ranked_lists:        for chunk in hits:            if chunk.cid not in seen:                seen.add(chunk.cid)                fused.append(chunk)    return fused

q1_hits = [    Chunk("E1", "Europe Q3 revenue 8.2m, up about 6% on Q2"),    Chunk("T1", "Global Q3 total 17.3m across both regions"),]q2_hits = [    Chunk("N1", "North America Q3 revenue 9.1m, up 4% on Q2"),    Chunk("T1", "Global Q3 total 17.3m across both regions"),]
fused = fuse([q1_hits, q2_hits])print([c.cid for c in fused])  # -> ['E1', 'T1', 'N1']print(len(fused) * 45)  # -> 135print(6 > 4)  # -> True

Three unique ids, 135 tokens of context, Europe ahead. That matches the worked example number for number, including T1 arriving twice and surviving once.

Watch Out For

The split drops the word that mattered

Each arm returns a good chunk, yet the answer compares the wrong thing. It reports the bigger revenue instead of the faster growth, or it ignores a "not" or an "except." The symptom is correct facts with a wrong verdict.

The cause sits in the split. The relation lives in QQ, in words like "faster," and no single arm carries it. Fix it twice: retrieve QQ itself as an extra arm, and paste QQ word for word into the synthesis prompt next to DD. If a question turns on comparison or negation and you cannot test the split, keep it whole.

Fan-out floods the context window

Answers get vaguer after you add decomposition, and latency climbs. The symptom is easy to misread as a weak generator, so people upgrade the final model first.

Count instead. Three arms at top 5 pull 15 chunks, and with big chunking strategies each chunk is heavy. The fix is arithmetic: cut to k=2k = 2 per arm, dedupe by id, rerank down to a fixed budget of three to five chunks, and skip decomposition for questions one query can answer. Width is not depth.

The Quick Version

  • Query decomposition splits one multi-part question into focused sub-queries that are retrieved in parallel and fused before answering.
  • One embedding of a two-fact question usually matches a chunk that mentions both facts weakly instead of the two chunks that state them.
  • Name the pieces: QQ is the original question, qiq_i are the sub-queries, kk is the hits kept per arm, and DD is the fused set after dedupe.
  • Two arms at top 2 give 4 hits; one duplicate leaves 3 unique chunks at about 45 tokens each, about 135 tokens of context.
  • Decomposition costs a planner call, parallel searches, and a wider context, so single-fact questions belong with query rewriting instead.
  • The original question stays in the final prompt, because the split drops comparison words like "faster" that no single arm carries.