Skip to content
AI360Xpert
Gen AI

Query Decomposition

When a user asks a multi-part question, vector databases fail to find a single document that answers all parts. Decomposition uses an LLM to split the complex question into several simple questions that are searched in parallel.

Decomposition breaks a complex user query into smaller sub-queries, executes them in parallel against the database, and concatenates all the results for the final LLM to synthesize.
Decomposition breaks a complex user query into smaller sub-queries, executes them in parallel against the database, and concatenates all the results for the final LLM to synthesize.

Why Does This Exist?

Vector databases are incredibly good at finding the answer to a single, focused question. But humans rarely ask single, focused questions. We ask complex, multi-part questions that require aggregating information from entirely different sources.

Imagine a user asks: "Did Q3 revenue grow faster in Europe or North America?"

If you embed this exact sentence and search your vector database, you are looking for a single chunk of text that happens to contain both the Europe Q3 revenue and the North America Q3 revenue side-by-side. If the Europe data is in a PDF from the Paris office, and the North America data is in an Excel sheet from the New York office, the search will fail. The database cannot do math, and it cannot dynamically stitch two documents together during the search phase.

Query Decomposition solves this by putting a "Planning LLM" in front of the database. When a complex query arrives, the planner breaks it down into multiple, simple sub-queries. The system searches for all of these sub-queries in parallel, gathers all the disparate documents, and hands the massive pile of context to the final LLM to figure out the answer.

Think of It Like This

A CEO asking for a comprehensive report

Imagine a CEO asks their Chief of Staff: "How does our new product's pricing compare to Competitor A and Competitor B?"

The Chief of Staff doesn't walk into the archives and try to find a single folder labeled "Our pricing vs Competitor A vs Competitor B." That folder doesn't exist.

Instead, the Chief of Staff decomposes the task and assigns it to three interns:

  • Intern 1: "Go find our new product's pricing sheet."
  • Intern 2: "Go find Competitor A's pricing sheet."
  • Intern 3: "Go find Competitor B's pricing sheet."

The interns execute these simple searches in parallel. They bring all three documents back to the Chief of Staff, who reads them all and synthesizes the final report for the CEO.

How It Actually Works

1. The Planning Phase

When a user submits a query, it is first sent to a fast Planning LLM (like GPT-4o-mini). The prompt instructs the model to analyze the user's input and determine if it requires multi-hop reasoning or multiple distinct facts.

If the query is complex, the LLM outputs a JSON array of sub-queries. User Input: "What is the capital of the country where Albert Einstein was born?" LLM Output:

  1. "Where was Albert Einstein born?"
  2. "What is the capital of [Country]?" (Note: Some systems do this sequentially, where query 2 waits for query 1's answer. This is called step-back prompting or chain-of-thought retrieval. More commonly, systems just search for both "Albert Einstein birthplace" and "Germany capital" if they can guess the entity).

2. Parallel Retrieval

The orchestrator (e.g., LangChain, LlamaIndex, or custom Python) takes the list of sub-queries and executes a vector search for each of them concurrently.

If the planner generated 3 sub-queries, and your database returns the top 3 documents per query, you will retrieve a total of 9 documents.

3. Synthesis

The 9 documents are concatenated together into a single massive context block. This context block, along with the original complex query, is passed to the final, powerful Generation LLM (like GPT-4). Because the LLM now has all the scattered puzzle pieces in its context window, it can synthesize the final answer.

Show Me the Code

This is a conceptual implementation of how a planning LLM breaks down a query and triggers parallel searches.

import openaiimport concurrent.futuresimport json
def generate_sub_queries(complex_query):    # Prompt the LLM to act as a query planner    prompt = f"""    You are an AI assistant. The user will ask a complex question.    Break it down into 2-4 simple, focused search queries required to answer the question.    Output ONLY a JSON array of strings.        User Question: {complex_query}    """        response = openai.chat.completions.create(        model="gpt-4o-mini",        messages=[{"role": "user", "content": prompt}],        response_format={"type": "json_object"}    )        # In a real app, ensure strict JSON parsing    return json.loads(response.choices[0].message.content).get("queries", [])
def mock_vector_search(query):    # Simulates calling Pinecone or Qdrant    print(f"  -> Executing DB search for: '{query}'")    return f"[Doc matching {query}]"
def retrieve_and_synthesize(complex_query):    print(f"Original Query: {complex_query}")        # 1. Decompose    sub_queries = generate_sub_queries(complex_query)    print(f"Decomposed into: {sub_queries}")        # 2. Parallel Retrieval    all_documents = []    with concurrent.futures.ThreadPoolExecutor() as executor:        results = executor.map(mock_vector_search, sub_queries)        for doc in results:            all_documents.append(doc)                # 3. Final Synthesis (omitted for brevity)    print(f"Documents gathered for final LLM: {all_documents}")    return "Final Answer generated here..."
# Executeretrieve_and_synthesize("Did the iPhone 15 or the Galaxy S24 have better battery life in tests?")
# -> Original Query: Did the iPhone 15 or the Galaxy S24 have better battery life in tests?# -> Decomposed into: ['iPhone 15 battery life test results', 'Galaxy S24 battery life test results']# ->   Executing DB search for: 'iPhone 15 battery life test results'# ->   Executing DB search for: 'Galaxy S24 battery life test results'# -> Documents gathered for final LLM: ['[Doc matching iPhone 15 battery life test results]', '[Doc matching Galaxy S24 battery life test results]']

Watch Out For

Context Window Exhaustion

Query decomposition acts as a multiplier on your retrieval volume. If your standard retrieval returns 5 chunks, and the planner breaks the user's query into 4 sub-queries, you are now pulling 20 chunks out of the database. If your chunks are large (e.g., from Parent-Child Chunking), you will instantly blow past the token limit of your generation LLM. You must strictly limit k (documents returned per sub-query) when using decomposition.

The Quick Version

  • Vector databases excel at answering focused questions but fail at answering complex, comparative, or multi-part questions because the required facts rarely live in the same document.
  • Query decomposition places a fast "Planning LLM" in front of the database to break the user's complex question into a list of simple sub-questions.
  • The system searches the vector database for all the sub-questions in parallel.
  • All the retrieved documents are concatenated and fed to the final LLM, giving it all the disparate facts it needs to synthesize the final answer.
  • Read Query Rewriting for a simpler precursor to decomposition that just cleans up the query without splitting it.
  • Read Planning and Reasoning to see how this concept evolves into fully autonomous agents.
  • Read RAG Architecture to understand how the orchestrator manages these multiple LLM calls.

Related concepts