Gen AI

Query Rewriting

Users write terrible search queries. Query rewriting intercepts the user's input and uses an LLM to rephrase it into something the vector database will actually understand.

Why Does This Exist?

When building a Retrieval-Augmented Generation (RAG) system, engineers spend weeks obsessing over embedding models, chunking strategies, and vector index configurations. But when the system goes to production, it fails.

Why? Because human beings write terrible search queries.

A user in a chat interface doesn't write: "What is the company policy regarding carry-over of paid time off (PTO) into the 2027 fiscal year?" They write: "pto carryover?" or even worse, "what about next year?" (referencing a previous chat message).

If you embed "what about next year?" and send it to your vector database, the database will return documents about corporate forecasting and calendar APIs, not PTO policies. The retrieval step fails, which means the generation step fails.

Query Rewriting solves this by inserting an LLM before the retrieval step. Its job is to intercept the lazy, vague, or context-dependent user input and translate it into a highly optimized query designed specifically for the vector database.

Think of It Like This

The research librarian translating for a patron

Imagine a college freshman walking up to a research librarian and saying, "I need stuff on that boat that sank. The big one."

If the librarian just types "stuff on that boat that sank the big one" into the academic database, they will get zero results.

Instead, the librarian uses their own brain (the rewriting LLM) to translate the user's intent. The librarian types: "Titanic sinking causes, marine engineering failures, 1912 maritime disasters." The database immediately returns the perfect books. Query rewriting is the automated version of that helpful librarian.

How It Actually Works

Query rewriting usually takes one of three forms, depending on the complexity of the application:

1. Contextualization (Chat History)

If you have a multi-turn chatbot, users rely heavily on pronouns and implied context.

User Turn 1: "What is the capital of France?"
Bot Turn 1: "Paris."
User Turn 2: "How many people live there?"

If you embed "How many people live there?", the vector database will fail. A rewriting LLM takes the chat history and the latest query, and generates a standalone query: "How many people live in Paris, France?" This standalone query is what gets embedded and searched.

2. Query Expansion (Synonyms and Jargon)

Sometimes the user writes a complete sentence, but they use the wrong terminology for your specific dataset. If a user asks about "firing someone", but your HR database exclusively uses the term "involuntary termination", a lexical search will fail. A rewriting LLM can be prompted with a glossary of your company's terms. It takes the user's query and expands it: "What is the process for firing someone (involuntary termination, employee dismissal)?"

3. HyDE (Hypothetical Document Embeddings)

HyDE is a fascinating and highly effective form of query rewriting. Instead of asking the LLM to rewrite the query, you ask the LLM to answer the query, even though it doesn't have the facts yet!

User asks: "How does the caching layer work in our app?"
The rewriting LLM hallucinates an answer: "The caching layer in the app uses Redis to store frequently accessed user profiles with a TTL of 60 minutes..." (This might be factually wrong, but the vocabulary and structure look exactly like a real architecture document).
You embed this hallucinated document and search the vector database.
The database returns the actual architecture document, because vectors match similar documents better than they match short queries.

Show Me the Code

This is a simple implementation of contextual query rewriting using the OpenAI API.

1import openai2
3# 1. The chat history and the new, vague user query4chat_history = [5    {"role": "user", "content": "I need to request a new laptop."},6    {"role": "assistant", "content": "You can request hardware through the IT portal."}7]8new_user_query = "how long does it take?"9
10# 2. The Prompt for the rewriting LLM11system_prompt = """12You are a search query rewriting engine.13Given a chat history and the latest user query, your job is to formulate a single, 14standalone search query that can be used to search a corporate wiki.15Do not answer the question. Only output the standalone query.16"""17
18# 3. Construct the API call19messages = [{"role": "system", "content": system_prompt}]20messages.extend(chat_history)21messages.append({"role": "user", "content": new_user_query})22
23# 4. Call a fast, cheap model (like gpt-4o-mini)24response = openai.chat.completions.create(25    model="gpt-4o-mini",26    messages=messages,27    temperature=0.028)29
30rewritten_query = response.choices[0].message.content31
32print(f"Original Query: '{new_user_query}'")33print(f"Rewritten Query: '{rewritten_query}'")34
35# -> Original Query: 'how long does it take?'36# -> Rewritten Query: 'What is the standard processing time for a new laptop hardware request through the IT portal?'37
38# We now embed `rewritten_query` and search the vector database!

Watch Out For

Latency accumulation

Query rewriting introduces an extra LLM call before the search even begins. If you use a massive, slow model like GPT-4 to rewrite queries, your users will wait 3-4 seconds just for the search to start, leading to an unacceptable UX. Query rewriting must always be done by the smallest, fastest model available (e.g., GPT-4o-mini, Claude 3 Haiku, or a local fine-tuned 8B model).

Over-correcting valid queries

If a user actually types a perfect, highly specific query (e.g., an exact error code like ERR_CONNECTION_REFUSED), a rewriting LLM might try to "help" by expanding it into a natural language sentence, which actually ruins the exact keyword match in the database. You often need a routing layer to decide if a query even needs rewriting before passing it to the LLM.

The Quick Version

Vector databases fail if the user's input is vague, uses pronouns, or lacks domain-specific vocabulary.
Query rewriting places a fast, cheap LLM between the user and the database to translate human intent into a database-optimized query.
It is essential for multi-turn chat applications to resolve pronouns (e.g., turning "it" into "the IT portal").
Advanced techniques like HyDE use the LLM to hallucinate a fake document, embedding the fake document to find the real one.