Skip to content
AI360XpertBeta

Query Rewriting

You ask a vague follow-up and the retriever hears the wrong thing. Query rewriting has a small model restate your words as a standalone question, so the search runs on what you meant.

A small rewriting model turns a vague follow-up plus chat history into a standalone query, lifting cosine to the right passage from 0.60 to 1.00 so the right chunk wins
A small rewriting model turns a vague follow-up plus chat history into a standalone query, lifting cosine to the right passage from 0.60 to 1.00 so the right chunk wins

Why Does This Exist?

You don't type the way your vector store wishes you typed. In a chat you write "pto carryover?" and then, one turn later, "what about next year?" If you embed that second message as is and search, the store sees a short string with almost no content words. It returns calendar pages and forecasting docs. The answer step then writes a confident reply from the wrong evidence.

Query rewriting fixes the input instead of blaming the index. A small fast model sits between you and the retriever. It reads your raw message plus the chat history, then writes one standalone query that carries the missing slot with it: "PTO carryover policy next year". That rewritten string is what gets embedded and searched.

You need one picture first. An embedding model turns text into a list of numbers, a vector. Cosine similarity scores the angle between two vectors: 1.00 means identical direction, 0.00 means unrelated. A RAG pipeline embeds your search string, keeps the passages with the largest cosine, and pastes them into the prompt. Rewriting earns its keep inside that pipeline, because a better search string gives a larger cosine to the right passage.

This page covers the restate then search trick. Its neighbours cover different upgrades. Hypothetical document embeddings drafts a fake answer and searches with that. Query decomposition splits one multi-part question into several searches. Hybrid search runs lexical and dense lists and fuses them. Read those for depth; here you'll see the rewrite rule, one worked rank flip, and what the extra model call costs.

The trade is plain. You pay for one more model call on every query: extra latency, extra spend, one more place where intent can get mangled. You buy back recall on exactly the queries that need it: short, pronoun heavy, history dependent. When a query is already precise, an exact error code or a pasted identifier, skip the rewrite or run it alongside the raw string. Rewriting a perfect query can only make it vaguer.

Think of It Like This

A front desk that rephrases your mumbled request

You walk up to a large office front desk and mumble, "what about next year?" The clerk does not shout your mumble down the hall to the filing room. She asks what you asked before, checks the slip from your last visit, and writes a clean new slip: "PTO carryover policy, next year, please pull the policy sheet." The filing room finds the right folder in seconds because the slip now names the thing.

The mapping holds part for part. Your mumble is the raw query qq. The earlier slips are the chat history HH. The clerk is the rewriting model. The clean slip is the standalone query q′q'. The filing room is the vector store.

It stops where office knowledge stops. A good clerk knows the office slang and your last two visits. She doesn't know the file contents, she can't fix a request when the earlier slips are missing, and she sometimes writes down the wrong year. Your rewriter is the same: it only knows what sits in the history and the prompt, and it can drift just as confidently as it can clarify.

How It Actually Works

Call the raw user string qq, where qq means the latest message only, such as "what about next year?" Call the earlier turns HH, where HH means the chat history the system keeps, such as your "pto carryover?" turn plus the assistant reply. Call an optional glossary or style hint GG, where GG means company terms you want injected, such as mapping "firing" to "involuntary termination". The rewriter is a function RewriteRewrite that returns one standalone string:

q′=Rewrite(q,H,G)q' = Rewrite(q, H, G)

In plain words: q′q' is the search string you would have typed if you had bothered. It resolves pronouns with HH ("it" becomes "the PTO policy"), adds the missing entity ("next year" joins "PTO carryover"), and swaps slang for index terms with GG. An encoder EncEnc then turns text into vectors. Write e=Enc(q)e = Enc(q) for the vague vector, e′=Enc(q′)e' = Enc(q') for the rewritten vector, and did_i for each stored passage vector. Retrieval scores cosine s(a,b)=a⋅b/(∣a∣∣b∣)s(a, b) = a \cdot b / (|a||b|), where a⋅ba \cdot b means the dot product and ∣a∣|a| means the vector length, and keeps the passages with the largest s(e′,di)s(e', d_i).

The three moves you will see

Most rewrites are one of three moves. Contextualize resolves the follow-up with HH: "how long does it take?" after a laptop thread becomes "laptop request processing time". Expand adds index terms with GG: "firing someone" becomes "involuntary termination process". Draft then search asks the model to sketch a plausible answer and searches with the sketch, which is the hypothetical document embeddings variant. Same slot, three fillings.

Worked example: the rank flip

Take two toy 2-D passage vectors. Let dpto=[3,1]d_{pto} = [3, 1] be the PTO policy passage and dcal=[1,3]d_{cal} = [1, 3] be a calendar forecasting passage. Let the vague query vector be q=[1,3]q = [1, 3] and the rewritten query vector be q′=[3,1]q' = [3, 1]. Norms first, since cosine needs them: ∣q∣=1+9=10≈3.16|q| = \sqrt{1 + 9} = \sqrt{10} \approx 3.16, ∣q′∣=9+1=10≈3.16|q'| = \sqrt{9 + 1} = \sqrt{10} \approx 3.16, and both passages also have norm 10\sqrt{10}.

Score the vague query. Against the right passage: q⋅dpto=1×3+3×1=6q \cdot d_{pto} = 1 \times 3 + 3 \times 1 = 6, so s=6/10=0.60s = 6 / 10 = 0.60. Against the wrong passage: q⋅dcal=1×1+3×3=10q \cdot d_{cal} = 1 \times 1 + 3 \times 3 = 10, so s=10/10=1.00s = 10 / 10 = 1.00. The vague string ranks the calendar page first. You feel this as "the store had the policy but returned fluff."

Now score the rewrite. Against the right passage: q′⋅dpto=3×3+1×1=10q' \cdot d_{pto} = 3 \times 3 + 1 \times 1 = 10, so s=10/10=1.00s = 10 / 10 = 1.00. Against the wrong passage: q′⋅dcal=3×1+1×3=6q' \cdot d_{cal} = 3 \times 1 + 1 \times 3 = 6, so s=6/10=0.60s = 6 / 10 = 0.60. Same two passages, flipped order. The standalone wording moved the query vector onto the policy direction, which is the whole trick. The diagram below draws these four numbers as two bars.

The method rests on one assumption: HH actually holds the missing slot and RewriteRewrite preserves intent. When the history is long and noisy, or the rewriter paraphrases an exact token into mush, the assumption fails and the flip goes the wrong way. That's why production setups log qq next to q′q' and let you diff them.

Code

The point of this snippet is the rank flip, not an API call. It replays the worked example with plain vectors so you can see the top hit change.

import math

def cosine(a: list[float], b: list[float]) -> float:    dot = sum(x * y for x, y in zip(a, b))    na = math.sqrt(sum(x * x for x in a))    nb = math.sqrt(sum(x * x for x in b))    return dot / (na * nb)

d_pto: list[float] = [3.0, 1.0]  # target policy passaged_cal: list[float] = [1.0, 3.0]  # distractor calendar passageq: list[float] = [1.0, 3.0]  # vague follow-up vectorqp: list[float] = [3.0, 1.0]  # rewritten standalone vector
print(f"{cosine(q, d_pto):.2f} vs {cosine(q, d_cal):.2f}")  # -> 0.60 vs 1.00print(f"{cosine(qp, d_pto):.2f} vs {cosine(qp, d_cal):.2f}")  # -> 1.00 vs 0.60

The first line prints the vague scores, wrong passage on top. The second line prints the rewritten scores, right passage on top. Same arithmetic as the worked example: dots of 6 and 10 over norms of 10\sqrt{10}, giving 0.60 and 1.00.

Watch Out For

Rewriting an exact code into mush

You type an exact string like E4021 or a quoted error line. The rewriter tries to help and turns it into a smooth sentence: "checkout payment failure issue." The semantic search half still works, but the literal token your index needed is gone, and the right page slides out of the shortlist.

Symptom: precise queries that used to hit suddenly miss after you switch rewriting on, while vague chat queries improve. Fix: route around the rewriter when the query carries a code-like token, digits plus letters, quotes, or an identifier. Better still, search with both strings and fuse, the way hybrid search fuses lexical and dense. And tell the rewrite prompt to preserve quoted spans verbatim.

Letting the rewriter drift on long history

With ten turns of history, the rewriter grabs a stale entity from turn two and injects it into turn ten. "What about next year?" becomes "PTO carryover policy next year for the Berlin office" when you moved to Austin three turns ago. Retrieval then returns the wrong office docs with high confidence.

Symptom: follow-ups early in a session improve, then quality sags as the session grows, and the logged q′q' names entities the user stopped mentioning. Fix: feed only the last one or two turns plus a small slot summary, keep temperature at 0, and log qq beside q′q' on every request so you can spot drift. If a rewrite adds an entity with no support in recent turns, fall back to the raw query.

The Quick Version

  • Query rewriting restates a vague message plus chat history as one standalone query, and that string is what gets embedded and searched.
  • It pays off on short follow-ups with pronouns and missing entities, where the raw cosine points at the wrong passage.
  • In the toy example the rewrite lifts cosine to the right passage from 0.60 to 1.00 and flips the top hit from the distractor to the policy.
  • It costs one extra model call per query, so use a small fast model and skip rewriting for already precise inputs like exact codes.
  • Log the raw and rewritten queries side by side, because a drifted rewrite fails with confidence.