Retrieval Augmented Agents

A retrieval-augmented agent looks things up before it answers: it searches its sources, keeps only the passages that pass a check, and writes from those.

A retrieval-augmented agent answers one revenue question with two searches and a 0.70 score gate, keeping 2 of 3 chunks and citing the survivors.

Why Does This Exist?

A model answers from frozen weights. Ask why Q4 revenue dropped and it writes a smooth paragraph with no figures, because training never saw your sales table. When that paragraph reaches a customer or a report, fluent and wrong is worse than silent.

A retrieval-augmented agent closes that gap by searching before writing. It treats the model as a reader that picks sources, reads results, and stops when the evidence answers the question. Knowledge updates without retraining: you edit the index, not the weights.

This page covers the agent-run loop: which source to ask, what survives the gate, and when to stop. The single search-then-write baseline is a fixed pipeline, not a loop. How one call is built lives in tool-use agents. Where the passages live long term lives in agent memory types. When the job needs several steps in order, read task decomposition.

Basics first: a tool call is a named request with typed arguments, such as search_docs(query="Q4 marketing change"). The model writes the request. Your code runs it and pastes the result back as text. If that round trip is new, read tool-use agents first.

Think of It Like This

A reference librarian, not a vending machine

A vending machine takes your coins, drops one item, and is done. A reference librarian works differently. You hand over a slip asking why the shop earned less last quarter. She walks to the ledger room, copies two figures, reads them, and sees she still lacks the reason. So she walks to the archive room, pulls three folders, keeps the two with matching dates, returns the third, and writes the answer with both slips pinned to it.

Map it point for point. Your question is the slip. Each room is a source. Each walk is one lookup. The pause to read before the next walk is what lets her change plans. The kept folders are the gated chunks. The pinned slips are the citations.

Where it stops: a librarian understands your words. The model matches patterns, so it can misread a slip or request a folder that was never listed. And a librarian goes home after a fair try, while the model keeps walking unless code caps its steps.

How It Actually Works

Name the pieces once. The step counter $t$ starts at $0$ and counts lookups. The context $s_t$ is everything seen so far: your question plus every result returned up to step $t$ . The thought $T_t$ is a private note about what is still missing. The action $a_t$ is one chosen lookup: which source plus the exact query. The observation $o_t$ is what that source returns. The cap $N_{\max}$ is a hard limit on steps. FINAL is a special action meaning answer now instead of looking again.

The loop in four moves

Thought. Read $s_t$ and name the missing piece. "I have no numbers yet" or "I have numbers but no reason."
Action. Emit exactly one $a_t$ : one source, one query. One per step keeps blame clear, so a wrong answer traces to the $o_t$ that fed it.
Observation. Real code runs the lookup and appends $o_t$ to make $s_{t+1}$ . The model never fetches directly. It asks, and code fetches.
Stop check. If $a_t$ is FINAL, or $t$ has reached $N_{\max}$ , write the answer from $s_t$ with each claim tied to the result that supplied it. Otherwise think again.

Worked example: one question, two lookups

Question: "Why did Q4 revenue drop vs Q3?" Set the cap $N_{\max} = 3$ .

Step 0 goes to the sales table. Thought: "I need figures before stories." Action: ask for Q3 and Q4 revenue. Observation: Q3 $= 15$ , Q4 $= 12$ , in millions of dollars. Check the math yourself: the drop is $15 - 12 = 3$ , and the share is $3 / 15 = 0.20$ , so times $100$ gives $20\%$ . Revenue fell by 3 million, about one fifth.

Step 1 goes to the docs. Thought: "Figures set, cause missing." Action: ask for the Q4 marketing change. Observation: three notes score $0.81$ , $0.62$ , and $0.74$ against a keep gate of $0.70$ . The rule is plain: keep a note when its score is at or above the gate. So $0.81 \ge 0.70$ stays, $0.74 \ge 0.70$ stays, and $0.62 < 0.70$ goes. Two of three survive: the Q3 influencer campaign drove sales, and the Q4 budget was cut.

Step 2 answers. The model holds figures plus two notes, has used $2$ of $3$ steps, and emits FINAL: Q4 fell 3 million ( $20\%$ ) from Q3, tied to the kept notes. That is the diagram above, with the same figures in the same places.

The trade is direct. You buy answers to mixed questions that join a table and prose. You pay in extra model calls, since each thought is a full call, plus wait and more ways to fail. Skip the loop when one passage settles the question. Reach for it when the first query is a guess the results must correct.

Code

The point here is the gate and the step count, not the model. This stdlib-only loop runs the same two lookups with a score gate, so you see where junk gets dropped and where the loop stops:

1def gate(scores: list[float], floor: float = 0.70) -> list[float]:2    return [s for s in scores if s >= floor]3
4
5def drop_pct(q3: float, q4: float) -> tuple[float, float]:6    drop = q3 - q47    return drop, drop / q3 * 1008
9
10notes = [0.81, 0.62, 0.74]11kept = gate(notes)12print(kept)  # -> [0.81, 0.74]13print(len(kept))  # -> 214print(drop_pct(15.0, 12.0))  # -> (3.0, 20.0)15print(2 <= 3)  # -> True

Two notes survive the $0.70$ floor, the revenue math gives a drop of $3.0$ and $20.0$ percent, and the call count $2$ sits under the cap of $3$ . Change any score below the floor and the kept list shrinks, which is the behavior to watch in your own logs.

Watch Out For

Trusting a bad result because it looks official

Symptom: the answer cites a table that returned no rows or a note that scored below your gate. The prose sounds sure. The evidence is empty.

Check results in code before they reach the model. Drop docs under the score floor, reject empty rows, and force the model to write "not found" when nothing passes. The gate belongs in your code, never in the prompt.

The loop that never says done

Symptom: the trace shows the same source asked with the same words twice, the token bill climbs, and the run ends at the cap with a vague answer. The model got a thin result, did not admit it, and re-asked hoping for better.

Cap the loop near three to five steps, record every source-plus-query pair, and force FINAL or a rephrased query after one repeat. Log the stop reason with the answer so you can tell "answered" apart from "timed out."

The Quick Version

A retrieval-augmented agent searches its sources before answering instead of writing from memory.
Each loop step appends one tool result to the context, so the next query can use names and numbers from the last result.
A worked trace uses two calls: a table lookup showing Q3 at 15 and Q4 at 12 for a 20 percent drop, plus two docs notes kept at 0.81 and 0.74 over a 0.62 note.
A score gate near 0.70 in code decides what survives, and a step cap near three stops the walk.
Fixed one-shot search wins for single-passage questions where extra model calls add cost without better answers.