Skip to content
AI360XpertBeta

Agentic RAG

You ask one tricky question and the model acts like a researcher: it looks something up, reads what came back, looks something else up, then answers with receipts.

An agent answers a two part revenue question with two targeted tool calls instead of one fixed retrieval, then stops
An agent answers a two part revenue question with two targeted tool calls instead of one fixed retrieval, then stops

Why Does This Exist?

Single-shot RAG has one plan: search once with your original wording, then write. That plan breaks on questions that need two kinds of proof. Ask "Why did Q4 revenue drop vs Q3?" One docs search returns marketing notes but never touches the sales table, so the model writes a smooth answer with no figures in it. Wrong, and confident.

If you have run a support assistant, you have seen this daily. Thousands of mixed questions arrive where the wording points at prose and the truth lives in a table, or the reverse. Each single-shot miss looks fluent. Each one teaches users to stop trusting the answers.

This page covers the fix where the model runs retrieval itself: it picks each source, reads each result, and stops when the evidence is enough. For the single-shot baseline, see RAG architecture. For fixed pipelines with built-in checks, see self-RAG and corrective RAG. For the raw Thought Action Observation loop, see the ReAct pattern. For how one call is built, see tool use and function calling.

What you need first: plain RAG plus tool calls. In RAG, a query is embedded, matched against an index, and the top passages are pasted into the prompt before the model writes. In tool use, the model names a function and its arguments, real code runs it, and the result comes back as text. Agentic RAG puts those two together under the model's control: retrieval stops being one search and becomes a short loop.

Think of It Like This

A reference librarian, not a vending machine

A vending machine takes your coins, drops one item, and is done. A reference librarian works differently. You hand over a slip: "Why did the shop earn less last quarter?" She walks to the ledger room, copies two figures, reads them, and sees she still lacks the reason. So she walks to the archive room, pulls two folders, keeps the one with a matching date, and returns the other. Then she writes the answer with both slips pinned to it.

Map it point for point. Your question is the slip. Each room is a place to look. Each walk is one lookup. The pause to read before the next walk is what lets her change plans. The pinned slips are the proof behind the answer. The machine has one move. The librarian has a loop and a stopping rule: stop when the slips answer the question.

It stops here. A librarian understands your words. The model matches patterns, so it can misread a slip or ask for a room number that was never listed. And a librarian goes home after a fair try, while the model keeps walking unless code caps its steps.

How It Actually Works

The model keeps a running context and may call sources before it answers. Name the pieces once. The step counter tt starts at 00 and counts lookups. The context sts_t is everything seen so far: your question plus every result returned up to step tt. The thought TtT_t is a private note about what is still missing. The action ata_t is one chosen lookup: which place to ask plus the exact query. The observation oto_t is what that place returns. The cap Nmax⁡N_{\max} is a hard limit on steps. FINAL is a special action that means answer now instead of looking again.

The loop in four moves

  • Thought. Read sts_t and name the missing piece. "I have no numbers yet" or "I have numbers but no reason."
  • Action. Emit exactly one ata_t: one place, one query. One per step keeps blame clear, so a wrong answer traces to the oto_t that fed it.
  • Observation. Real code runs the lookup and appends oto_t to make st+1s_{t+1}. The model never fetches directly; it asks, and code fetches.
  • Stop check. If ata_t is FINAL, or tt has reached Nmax⁡N_{\max}, write the answer from sts_t with each claim tied to the result that supplied it. Else think again.

Worked example: one question, two lookups

Question: "Why did Q4 revenue drop vs Q3?" Set the cap Nmax⁡=3N_{\max} = 3.

Step 0 starts from numbers. Thought: "I need figures before stories." Action: ask the sales table for Q3 and Q4 revenue. Observation: Q3 =15= 15, Q4 =12= 12, in $M. Check the math yourself: the drop is 15−12=315 - 12 = 3, and the share is 3/15=0.203 / 15 = 0.20, so times 100100 gives 20%20\%. Revenue fell by $3M, about one fifth.

Step 1 starts from the reason. Thought: "Figures set, cause missing." Action: ask the docs for the Q4 marketing change. Observation: two candidates score 0.810.81 and 0.620.62 against a keep gate of 0.700.70. The rule is plain: keep a note when its score is at or above the gate. So 0.81≥0.700.81 \ge 0.70 stays and 0.62<0.700.62 < 0.70 goes. One note survives: the Q3 influencer campaign drove sales.

Step 2 answers. The model holds figures plus one supporting note, has used 22 of 33 steps, and emits FINAL: Q4 fell $3M (20%20\%) from Q3, and the kept note ties Q3 strength to the influencer campaign. Both claims point at the result that supplied them. That is the diagram above, with the same figures in the same places.

What it costs, and when to skip it

The trade is direct. You buy answers to mixed, two-source questions. You pay in extra model calls, since each thought is a full call, plus more tokens, more wait, and more ways to fail. Skip the loop when one passage settles the question; fixed RAG architecture is cheaper and faster there. Reach for the loop when the question joins two sources or the first query is a guess that the results must correct.

It rests on one assumption: the tool descriptions tell the truth and the results deserve trust. When a source returns junk and the model believes it, the loop polishes a bad fact into a smooth wrong answer. So gates live in code, not in the prompt: a score floor for docs, a row check for tables, and a forced "not found" when nothing passes. Your next read on shipping this is RAG in production.

Code

The point here is the guardrails, not the model. This small loop runs the same two lookups with a score gate and a step count, so you see where junk gets dropped and where the loop stops.

from typing import Callable

def sales_db(query: str) -> str:    return "Q3=15 Q4=12" if "revenue" in query else "no rows"

def vector_search(query: str) -> list[tuple[str, float]]:    _ = query    return [("q3 influencer campaign drove sales", 0.81), ("office holiday hours", 0.62)]

def run_agent(question: str, max_steps: int = 3) -> str:    calls = 0    notes: list[str] = []    row = sales_db("q3 q4 revenue")    calls += 1    q3, q4 = 15.0, 12.0  # parsed from row above    drop = q3 - q4  # 3.0    pct = drop / q3 * 100  # 20.0    notes.append(f"revenue drop {drop:.0f} ({pct:.0f}%)")    docs = [d for d in vector_search("q4 marketing change") if d[1] >= 0.70]    calls += 1    notes.append(f"kept {len(docs)} of 2 chunks")    _ = (question, max_steps)    return f"{notes[0]}; {notes[1]} in {calls} calls"

print(run_agent("Why did Q4 revenue drop vs Q3?"))  # -> revenue drop 3 (20%); kept 1 of 2 chunks in 2 calls

It prints one line. The gate keeps the 0.810.81 note and drops the 0.620.62 one, and the count proves the loop stopped after two calls with figures and reason joined.

Watch Out For

The loop that never says done

Symptom: the trace shows the same place asked with the same words twice, the token bill climbs, and the run ends at the cap with a vague answer. The model got a thin result, did not admit it, and re-asked hoping for better.

Cap the loop and ban repeats. Set Nmax⁡N_{\max} around three to five, record every (place,query)(place, query) pair, and force FINAL or a rephrased query after one repeat. Log the stop reason with the answer so you can tell "answered" apart from "timed out."

Trusting a bad result because it looks official

Symptom: the answer cites a table that returned "no rows" or a doc note that scored below your gate. The prose sounds sure. The evidence is empty.

Check results in code before they reach the model. Drop docs under the score floor, reject empty table rows, and make the model write "not found" when nothing passes. For the full failure list this joins, see agent failure modes.

The Quick Version

  • Agentic RAG is an LLM running a Thought Action Observation loop over lookup tools, not a fixed search then write pipeline.
  • Each loop step appends one tool result to the context, so the next query can use names and numbers from the last result.
  • A worked trace uses two calls: a table lookup showing Q3 at 15 and Q4 at 12 for a 20 percent drop, plus one docs note at 0.81 kept over a 0.62 note.
  • The loop ends on a FINAL answer or a step cap near three to five, and a repeated identical lookup forces a stop.
  • Fixed single-shot RAG wins for one-passage questions where extra model calls add cost without better answers.