Tool Retrieval
Your agent cannot read 500 tool manuals at once. Tool retrieval finds the two it needs for this request and shows it only those.
Why Does This Exist?
Give an agent five tools and you can paste all five manuals into the prompt. Give it two hundred and that plan falls apart. Each schema costs tokens, the prompt turns into a phone book, and the model starts picking the wrong tool or inventing a name that was never there. You feel it as rising cost first, then as silly mistakes on tools that used to work.
Tool retrieval fixes the selection step. You store every tool description as a vector once, and per request you fetch only the few schemas worth showing. The model then chooses from two or three candidates instead of two hundred.
This page covers that fetch: index, rank, inject. The call itself, meaning how the model names a function and how your code checks and runs it, lives in tool use and function calling. Ranking in general is semantic search, and the storage layer is a vector database. Why a smaller prompt behaves better is context budgeting.
The one idea you need first. An embedding turns text into a list of numbers placed so that closeness means relatedness. Cosine similarity then scores two vectors by the angle between them: 1 means same direction, 0 means unrelated, negative means opposite. Tool retrieval is that same machinery with tool manuals as the passages.
Think of It Like This
The parts counter at a repair shop
You walk up and say your hatchback rattles over bumps. You do not get a tour of all 500 bins in the stockroom. The clerk disappears down the aisle and comes back with two boxes that fit your model: a strut mount and a sway bar link. You pick one and get to work.
The mapping is direct. Your sentence is the user request. The stockroom is the tool store. The clerk's memory of which box fixes which rattle is the embedding space plus cosine ranking. The two boxes on the counter are the top-k schemas in the prompt. You are the agent making the final call.
Where it stops: a good clerk knows your car, asks a follow-up question, and remembers last week's repair. Retrieval sees only the current wording. If your request is vague, it returns the two most plausible tools, not necessarily the right ones.
How It Actually Works
Three steps run in order: index every tool once at setup time, rank them per request, inject only the winners into the prompt.
Index every tool once
For each tool you write one short description that says what it does and when to reach for it, for example "Cancel a subscription and stop future billing. Use when the user says cancel, stop, or end a plan." You embed that text with the same model you will use for queries and store the vector alongside the full JSON schema. Let stand for the stored vector of tool , where runs over all tools. In plain words: is the meaning of tool as a list of numbers.
Because this happens once, adding tool 201 costs one embedding, not a bigger prompt. That is the whole scaling win.
Rank each request
When the user writes a request you embed it the same way. Call that vector , the meaning of this request as a list of numbers. Score every tool with cosine similarity:
Here is the dot product (multiply matching positions and add), and and are the lengths of the two vectors. So is a number between and that says how closely tool points the same way as the request. Sort the scores and keep the best , where is the small cutoff you chose, often 2 or 3. The rest never reach the model.
Inject only the winners
Your backend builds the prompt with just those schemas and calls the model. The model picks one, your harness runs the real function, and the result flows back into the chat. If nothing fits, the model should answer directly instead of forcing a call, which is why the prompt keeps that option open.
Worked example
Take the request "cancel my subscription and refund my last charge" with a tiny two-number embedding . Five tools sit in the store:
- CancelSub at
- RefundCharge at
- LookupUser at
- CreateUser at
- ResetPassword at
Do CancelSub fully. The dot product is . The lengths are and . So .
The same arithmetic for the rest gives:
- CancelSub:
- RefundCharge:
- LookupUser:
- CreateUser:
- ResetPassword:
With top-k set to , the prompt gets CancelSub and RefundCharge. The gap tells the story: second place at clears third place at by a wide margin, so the cutoff is clean. ResetPassword points slightly the other way, which is why its score goes negative. Those five numbers are the ones drawn in the diagram above.
What it costs and when to skip it
The trade-off is recall against prompt size. A larger misses the right tool less often but hands the model a longer prompt, more tokens per call, and more chances to pick a near miss. A smaller is cheap and focused but drops the second tool a multi-step job needs.
Do not use retrieval when the choice is not real. With fewer than about ten tools, paste them all in and skip the extra moving part. With one fixed workflow, hardcode the tool and skip search entirely. Retrieval assumes the description wording overlaps the request wording in embedding space. When your tools have thin or jargon-filled descriptions, that assumption fails and the right tool scores while a plausible wrong one scores .
Code
The point of this snippet is the ranking plus the cutoff. It embeds nothing itself: the vectors stand in for real embedding output so you can see the selection rule run.
import math
def cosine(a: list[float], b: list[float]) -> float: dot = sum(x * y for x, y in zip(a, b)) na = math.sqrt(sum(x * x for x in a)) nb = math.sqrt(sum(y * y for y in b)) return dot / (na * nb)
q: list[float] = [1.0, 0.5]tools: dict[str, list[float]] = { "CancelSub": [1.0, 0.4], "RefundCharge": [0.9, 0.2], "LookupUser": [0.3, 0.8], "CreateUser": [0.1, 1.0], "ResetPassword": [0.2, -0.8],}
ranked: list[tuple[str, float]] = sorted( ((name, cosine(q, vec)) for name, vec in tools.items()), key=lambda item: item[1], reverse=True,)for name, score in ranked: print(f"{name}: {score:.3f}")# -> CancelSub: 0.997# -> RefundCharge: 0.970# -> LookupUser: 0.733# -> CreateUser: 0.534# -> ResetPassword: -0.217print([name for name, _ in ranked[:2]])# -> ['CancelSub', 'RefundCharge']The printed order matches the hand computation: CancelSub at edges RefundCharge at , and the top two clear the pack.
Watch Out For
Thin descriptions never surface
The symptom is specific: the correct tool exists yet never makes the top-k. You log scores and see it sitting at while chatty generic tools score . The cause is usually a three-word description like "Handles billing operations," which gives the embedding almost nothing to match against "refund my last charge."
Fix it at the source. Give each tool one sentence of purpose plus trigger phrases, for example "Cancel a subscription and stop future billing. Use for cancel, stop, end plan, close account." Re-embed after every description edit, because the stored vector is the old text until you do.
Top-k of 2 starves a two-step task
The symptom: the agent cancels correctly, then stalls or asks the user for something it could have done itself, because the refund tool was rank 3 and never entered the prompt. You will see one clean call followed by confusion, not a wrong first call.
Fix it by retrieving per step, not once per chat. Re-run the search with the latest assistant message as the query before each call, or raise to 3 for multi-step flows. Never paper over this by stuffing all schemas back in.
The Quick Version
- Tool retrieval is ranking applied to tool manuals: embed each description once, score every request with cosine, inject only the top-k schemas.
- The model chooses from 2 or 3 tools instead of hundreds, which keeps the prompt small and the choice clean.
- Cosine measures the angle between the request vector and each tool vector , from to .
- In the worked example, CancelSub at and RefundCharge at clear LookupUser at , so keeps exactly the right pair.
- Retrieval assumes descriptions share wording with real requests. Thin descriptions break that assumption and need trigger phrases plus a re-embed.
- Skip retrieval under about ten tools or for one fixed workflow. Retrieve per step when the task needs two different tools in a row.