Skip to content
AI360XpertBeta

Tool Retrieval

Your agent cannot read 500 tool manuals at once. Tool retrieval finds the two it needs for this request and shows it only those.

Tool retrieval ranks five tool descriptions by cosine to the query and injects only the top two schemas, so the agent chooses from two tools instead of five.
Tool retrieval ranks five tool descriptions by cosine to the query and injects only the top two schemas, so the agent chooses from two tools instead of five.

Why Does This Exist?

Give an agent five tools and you can paste all five manuals into the prompt. Give it two hundred and that plan falls apart. Each schema costs tokens, the prompt turns into a phone book, and the model starts picking the wrong tool or inventing a name that was never there. You feel it as rising cost first, then as silly mistakes on tools that used to work.

Tool retrieval fixes the selection step. You store every tool description as a vector once, and per request you fetch only the few schemas worth showing. The model then chooses from two or three candidates instead of two hundred.

This page covers that fetch: index, rank, inject. The call itself, meaning how the model names a function and how your code checks and runs it, lives in tool use and function calling. Ranking in general is semantic search, and the storage layer is a vector database. Why a smaller prompt behaves better is context budgeting.

The one idea you need first. An embedding turns text into a list of numbers placed so that closeness means relatedness. Cosine similarity then scores two vectors by the angle between them: 1 means same direction, 0 means unrelated, negative means opposite. Tool retrieval is that same machinery with tool manuals as the passages.

Think of It Like This

The parts counter at a repair shop

You walk up and say your hatchback rattles over bumps. You do not get a tour of all 500 bins in the stockroom. The clerk disappears down the aisle and comes back with two boxes that fit your model: a strut mount and a sway bar link. You pick one and get to work.

The mapping is direct. Your sentence is the user request. The stockroom is the tool store. The clerk's memory of which box fixes which rattle is the embedding space plus cosine ranking. The two boxes on the counter are the top-k schemas in the prompt. You are the agent making the final call.

Where it stops: a good clerk knows your car, asks a follow-up question, and remembers last week's repair. Retrieval sees only the current wording. If your request is vague, it returns the two most plausible tools, not necessarily the right ones.

How It Actually Works

Three steps run in order: index every tool once at setup time, rank them per request, inject only the winners into the prompt.

Index every tool once

For each tool you write one short description that says what it does and when to reach for it, for example "Cancel a subscription and stop future billing. Use when the user says cancel, stop, or end a plan." You embed that text with the same model you will use for queries and store the vector alongside the full JSON schema. Let did_i stand for the stored vector of tool ii, where ii runs over all tools. In plain words: did_i is the meaning of tool ii as a list of numbers.

Because this happens once, adding tool 201 costs one embedding, not a bigger prompt. That is the whole scaling win.

Rank each request

When the user writes a request you embed it the same way. Call that vector qq, the meaning of this request as a list of numbers. Score every tool with cosine similarity:

si=q⋅di∥q∥ ∥di∥s_i = \frac{q \cdot d_i}{\lVert q \rVert \, \lVert d_i \rVert}

Here q⋅diq \cdot d_i is the dot product (multiply matching positions and add), and ∥q∥\lVert q \rVert and ∥di∥\lVert d_i \rVert are the lengths of the two vectors. So sis_i is a number between −1-1 and 11 that says how closely tool ii points the same way as the request. Sort the scores and keep the best kk, where kk is the small cutoff you chose, often 2 or 3. The rest never reach the model.

Inject only the winners

Your backend builds the prompt with just those kk schemas and calls the model. The model picks one, your harness runs the real function, and the result flows back into the chat. If nothing fits, the model should answer directly instead of forcing a call, which is why the prompt keeps that option open.

Worked example

Take the request "cancel my subscription and refund my last charge" with a tiny two-number embedding q=[1,0.5]q = [1, 0.5]. Five tools sit in the store:

  • CancelSub at [1,0.4][1, 0.4]
  • RefundCharge at [0.9,0.2][0.9, 0.2]
  • LookupUser at [0.3,0.8][0.3, 0.8]
  • CreateUser at [0.1,1][0.1, 1]
  • ResetPassword at [0.2,−0.8][0.2, -0.8]

Do CancelSub fully. The dot product is 1×1+0.5×0.4=1.21 \times 1 + 0.5 \times 0.4 = 1.2. The lengths are ∥q∥=1+0.25=1.25≈1.118\lVert q \rVert = \sqrt{1 + 0.25} = \sqrt{1.25} \approx 1.118 and ∥d∥=1+0.16=1.16≈1.077\lVert d \rVert = \sqrt{1 + 0.16} = \sqrt{1.16} \approx 1.077. So s=1.2/(1.118×1.077)=1.2/1.204≈0.997s = 1.2 / (1.118 \times 1.077) = 1.2 / 1.204 \approx 0.997.

The same arithmetic for the rest gives:

  • CancelSub: 0.9970.997
  • RefundCharge: 0.9700.970
  • LookupUser: 0.7330.733
  • CreateUser: 0.5340.534
  • ResetPassword: −0.217-0.217

With top-k set to k=2k = 2, the prompt gets CancelSub and RefundCharge. The gap tells the story: second place at 0.9700.970 clears third place at 0.7330.733 by a wide margin, so the cutoff is clean. ResetPassword points slightly the other way, which is why its score goes negative. Those five numbers are the ones drawn in the diagram above.

What it costs and when to skip it

The trade-off is recall against prompt size. A larger kk misses the right tool less often but hands the model a longer prompt, more tokens per call, and more chances to pick a near miss. A smaller kk is cheap and focused but drops the second tool a multi-step job needs.

Do not use retrieval when the choice is not real. With fewer than about ten tools, paste them all in and skip the extra moving part. With one fixed workflow, hardcode the tool and skip search entirely. Retrieval assumes the description wording overlaps the request wording in embedding space. When your tools have thin or jargon-filled descriptions, that assumption fails and the right tool scores 0.50.5 while a plausible wrong one scores 0.90.9.

Code

The point of this snippet is the ranking plus the cutoff. It embeds nothing itself: the vectors stand in for real embedding output so you can see the selection rule run.

import math

def cosine(a: list[float], b: list[float]) -> float:    dot = sum(x * y for x, y in zip(a, b))    na = math.sqrt(sum(x * x for x in a))    nb = math.sqrt(sum(y * y for y in b))    return dot / (na * nb)

q: list[float] = [1.0, 0.5]tools: dict[str, list[float]] = {    "CancelSub": [1.0, 0.4],    "RefundCharge": [0.9, 0.2],    "LookupUser": [0.3, 0.8],    "CreateUser": [0.1, 1.0],    "ResetPassword": [0.2, -0.8],}
ranked: list[tuple[str, float]] = sorted(    ((name, cosine(q, vec)) for name, vec in tools.items()),    key=lambda item: item[1],    reverse=True,)for name, score in ranked:    print(f"{name}: {score:.3f}")# -> CancelSub: 0.997# -> RefundCharge: 0.970# -> LookupUser: 0.733# -> CreateUser: 0.534# -> ResetPassword: -0.217print([name for name, _ in ranked[:2]])# -> ['CancelSub', 'RefundCharge']

The printed order matches the hand computation: CancelSub at 0.9970.997 edges RefundCharge at 0.9700.970, and the top two clear the pack.

Watch Out For

Thin descriptions never surface

The symptom is specific: the correct tool exists yet never makes the top-k. You log scores and see it sitting at 0.40.4 while chatty generic tools score 0.80.8. The cause is usually a three-word description like "Handles billing operations," which gives the embedding almost nothing to match against "refund my last charge."

Fix it at the source. Give each tool one sentence of purpose plus trigger phrases, for example "Cancel a subscription and stop future billing. Use for cancel, stop, end plan, close account." Re-embed after every description edit, because the stored vector is the old text until you do.

Top-k of 2 starves a two-step task

The symptom: the agent cancels correctly, then stalls or asks the user for something it could have done itself, because the refund tool was rank 3 and never entered the prompt. You will see one clean call followed by confusion, not a wrong first call.

Fix it by retrieving per step, not once per chat. Re-run the search with the latest assistant message as the query before each call, or raise kk to 3 for multi-step flows. Never paper over this by stuffing all schemas back in.

The Quick Version

  • Tool retrieval is ranking applied to tool manuals: embed each description once, score every request with cosine, inject only the top-k schemas.
  • The model chooses from 2 or 3 tools instead of hundreds, which keeps the prompt small and the choice clean.
  • Cosine sis_i measures the angle between the request vector qq and each tool vector did_i, from −1-1 to 11.
  • In the worked example, CancelSub at 0.9970.997 and RefundCharge at 0.9700.970 clear LookupUser at 0.7330.733, so k=2k = 2 keeps exactly the right pair.
  • Retrieval assumes descriptions share wording with real requests. Thin descriptions break that assumption and need trigger phrases plus a re-embed.
  • Skip retrieval under about ten tools or for one fixed workflow. Retrieve per step when the task needs two different tools in a row.