Skip to content
AI360XpertBeta

Parallel Tool Calling

When one request needs several independent lookups, the model lists them all in a single reply and your code runs them together, so you pay for one wait instead of three in a row.

One assistant turn fans out into three independent weather calls that run side by side, so three 2 second waits collapse into one before the final answer
One assistant turn fans out into three independent weather calls that run side by side, so three 2 second waits collapse into one before the final answer

Why Does This Exist?

Ask for the weather in three cities and a plain ReAct loop handles them one at a time. It writes a thought, calls get_weather for the first city, waits, reads the result, then starts over for the second city. If each lookup takes 2 seconds and each model reply takes 1 second, three cities cost about 9 seconds of waiting before you see anything useful. That stop and go rhythm is why agents feel slow even when each step is simple.

Parallel tool calling removes the queue. When the calls don't depend on each other, the model lists all of them in one reply and your code runs them side by side. You pay for one slow wait instead of three in a row.

This page covers that batching step: how one reply carries many calls and how your code rejoins the results. The single call itself lives in tool use and function calling. The loop it speeds up lives in ReAct pattern. The executor that runs the batch lives in agent harness design.

The one idea you need first: a tool call is a structured request, not code the model runs. The model emits a name plus arguments such as get_weather(city="Tokyo"), and only your code executes it and pastes the result back into the conversation. If that round trip is new, read tool use and function calling first, then come back.

Think of It Like This

Ordering for the whole table at a diner

You sit down with two friends and the server takes all three orders at once: burger, fries, soda. The kitchen fires all three together and brings one tray. You didn't order, wait, reorder, wait again. One conversation, one wait, three results.

The mapping is direct. Your one message is the table's order. Each tool call is one dish. The model is the server writing the whole ticket. Your executor is the kitchen working that ticket in parallel.

Where it stops: the kitchen can't cook fries that need the burger's leftovers, and neither can your executor. If the second call needs an ID the first call returns, there is no ticket to fire together. You're back to order, wait, reorder.

How It Actually Works

What one parallel turn looks like

A normal tool turn emits one call. A parallel turn emits a list.

Call the count nn, so n=3n = 3 means three calls in one reply. Call each entry cic_i, where ii runs from 11 to nn. Each cic_i carries three fields: idiid_i, a short unique label your code uses to match answers to questions; namename, which tool to run; and argumentsarguments, the JSON inputs for that tool. For the weather question the list holds c1c_1 for New York, c2c_2 for Tokyo, and c3c_3 for London, each with namename set to get_weather and argumentsarguments holding one city.

The model may only pack calls that are independent into that list. Independent means the argumentsarguments of cic_i don't need the output of cjc_j. Three city names you already know pass. A lookup whose input is an ID from another lookup fails that test.

How your code rejoins the batch

Your code does four jobs, in order. First it checks every cic_i against its schema before running anything, the same validation tool use and function calling describes. Then it runs the valid calls together, not one after another: asyncio.gather in Python or Promise.all in JavaScript, or a thread pool for blocking APIs. Call the wait for cic_i by the name tit_i, measured in seconds.

Third it collects one result rir_i per call, keyed by the same idiid_i the model sent. If Tokyo fails you still know which result is missing because id2id_2 has no partner. Fourth it sends every pair (idi,ri)(id_i, r_i) back in a single message and lets the model write the final answer with all three facts in context.

You don't get three answers for the price of one. You pay output tokens for all nn calls in that big reply, and you pay input tokens again when every rir_i comes back. What you save is wall time.

Worked example: three cities, one wait

Suppose each weather lookup takes the same assumed wait so the arithmetic stays clear: t1=2.0t_1 = 2.0, t2=2.0t_2 = 2.0, and t3=2.0t_3 = 2.0 seconds. Suppose each model reply costs tllm=1.0t_{llm} = 1.0 second. These are illustrative assumptions, not measurements of any API.

Sequential gathering pays the sum. Write TseqT_{seq} for the sequential gather time:

Tseq=t1+t2+t3+3×tllm=2.0+2.0+2.0+3.0=9.0T_{seq} = t_1 + t_2 + t_3 + 3 \times t_{llm} = 2.0 + 2.0 + 2.0 + 3.0 = 9.0

So about 9.09.0 seconds before the final answer starts.

Parallel gathering pays the maximum, because the three waits overlap. Write TparT_{par} for the parallel gather time:

Tpar=max⁡(t1,t2,t3)+1×tllm=2.0+1.0=3.0T_{par} = \max(t_1, t_2, t_3) + 1 \times t_{llm} = 2.0 + 1.0 = 3.0

So about 3.03.0 seconds. You save 9.0−3.0=6.09.0 - 3.0 = 6.0 seconds on this toy ticket. The diagram above draws exactly this case: three side by side waits labeled 2.02.0 s each collapsing into one 2.02.0 s block, with 99 s on the left and 33 s on the right.

Don't use this shape when the calls depend on each other, when they write to the world instead of reading from it, or when the API punishes bursts with rate limits. A billing lookup that needs a customer ID from a prior call, a book then charge pair you can't safely retry, or three calls against a quota of one request per second all belong back in sequence. The assumption underneath the speedup is that the calls are independent, retry safe, and small enough that all results fit comfortably in context. Break it and you trade a slow correct run for a fast wrong one.

Code

The point of this snippet is the join, not the weather. It builds three labeled calls, runs the same function for each, and keeps the idiid_i to rir_i pairing your follow-up message needs. The timing helper shows why the batch wins: a sum versus a maximum.

from typing import Callable

def get_weather(city: str) -> str:    return f"{city}: 21C and sunny"

def run_parallel(cities: list[str]) -> dict[str, str]:    calls: dict[str, str] = {f"call_{i}": city for i, city in enumerate(cities)}    return {cid: get_weather(city) for cid, city in calls.items()}

def gather_seconds(waits: list[float], parallel: bool) -> float:    if not waits:        return 0.0    return max(waits) if parallel else sum(waits)

cities = ["New York", "Tokyo", "London"]print(run_parallel(cities))# -> {'call_0': 'New York: 21C and sunny', 'call_1': 'Tokyo: 21C and sunny', 'call_2': 'London: 21C and sunny'}
waits = [2.0, 2.0, 2.0]print(gather_seconds(waits, False))  # -> 6.0print(gather_seconds(waits, True))  # -> 2.0

The dict keeps each result beside its call ID, and the two timing numbers 6.06.0 versus 2.02.0 are the tool wait sum and maximum from the worked example. Add one tllmt_{llm} of 1.01.0 per model turn to reach the full 9.09.0 versus 3.03.0 totals above.

Watch Out For

Firing dependent calls in one batch

Symptom: the second call carries a guessed ID and fails, or worse, returns the wrong customer's data with a success code. You'll see it when a request like "look up Maya, then fetch her invoices" produces two calls at once, with the invoice call holding an ID the lookup hasn't returned yet.

Fix: check the dependency before you fan out. If any argumentsarguments field references an output you don't have yet, run in sequence. Only batch calls whose inputs are fully known up front. When in doubt, let the model take the extra turn.

Calling it parallel while your code runs serially

Symptom: the model emits three calls together but the wall time barely moves, or the burst trips a rate limit and you get three 429 errors at once. The usual cause is a plain for loop that awaits each tool in turn, so the "parallel" batch still pays the sum.

Fix: run the batch with a truly concurrent executor such as asyncio.gather or a thread pool, cap concurrency for tight quotas, and keep parallel calls read only and retry safe. Handle partial failure by idiid_i so one slow tool doesn't silently orphan the other results.

The Quick Version

  • Parallel tool calling lets one model reply list several independent tool calls, each tagged with its own ID.
  • Your code runs that list side by side and returns every ID result pair in one message before the final answer.
  • Three 2 second lookups cost about 9 seconds in sequence and about 3 seconds in parallel when each model turn costs 1 second.
  • Parallel calls must share no data: a call that needs another call output belongs back in sequence.
  • A parallel turn saves wall time but costs output tokens for every call and needs a truly concurrent executor to pay off.