Parallel Tool Calling
When one request needs several independent lookups, the model lists them all in a single reply and your code runs them together, so you pay for one wait instead of three in a row.
Why Does This Exist?
Ask for the weather in three cities and a plain ReAct loop handles them one at a time. It writes a thought, calls get_weather for the first city, waits, reads the result, then starts over for the second city. If each lookup takes 2 seconds and each model reply takes 1 second, three cities cost about 9 seconds of waiting before you see anything useful. That stop and go rhythm is why agents feel slow even when each step is simple.
Parallel tool calling removes the queue. When the calls don't depend on each other, the model lists all of them in one reply and your code runs them side by side. You pay for one slow wait instead of three in a row.
This page covers that batching step: how one reply carries many calls and how your code rejoins the results. The single call itself lives in tool use and function calling. The loop it speeds up lives in ReAct pattern. The executor that runs the batch lives in agent harness design.
The one idea you need first: a tool call is a structured request, not code the model runs. The model emits a name plus arguments such as get_weather(city="Tokyo"), and only your code executes it and pastes the result back into the conversation. If that round trip is new, read tool use and function calling first, then come back.
Think of It Like This
Ordering for the whole table at a diner
You sit down with two friends and the server takes all three orders at once: burger, fries, soda. The kitchen fires all three together and brings one tray. You didn't order, wait, reorder, wait again. One conversation, one wait, three results.
The mapping is direct. Your one message is the table's order. Each tool call is one dish. The model is the server writing the whole ticket. Your executor is the kitchen working that ticket in parallel.
Where it stops: the kitchen can't cook fries that need the burger's leftovers, and neither can your executor. If the second call needs an ID the first call returns, there is no ticket to fire together. You're back to order, wait, reorder.
How It Actually Works
What one parallel turn looks like
A normal tool turn emits one call. A parallel turn emits a list.
Call the count , so means three calls in one reply. Call each entry , where runs from to . Each carries three fields: , a short unique label your code uses to match answers to questions; , which tool to run; and , the JSON inputs for that tool. For the weather question the list holds for New York, for Tokyo, and for London, each with set to get_weather and holding one city.
The model may only pack calls that are independent into that list. Independent means the of don't need the output of . Three city names you already know pass. A lookup whose input is an ID from another lookup fails that test.
How your code rejoins the batch
Your code does four jobs, in order. First it checks every against its schema before running anything, the same validation tool use and function calling describes. Then it runs the valid calls together, not one after another: asyncio.gather in Python or Promise.all in JavaScript, or a thread pool for blocking APIs. Call the wait for by the name , measured in seconds.
Third it collects one result per call, keyed by the same the model sent. If Tokyo fails you still know which result is missing because has no partner. Fourth it sends every pair back in a single message and lets the model write the final answer with all three facts in context.
You don't get three answers for the price of one. You pay output tokens for all calls in that big reply, and you pay input tokens again when every comes back. What you save is wall time.
Worked example: three cities, one wait
Suppose each weather lookup takes the same assumed wait so the arithmetic stays clear: , , and seconds. Suppose each model reply costs second. These are illustrative assumptions, not measurements of any API.
Sequential gathering pays the sum. Write for the sequential gather time:
So about seconds before the final answer starts.
Parallel gathering pays the maximum, because the three waits overlap. Write for the parallel gather time:
So about seconds. You save seconds on this toy ticket. The diagram above draws exactly this case: three side by side waits labeled s each collapsing into one s block, with s on the left and s on the right.
Don't use this shape when the calls depend on each other, when they write to the world instead of reading from it, or when the API punishes bursts with rate limits. A billing lookup that needs a customer ID from a prior call, a book then charge pair you can't safely retry, or three calls against a quota of one request per second all belong back in sequence. The assumption underneath the speedup is that the calls are independent, retry safe, and small enough that all results fit comfortably in context. Break it and you trade a slow correct run for a fast wrong one.
Code
The point of this snippet is the join, not the weather. It builds three labeled calls, runs the same function for each, and keeps the to pairing your follow-up message needs. The timing helper shows why the batch wins: a sum versus a maximum.
from typing import Callable
def get_weather(city: str) -> str: return f"{city}: 21C and sunny"
def run_parallel(cities: list[str]) -> dict[str, str]: calls: dict[str, str] = {f"call_{i}": city for i, city in enumerate(cities)} return {cid: get_weather(city) for cid, city in calls.items()}
def gather_seconds(waits: list[float], parallel: bool) -> float: if not waits: return 0.0 return max(waits) if parallel else sum(waits)
cities = ["New York", "Tokyo", "London"]print(run_parallel(cities))# -> {'call_0': 'New York: 21C and sunny', 'call_1': 'Tokyo: 21C and sunny', 'call_2': 'London: 21C and sunny'}
waits = [2.0, 2.0, 2.0]print(gather_seconds(waits, False)) # -> 6.0print(gather_seconds(waits, True)) # -> 2.0The dict keeps each result beside its call ID, and the two timing numbers versus are the tool wait sum and maximum from the worked example. Add one of per model turn to reach the full versus totals above.
Watch Out For
Firing dependent calls in one batch
Symptom: the second call carries a guessed ID and fails, or worse, returns the wrong customer's data with a success code. You'll see it when a request like "look up Maya, then fetch her invoices" produces two calls at once, with the invoice call holding an ID the lookup hasn't returned yet.
Fix: check the dependency before you fan out. If any field references an output you don't have yet, run in sequence. Only batch calls whose inputs are fully known up front. When in doubt, let the model take the extra turn.
Calling it parallel while your code runs serially
Symptom: the model emits three calls together but the wall time barely moves, or the burst trips a rate limit and you get three 429 errors at once. The usual cause is a plain for loop that awaits each tool in turn, so the "parallel" batch still pays the sum.
Fix: run the batch with a truly concurrent executor such as asyncio.gather or a thread pool, cap concurrency for tight quotas, and keep parallel calls read only and retry safe. Handle partial failure by so one slow tool doesn't silently orphan the other results.
The Quick Version
- Parallel tool calling lets one model reply list several independent tool calls, each tagged with its own ID.
- Your code runs that list side by side and returns every ID result pair in one message before the final answer.
- Three 2 second lookups cost about 9 seconds in sequence and about 3 seconds in parallel when each model turn costs 1 second.
- Parallel calls must share no data: a call that needs another call output belongs back in sequence.
- A parallel turn saves wall time but costs output tokens for every call and needs a truly concurrent executor to pay off.