One tool on a client's agent takes ninety seconds on a bad day. It reads from a warehouse, waits on a partner API that has never published an SLA, and writes a report. The edge in front of that agent times out at sixty. For most of this year our answer was the answer everyone gives: make it faster, then split it, then cache it, then accept that some share of turns die and retry them. None of that was engineering. It was rationing against a number nobody had chosen for this workload.
On 3 September the Responses API picked up async tool calling, mid-turn steering, and the ability to change reasoning effort inside a conversation. Three controls in one release. It reads like a latency feature and it is not. It is the provider saying the model is no longer the thing holding your request open — which means whatever else was holding it open is now visible, and all of it belongs to you.
We spent last week moving one production agent onto this shape and scoping a second. The prompt changes took an afternoon. Everything else was backend work that had been quietly deferred since the day the agent shipped.
The synchronous loop was a deadline, not a design
Every agent backend written in the last two years is the same while loop. Call the model, read the tool calls, execute them, append the outputs, call the model again, stop when it stops asking. It is a good loop. It is also entirely inside one HTTP request, which means the ceiling on a turn is not a product decision — it is the shortest timeout anywhere on the path.
On that client the binding constraint was a sixty-second edge timeout set three years ago for a REST API that no longer exists. Nobody picked it for the agent. It was simply the first thing to give up, and every design decision above it had bent around it for eight months.
- Tools shaped by the timeout, not the task.— the ninety-second report became three tools that each fit under sixty, reassembled by the model in the prompt. That is three round trips, three chances to hallucinate a join, and a prompt that has to explain our infrastructure to a language model.
- Retries that redo work you already paid for.— a turn that dies at fifty-eight seconds has already spent tokens and already run side effects. We measured a month of it: 6% of turns retried, and roughly a fifth of tool spend went on results nobody ever read.
- Latency budgets that are actually one number.— a fast tool and a slow tool share a single deadline, so the slow one sets the policy for all of them. Raising the edge timeout to fix that just moves the ceiling and keeps the shape.
The registry is the migration
Read the release carefully, because the boundary is the whole story. The API will accept a tool result later, correlated against the original call_id, and will resume the conversation when it arrives. It will not run your job. It will not remember that a job exists. It will not retry it, time it out, or tell you it finished. Async tool calling moves the correlation problem into the protocol and leaves the execution problem exactly where it was.
So the first thing we write is not a prompt and not a tool definition. It is a table.
Four decisions are load-bearing here. The provider's call_id is the primary key, because it is the only identifier both sides agree on and the only one a late result can be matched against — generate your own and you will spend a week writing a mapping table that exists to undo that choice. The partial unique index is the idempotency story: a model that asks for the same report twice in one conversation gets one run, and the second dispatch is a no-op instead of a second invoice. And deadline_at is set at dispatch, not derived at read time, so a job that has outlived the reason it was started is expired by a sweeper rather than discovered by a support ticket.
Dispatch is a routing decision, not a rewrite
The version of this migration that gets abandoned halfway is the one where every tool goes async. Do not do that. Most tools are a database read that returns in 200 milliseconds, and putting those through a job table adds a write, a poll, a second model call and a whole class of failure in exchange for nothing. Of the twenty-three tools on that agent, four crossed the line.
The budget number should come from p99 over the last thirty days, per tool, not from an average and definitely not from a guess in a design doc. Averages hide exactly the runs this feature exists for. On our four tools the averages were 1.2 to 4 seconds and the p99s were 11, 34, 61 and 88 — and it was the 11-second tool, the one everybody described as fast, that was killing the most turns simply because it ran on every conversation.
Done this way the cheapest useful version of the migration is one table, one worker, and four tool handlers that return an acknowledgement instead of a payload. The other nineteen keep the code they already have.
The late result is where this actually gets hard
Four minutes after the tool was dispatched, it finishes. Now you have a result and a conversation that has moved on without it. In our first week of production traffic, roughly one delivery in nine landed in a conversation whose state had changed since dispatch: the user had asked something else, closed the tab, cancelled, or been answered from cache by a second run that finished first.
Two lines in there are opinions, not plumbing. The first is that a cancellation timestamp beats a completion timestamp — if the user said stop while the tool was running, the finished result is not a bonus, it is noise, and delivering it produces an assistant message about a question the user visibly abandoned. The second is that expiry archives rather than delivers. A stale answer arriving into a live conversation is worse than no answer, because the user has no way to tell it is stale and every downstream metric will count it as a success.
An abandoned run still queries the warehouse, still calls the partner API, still bills. Under the old synchronous loop that cost was invisible — the request died and took the evidence with it. Once every run has a row, it is one query, so put spend on runs nobody read on the dashboard the same day you ship the table. Ours was just under 12% of tool cost in week one, and that number is the entire business case for cancellation being real.
Steering only means something if you can interrupt
Mid-turn steering is the control everyone demos and the one with the deepest backend dependency. The user types "actually, EU rows only" while the report tool is thirty seconds into a ninety-second run. Accepting that message is cheap. Making it true is not, because there is a worker somewhere that has not heard about it and will happily finish the wrong report and hand it back with a perfectly valid call_id.
Our shape is deliberately dull. Cancellation writes a timestamp on the conversation, workers check it between declared stages, and every async tool has to expose its stages. A tool that cannot be interrupted between stages does not get to be async — it gets to be a job with a status the user can watch, which is a different product decision and should be made by a person, not inherited from a retry loop.
This is also the honest limit of the release. Three new controls arrived together and they compose well, but every one of them assumes an application that can name its in-flight work, address it, and stop it. If you cannot do those three things today, async tool calling will not give you interruptibility. It will give you a faster way to accumulate orphans.
The order we do it in
- 1Measure p99 per tool over thirty days. The list of tools that need this is almost never the list people expect, and it is usually shorter.
- 2Write the table before touching a prompt.If there is no row per run, there is no cancellation, no idempotency, no expiry and no cost number — and none of those can be retrofitted onto a fire-and-forget worker later.
- 3Move one tool, on one surface, behind a flag.The slowest one. Let it run a week and read the archive reasons before moving the second.
- 4Make cancellation real before you advertise steering.A steer that the worker never hears about is worse than no steer, because the user believes it landed.
- 5Ship the abandoned-run cost metric on day one.It is the only number that tells you whether the async path is paying for itself.
The reframing worth keeping is that none of this is new work created by the release. Long-running tools always needed a registry, an owner, a deadline and a way to be stopped. The synchronous loop was hiding all four behind a timeout that failed loudly enough to look like a policy. The timeout is gone now, and what is left is an ordinary distributed systems problem that our industry has known how to solve since well before any of us were putting language models in front of it.
