AI Development

The Failover Worked and the Answers Got Worse: A Model Handoff Is a Quality Event

Routers, fallbacks and forced upgrades mean a second model regularly finishes a conversation the first one started. Benchmarking each model alone never tests that handoff — and the drift it causes is directional, measurable, and invisible on your uptime dashboard.

KAKabir AnandLead Developer
8 min read
Two runners' hands mid baton exchange on a dark track, shallow depth of field

Between 28 October and 5 November 2026, a published diff of one model aggregator's catalogue tracked 665 matched routes across eight days. In that window 35 changed price, 33 were added, 24 picked up an expiration date, and 27 disappeared. Nine of the departures were re-listings under a new prefix and seven matched a vendor deprecation page dated 23 October. Eleven had no vendor notice anyone could find.

If one of those ids was in your config, the next request came back as HTTP 404 with a body saying no endpoints were found for the model. That is the easy failure. It is loud, it is typed, and every gateway in common use already handles it: the router walks to the next entry in your model list, that one answers, the user gets a reply. Latency is fine. The dashboard stays green.

That is the feature working exactly as designed. It is also the moment nobody evaluates.

A 404 is the easy part

Failover is sold as an availability feature, and it is a good one. The catalogue churn above is the argument for it — model ids go away on somebody else's schedule, and a fallback list is the cheapest insurance against waking up to a dead primary.

But a fallback does not resume a conversation the way a retry does. A retry sends the same turn to the same model family. A fallback sends the same turn to a different model, along with a transcript that a different model wrote. Every earlier turn in that history — its formatting habits, its hedging, the promises it made to the user, the shape of its tool arguments — is now input to a system that would not have produced any of it.

We have shipped enough assistant and agent work to say where this lands in practice: not as an outage, but as a support ticket that reads "it used to do this correctly." Nothing deployed. Nothing errored. A model you did not choose finished a conversation you did not re-test.

“Failover is measured in uptime and paid for in quality. Most teams only instrument the half they are measuring.”

The thing nobody benchmarks

Standard evaluation runs a model against a conversation it authored itself. Every turn in the history matches the model being scored. That tells you how good A is, and how good B is. It does not tell you how good B is at continuing A.

A March 2026 paper from Khraishi, Zafar, Myles and Cowan measured exactly that gap. Their switch-matrix benchmark runs a prefix model for the early turns of an episode, hands the final turn to a suffix model, and compares the result against the no-switch baseline — a 9x9 grid of model pairs, 200 sampled episodes per benchmark, paired bootstrap confidence intervals over 1,000 resamples per cell.

One switched turn was enough to move the number. On Multi-IF, strict success shifted anywhere from eight percentage points down to thirteen points up depending on which pair you picked. On CoQA the swing was roughly four absolute F1 in either direction. The effects were directional rather than noisy, and they were not evenly distributed across suffix models — some models are more disrupted by inheriting a foreign prefix than others. The authors decompose the drift into a prefix-influence term and a suffix-susceptibility term that together account for about seventy percent of the variance.

Grid of tiles with a pale neutral diagonal and off-diagonal tiles in varying warm and cool tones
A switch matrix: the diagonal is the no-switch baseline everyone benchmarks. The drift lives off it.

Two caveats we would put in front of any client before quoting those numbers. This is one study on two benchmarks, not a universal failure rate — the right reading is that the risk exists and can be large, not that your stack loses eight points. And the direction cuts both ways: some pairs improved. The problem is not that handoffs are bad. The problem is that you do not know which of yours are, because you have never run the off-diagonal cells.

Where the drift shows up first

Aggregate scores are a lagging indicator. In the journeys we instrument, the handoff gets noticed in a predictable order:

  • Format contracts— the first model settled into a structure across six turns; the suffix model re-derives one from the system prompt. Headings change, a JSON field goes missing, a table becomes prose. Downstream parsers see this before users do.
  • Earlier commitments— the assistant promised a refund window, a config value, a next step. The suffix model reads that as context, not as a binding it made, and will cheerfully contradict it.
  • Refusal and tone boundaries— two models rarely draw the same line in the same place. A conversation that was being handled gets a refusal on turn nine, or a conversation that should have been deflected keeps going.
  • Tool-call conventions— argument naming, how aggressively the model re-reads state, whether it batches calls. In an agent loop this is the expensive one: the handoff changes the trajectory, not just the sentence.

None of these raise an exception. All of them are visible in a trace, if the trace records which model wrote which turn.

The fallback that stops because it succeeded

There is a second failure mode here that is purely mechanical, and it is worth naming because it is so easy to ship. A fallback chain is a classifier: for every error it sees, it decides whether that candidate failed and the chain should continue, or whether it is done.

A bug filed against an open-source agent runtime this year describes the exact shape of getting that wrong. A deprecated model returns HTTP 404 with no endpoints found; the runtime records the decision as the candidate having succeeded; the chain halts there and never tries the next model. The correct classification is a model-not-found failure that continues the walk. Same status code, same response body — the whole behaviour turns on one branch in error handling.

Isometric illustration of a request passing through a gateway and branching toward two server stacks, one dimmed
The branch you never exercise in the happy path is the one that decides whether failover happens at all.

This is ordinary code with no tests on it, because the path only runs when something is already broken. We treat a fallback chain the way we treat a backup: it is not configured until it has been restored from. Deliberately point a staging config at a dead model id and watch the chain walk. If you have a context-window fallback as well, overflow it on purpose and confirm the request lands where you think it does.

Model sequence is a trace field

The fix is not to stop using fallbacks. It is to stop treating the model lineup as a constant. Five things we now expect to see in any production LLM system we take over:

What we add before the next provider incident

  • Record the model id per turn, not per session. A conversation is a sequence of models, and the trace should let you ask which sequence produced a given complaint. One field, added once, and it is the difference between a reproducible bug and an unreproducible one.
  • Alert on handoffs, not just on errors. A fallback firing is a business event. If you cannot answer how many conversations switched models yesterday, you cannot correlate a quality dip with anything.
  • Evaluate the pairs you can actually land on, not every pair. The switch matrix is nine by nine in a paper; in your stack it is primary plus two fallbacks, which is six off-diagonal cells on the journeys that matter.
  • Build eval episodes from real mixed-model histories. Replaying a transcript your primary wrote against your fallback is the cheapest version of this test and the one closest to production.
  • Pin and monitor ids, and keep a dead-id check in CI. A fallback pointing at a model that no longer exists is not a fallback, and catalogues move faster than your release cycle.

The cost of all of this is small — a trace field, a scheduled job, and a handful of extra eval runs. The cost of skipping it is a class of regression that has no deploy attached to it, which is the most expensive kind to debug because the first week is spent looking in the wrong place.

“If your evals only cover the diagonal, your coverage number is describing a system you do not run.”

If you change one thing this week

Log the model id on every turn and put a counter on fallback activations. That is an afternoon of work, it is reversible, and it converts the whole problem from an argument into a measurement. Most teams who do it are surprised by the number — fallbacks fire more often than the incident channel suggests, because by definition they fire instead of an incident.

Then run the one test that matters: take a real transcript your primary model produced, hand the final turn to your first fallback, and read both answers side by side. If they are interchangeable, you have cheap redundancy and you are done. If they are not, you have just found the quality gap your users have been absorbing quietly, and you found it before they told you about it.

Keep Reading

More from the blog

Track Record

The Engineering Partner You Can Build On

Reliable software takes an experienced team that owns delivery end to end. Here’s the track record behind ours.

Book a Free Consultation
01

11+

Years Building Custom Software

02

320+

Projects Delivered Across Web, Mobile & AI

03

85%

Repeat Client Rate

04

12+

Countries Served Worldwide

Trusted by startups and enterprises worldwide

Work With Us

Let’s create
with purpose

Share your goals, timeline, and challenges — we’ll respond with clarity and next steps.

Ambitious ideas deserve thoughtful execution. Start the conversation and let’s define what success looks like.

Team

Acetrum

Est. 2015

4.9/5

Trusted by
top brands

Services interested in:

By submitting, I confirm I’ve read and agree with Privacy and Cookie Policies.

Newsletter

Signals worth
paying attention

No recycled headlines — just the patterns we’re seeing across real client work, distilled into one read a month.

A curated digest of practical thinking and real-world brand perspectives monthly.

No spam. Unsubscribe anytime.