AI Development

Same Weights, Different Answers: We Stopped Signing Off Inference Upgrades on Latency

Ninety catalogued bugs across two open-source serving frameworks, and the single most common symptom was not a crash — it was an accuracy regression. Changing only the inference backend, with the weights untouched, has been measured moving a benchmark by 16.6 points. Here is the gate that replaced our latency dashboard.

KAKabir AnandLead Developer
8 min read
Two identical brass balance scales side by side on a dark workbench, one pan resting slightly lower than the other

A model-serving container gets bumped a minor version, or a kernel backend gets switched on because it is faster, or a quantised build replaces the full-precision one to fit a smaller GPU. Throughput improves. P99 drops. Every request returns 200, every structured output still parses, and the error rate on the dashboard does not move at all. Six weeks later somebody notices that extraction quality is worse than it was in September, and nobody can point at a deploy — because the deploy that caused it was logged as a performance win.

This is not a thought experiment. A June 2026 paper from the University of Washington's SyFI lab and Amazon, presented at ICML this year, catalogued 90 of these across two of the most widely deployed open-source inference frameworks, vLLM and SGLang. The authors call them silent errors: the model answers, the answer is worse, and nothing in the stack says so.

We used to sign off inference upgrades on latency, GPU utilisation and HTTP error rate. We stopped.

A 200 is not a verdict

The thing that makes this class of bug expensive is that the symptom and the cause live nowhere near each other. In the catalogued set, the single most common symptom was an accuracy regression — 43.8% of cases — rather than a crash, a hang, or malformed output. The paper's own observation is the one worth taping to the wall: the symptom almost never tells you where the bug is.

The root causes are spread across framework code, model implementations and kernel backends, with model implementations and kernel backends together accounting for roughly half. And about 19% come down to pure numerical precision with no logical bug anywhere — nothing to revert, nothing to blame, just a different sequence of floating-point operations producing a different answer.

The named examples read like a list of things nobody tests for: an RMSNorm dimension mismatch in a Llama 4 implementation, an incorrect dtype in ERNIE 4.5, a data overflow in a Qwen3 NVFP4 path, FP16 weights loaded without the quantisation that was supposed to be applied. The same work later diagnosed four new ones, all confirmed by maintainers — including a Gemma 3 regression on GSM8K caused by a BOS token not being applied, and an audio model producing wrong output because of how a CUDA graph was captured.

Two of those are bugs that got fixed. One is not a bug at all. That distinction matters, because it means you cannot wait this out by pinning a known-good version — precision differences are a property of the implementation you chose, and they do not have a patch coming.

Flat vector illustration of a single file icon branching into five coloured pipelines that end at bars of unequal height
One weights file, five serving paths, five different scores. None of the branches error.

The backend is a hyperparameter nobody writes down

If silent errors were only about bugs, the answer would be patience. A second line of work this year says the floor is higher than that. A study published in May 2026 evaluated five widely used inference engines — vLLM, SGLang and llama.cpp among them — across multiple open-weight models and established benchmarks while holding the model weights, the decoding parameters and the hardware constant. Backend choice alone shifted benchmark scores by up to 16.6 percentage points, along with high rates of raw output disagreement.

The drivers named are all things you would describe as infrastructure in a planning doc: prefix caching, CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Every one of those is something a platform team turns on for throughput, in a ticket that has no model-quality reviewer on it.

A fully-crossed follow-up in August — three instruction-tuned models, five frameworks, six benchmarks, four generation modes — put a number on the share: roughly 39% of the variability a practitioner sees out of the box can be attributed to the backend, with the rest coming from sampling noise and each framework's default generation parameters. It persisted under greedy decoding, where sampling variance is removed entirely, and it was strongly model-dependent. Which is the part that defeats intuition: you cannot carry a conclusion from one model to the next one you deploy.

“We report the model and the weights and treat everything underneath as plumbing. A third of what we measure is coming from the plumbing.”

The honest caveat, which we give clients before we quote any of this: these are research findings on public benchmarks, not a universal failure rate for your workload. A 16.6-point swing is the worst cell in somebody else's grid, not a prediction about your extraction pipeline. The correct reading is narrower and more useful — the risk is real, it can be large, and it is invisible to every signal your deployment pipeline currently gates on. So measure it on your own stack rather than arguing about the number.

Why our old upgrade checklist passed everything

Our pre-flight for a serving change used to be four checks, and they were all real checks that have caught real problems. They just happen to be blind in exactly the same place.

  • Latency and throughput— the reason the upgrade was proposed. A silent error usually makes these _better_, because the thing that changed the numerics is the same thing that bought the speed. The metric that justifies the change is the metric least able to question it.
  • HTTP error rate and 5xx alerts— by definition unmoved. The model produced a response. The request succeeded.
  • Schema validation on structured output— this one feels like a quality check and is not. A JSON object with the right keys and the wrong values validates perfectly. In extraction and classification work, this is where the damage actually lands.
  • A smoke prompt someone eyeballs— the engineer running the upgrade sends two or three prompts and reads the answers. They look fine, because they usually are. A drop of a few points is invisible at a sample size of three, and a human reading their own deploy is not a blind test.

Put together, that checklist answers one question — did the service stay up — and then gets treated as if it answered a second one. The gap is not carelessness. It is that nothing in a normal deployment pipeline is positioned to notice a quality delta, because quality is not a property of a response, it is a property of a distribution of responses.

Hand-drawn ink and wash sketch of a panel of gauges all reading in the safe zone while a small dial below sits far off its mark
Everything on the panel is green. The dial nobody put on the panel is not.

The gate we run now

The reframe that fixed this for us is one sentence long: an inference-stack change is a model change. Version bump, quantisation, tensor-parallel degree, kernel backend, dtype, a cache that was off and is now on — all of it ships through the same gate a new model checkpoint would, and none of it ships on a latency graph alone. Concretely, five things:

What an inference upgrade has to clear before it ships

  • Keep a reference implementation and keep it boring. One configuration — unoptimised, full precision, whatever your framework's most conservative path is — that you never tune for speed. It exists to be the thing you diff against. Without a reference, you can detect that two builds disagree but never which one is wrong.
  • Replay task-level canaries, not benchmark suites. A few hundred real inputs from the workload you actually run, with the outputs you already accepted, is worth more than a public benchmark score. It is also the only version of this test that your client will recognise as their product.
  • Compare outputs, not just scores. The rate of exact disagreement between reference and candidate is a faster and more sensitive signal than an aggregate metric, and it is what the reproducibility work found moves first. An aggregate can hold steady while a quarter of individual answers change.
  • Record the full serving configuration in the trace, next to the model id. Framework and version, quantisation, dtype, tensor-parallel degree, kernel backend, which caches are enabled. When quality moves three weeks later, this is the field that turns an unreproducible complaint into a one-line diff.
  • Make the quality delta a release gate with a threshold, not a report someone reads. If the candidate drops more than the margin you agreed, the rollout stops. A dashboard nobody is obliged to act on has the same effect as no dashboard.

The cost of this is smaller than it sounds. The canary set is built once from data you already have. The comparison is a script. The trace fields are an afternoon. What it buys is that the class of regression with no deploy attached to it — the most expensive kind to debug, because the first fortnight goes on looking in the wrong place — now has a deploy attached to it.

“Treat the serving stack as infrastructure and you will find out about its quality effects from a customer, six weeks late, with no changelog to point at.”

If you only do one thing out of this, do the third one. Take two hundred inputs from production, run them through your current serving configuration and through the next one you were going to deploy, and count how many answers are not identical. You do not need a grading rubric or an eval framework to start — you need the disagreement rate. If it is near zero, you have a free upgrade and you can stop reading. If it is not, you have just found out what you were about to ship, before your users did.

Keep Reading

More from the blog

Track Record

The Engineering Partner You Can Build On

Reliable software takes an experienced team that owns delivery end to end. Here’s the track record behind ours.

Book a Free Consultation
01

11+

Years Building Custom Software

02

320+

Projects Delivered Across Web, Mobile & AI

03

85%

Repeat Client Rate

04

12+

Countries Served Worldwide

Trusted by startups and enterprises worldwide

Work With Us

Let’s create
with purpose

Share your goals, timeline, and challenges — we’ll respond with clarity and next steps.

Ambitious ideas deserve thoughtful execution. Start the conversation and let’s define what success looks like.

Team

Acetrum

Est. 2015

4.9/5

Trusted by
top brands

Services interested in:

By submitting, I confirm I’ve read and agree with Privacy and Cookie Policies.

Newsletter

Signals worth
paying attention

No recycled headlines — just the patterns we’re seeing across real client work, distilled into one read a month.

A curated digest of practical thinking and real-world brand perspectives monthly.

No spam. Unsubscribe anytime.