The incidents that cost the most are the ones where nothing fails. For six days in July, a widely deployed LLM gateway returned a correct completion for every request it handled. Error rates stayed flat. Latency moved inside normal variance. Output quality did not change, so nothing an eval suite looks at moved either. Over the same six days, teams routing Claude Code sessions through Amazon Bedrock's Invoke path were paying two to three times what they had paid the week before, for identical usage.
The cause, published by the gateway's own maintainers once they had shipped the fix, was a change that is functionally a no-op. A patch moved every entry with role "system" out of the message list and into the top-level system field. The model sees the same instructions in the same order and produces the same answer, which is exactly why the change passed review and passed CI. What it also did was invalidate every cache breakpoint past the tool definitions and the system prompt. Warm-session cache hit rates fell from roughly 90 per cent to somewhere between 25 and 45 per cent, and the difference went onto the bill.
We run gateways in front of model traffic for several clients, so we read that report the way you read a near miss in someone else's industry. The bug itself is fixed and is not the interesting part. The interesting part is that we went looking for which of our own checks would have caught it, and the honest answer was none of them. Not the health checks, not the error budgets, not the eval suite, not the latency SLOs. We had built a monitoring stack that could tell us whether the model was right and had no opinion whatsoever on whether it was affordable.
A 200 is not a cost test
Almost every signal in a production LLM stack is a correctness signal wearing different clothes. Status codes, error rates, retry counts, timeout ratios, eval pass rates, human thumbs-up — all of them answer the question "did we get a good answer?" and all of them stayed green through this. Even the two signals people assume are financial are not. Request volume was unchanged. Input token counts were unchanged, because the number of tokens in the prompt is not what moved; what moved was which price band those same tokens were billed in.
That last point took us an embarrassingly long time to internalise. We had a dashboard with input tokens on it and we thought of it, loosely, as the cost graph. It is not. With prompt caching in play, the same token count can cost anything from a tenth of the base rate to one and a quarter times it depending on whether it was read from a cache or written to one. A flat input-token line is perfectly compatible with a bill that has tripled, and that is not an edge case — it is the normal behaviour of a system where caching does the heavy lifting.
| Signal | What it showed during the window | What was actually happening |
|---|---|---|
| HTTP status | 200 on every request | 200 on every request — the model was fine |
| Output quality | Unchanged; evals passing | Unchanged; the prompt content was identical |
| p95 turn latency | Slightly worse, within normal variance | Every turn reprocessing a cold prefix |
| Input tokens | Flat | Flat — only the billing band moved, not the count |
| Cache read tokens | Not collected | Collapsed from ~90% of input to 25–45% |
| Daily spend | Reviewed at month end | 2–3× higher from the first day |
Read that table as an argument about ordering rather than about tooling. Every row above the last one updates in seconds and is watched by an on-call rotation. The last row updates once a month and is read by whoever opens the invoice. A regression that only expresses itself in the bottom row has a four-week head start, by design.
The prefix is the cache key, and it is byte-for-byte
Prompt caching is usually explained as a discount, which is a bad frame because discounts are forgiving. It is better understood as a lookup against a prefix, and a prefix is a cache key with all the brittleness that implies. The request is assembled in a fixed order before any of it is hashed, and the order is not up to you.
“Cache prefixes are created in the following order: tools, system, then messages.”
One sentence, and it contains the entire failure mode. Because the order is fixed and the match is on a prefix, invalidation cascades forwards and only forwards. Change a byte of a tool definition and you lose tools, system and messages together, even though the system prompt text never moved. Change the system prompt and you lose system and messages. Move a block across the boundary between two of those sections — which is precisely what hoisting a system message does — and everything after the boundary has to be written again.
Now put the pricing next to it. Writing a prefix into the cache costs 1.25 times the base input rate with a five-minute lifetime, or twice the base rate for an hour. Reading one back costs a tenth of the base rate. So a write is 12.5 times a read, and the break-even is not where intuition puts it: on a stable-prefix workload, a cache that misses most of the time is more expensive than no caching at all, because you pay the write premium on every turn and almost never collect the read. There is no graceful degradation here. A broken cache does not get slower or cheaper or less accurate. It gets expensive, silently, at full quality.
Which also explains why a long agent session is where this bites hardest. A single chat turn with a short system prompt has almost nothing at stake. An agent with thirty thousand tokens of tool schemas and instructions, running fifty turns, is re-billing that entire prefix at the write rate on every single turn once the breakpoints stop matching — and it will cheerfully finish the task and report success.
Translation layers rewrite the prefix
Here is the part that should worry anyone running a gateway, a proxy, a compatibility shim or an SDK adapter, which by now is most of us. Caching is a property of the exact bytes that reach the provider. Your application does not control those bytes. Every hop between your code and the model is a deserialise-and-reserialise step, and each one is written by someone optimising for semantic equivalence, because semantic equivalence is what the tests check.
"Semantically equivalent" is the dangerous phrase. Hoisting a system message is semantically equivalent. Sorting tool definitions into a different order is semantically equivalent. Rewriting a provider-native request into a generic chat-completions shape and back is semantically equivalent, and it is also where cache_control markers go to die: a publicly filed report measured the cache-read share of input at 99 per cent when Claude Code talked to the provider directly, and 9 per cent for the same workload through a chat-completions bridge, because the conversion dropped the breakpoints and every request re-billed the whole context as a write. Same prompts, same model, same answers, 12 to 32 times the cost on the cells they compared.
We now treat any translation hop as a cache boundary until proven otherwise, and the proof has to be empirical. Not reading the adapter's source, not trusting its changelog — recording the request body the provider actually received and diffing it against the one our application sent. In two of the four stacks we checked this way, the bytes differed in ways nobody had intended and nobody had noticed, because in both cases the output was correct.
The telemetry breaks in the same place as the cache
If the previous section is the problem, this one is the reason it survives. The obvious response to everything above is to put cache read and cache write tokens on a dashboard and alert when the read share drops. Good instinct. It has a failure mode that is genuinely nasty, and there is a bug title that states it better than we could.
“Bedrock Invoke streaming drops cacheRead/cacheWrite token counts → cache-heavy traffic billed as fresh input”
The mechanics are mundane. On that streaming path the final chunk carries usage in a provider-specific metrics block, the cache counts live under differently-cased keys inside it, and they were dropped before the step that promotes usage into the normalised response could see them. Input and output token counts came through fine. Cache read and cache write came through as zero. The provider had billed them correctly; the numbers simply did not survive the journey back.
So your monitoring reports a cache hit rate of zero. And a cache hit rate of zero is exactly what a catastrophic cache regression also looks like. The two are indistinguishable from the dashboard, which puts you in the worst position available: an alert that fires loudly for a condition you cannot diagnose, on a path where the real cache might be working perfectly. Teams resolve that ambiguity the way humans always resolve a noisy alert, by turning it off — and the metric that would have caught the real regression is now switched off because of a reporting bug that never cost anyone a rupee.
The conclusion we drew is that usage telemetry has to be reconciled against something that is not usage telemetry. We pull the provider's own billing or usage export on a schedule and compare its cache-read and cache-write totals against what our gateway recorded for the same window. When those two disagree the alert is about the pipeline, not the cache, and it says so. It is a dull piece of plumbing and it is the only thing that can tell those two failures apart.
Functional tests structurally cannot see this
There is a reason the review process let this through, and it is not carelessness. A cache hit requires a second request whose prefix matches a first one, inside the lifetime of that first one. A test suite that sends one request per case cannot observe a cache read at all — not because it is a bad suite, but because the phenomenon does not exist in a single-turn world. Every assertion such a suite can make will be true, and the regression will pass.
Mocked providers make it worse again. A mock returns the response shape you told it to return, which means it returns the cache token fields you believed were there. Mock-based tests are structurally incapable of catching a regression in the fields the real provider sends, because the mock is a transcription of your beliefs about the provider rather than the provider.
- Multi-turn, or it measures nothing— the test has to be a warm session: a real first request that writes a prefix, then several follow-ups inside the cache lifetime that should read it. We run ours with concurrency, because serialised turns hide the interleaving bugs that only appear when two sessions share a gateway process.
- Against the real deployment, on every route you ship— the regression here lived on one provider path while the others were unaffected. A suite that exercises your default route and nothing else gives you a confident green for the ninety per cent of traffic that was never at risk. Every route in the routing table needs its own baseline.
- Assert on the request bytes, not just the response— capture what the provider received and compare the prefix to the previous turn’s byte for byte. This is the assertion that names the cause instead of the symptom: it tells you the breakpoint moved, rather than that the number went down.
The maintainers reached the same conclusion, and their write-up describes CI that now runs concurrent Claude Code-shaped sessions weekly against real deployments, failing if error rate, warm-turn cache reads, cache write volume, p95 turn latency or recorded spend leaves its baseline. We copied the idea almost verbatim, with one change that matters: ours runs on every dependency bump to the gateway or the provider SDK, not weekly. Six days of unnoticed triple spend is roughly what a weekly cadence buys you in the worst case, and dependency bumps are when this class of change actually arrives.
What we gate on now
None of this is specific to one gateway, one provider or one caching implementation. The reusable shape is that caching turns a performance optimisation into a correctness-invisible cost contract, and that contract is written in bytes you do not own and enforced by nobody. Here is what we changed.
Five things our LLM regression suite does that a functional suite does not
- Baseline cache-read share per route, not globally. Store the warm-turn cache-read fraction for each provider path you ship and fail the build on a relative drop, not an absolute threshold — an absolute number is wrong for every workload except the one you tuned it on.
- Gate on cost per completed task, not cost per request or tokens per request. A regression that keeps request counts and token counts flat while moving the billing band is invisible to both of those, and obvious to the first.
- Diff the outbound request bytes across turns. The prefix up to your last breakpoint must be byte-identical from turn two onwards. When it is not, the diff names the field that moved, which turns a cost mystery into a one-line fix.
- Reconcile usage telemetry against the provider's own billing export. Without this you cannot distinguish a broken cache from broken cache reporting, and the ambiguity will get your alert muted inside a month.
- Re-run the warm-session suite on every gateway and SDK upgrade, before merge. This class of change arrives through dependency updates and is functionally a no-op by definition, so the dependency PR is the only place it is cheap to catch.
If you adopt one of those, adopt the third. Cost per task tells you something regressed; the byte diff tells you what. We spent most of an afternoon on the first version of ours and it has already paid for itself twice, both times on changes that were correct, well-reviewed, and shipped by people who had no reason to think about caching at all.
The wider lesson is one we keep relearning in different clothes. A system that degrades loudly is a system you can run with ordinary discipline. Prompt caching does not degrade loudly: it degrades into full-quality, fully-correct, fully-available output at several times the price, which is the one failure mode that every instrument on the wall is built to call healthy. Until the bill becomes a monitored signal with the same latency as the error rate, the only honest thing to say about your cache hit rate is that you do not currently know it.
