At 16:12 on a Tuesday, eleven days before the end of the billing period, a client's in-product assistant started returning errors to every user who opened it. Nothing had deployed. No model was down. An org-level spend cap had been reached, and every request under that organisation began coming back 429.
The cap was ours. We had recommended it three weeks earlier, in a document that described it as a safety net. It behaved exactly as documented. The part we had not written down is that a hard limit is not a budget control at all — it is a kill switch wired to a number that a background job can reach on its own, at a time nobody chose, with no human in the loop. That week roughly four percent of the spend belonged to the feature customers were actually paying for. The other ninety-six percent was a re-embedding job somebody had left on a nightly schedule after a migration, and it was the one that pulled the trigger.
The cap is a failure path, not a finance setting
Enforceable monthly limits at both organisation and project level are a genuinely good change. Before them, cost control was an email alert arriving after the money was gone, and every agency has at least one story about a runaway loop discovered on an invoice. Having the platform refuse to spend past a number is strictly better than finding out in arrears.
What changed with them is not the finance conversation. It is where the control lives. A dashboard alert is an operational nuisance; an enforced limit is a branch in your request handling that executes under load, in production, at the worst moment in the month. It belongs in the same category as a database connection pool exhausting or a third-party timing out, and it deserves the same treatment: a defined behaviour, a test, and a runbook. Almost nobody gives it that, because it arrives dressed as a billing checkbox.
Two details make it sharper than a generic dependency failure. The first is the status code. A reached cap returns 429, which is also what rate limiting returns, which is what every sensible client library already retries with exponential backoff. So the default behaviour of a well-built system, on the day the cap fires, is to hammer a limit that will not lift for eleven days — burning latency budget and filling logs with retries that were correct code doing exactly the wrong thing. The distinguishing signal is in the error body, not the status.
The second is that enforcement is not instantaneous. Limit state has to propagate, so spend can overshoot the number slightly before requests start failing. That is fine as an accounting matter and important as an engineering one: the cap is not a precise ceiling you can budget against to the dollar, it is an approximate cliff. Treating it as exact is how teams end up setting it so close to expected spend that ordinary month-end traffic tips them over.
One org, one key, one cap — the thing we stopped doing
The old shape was convenient and it is the default nearly everywhere: one organisation, one API key in the secret store, every workload sharing it. Under that shape a spend cap cannot distinguish between the chat assistant a customer is waiting on and a nightly enrichment job nobody would notice for a week. It only knows the total, so when the total is reached it stops both.
We now split by workload before we set any limit at all, and the split is by blast radius rather than by team or by feature. Three tiers has been enough on every platform we have done this on:
- 1Revenue-critical, synchronous.A person is waiting and the product looks broken without it. Its own project, its own key, its own cap set generously above the worst month you have ever had. This tier should be the last thing that ever fails and the smallest share of spend — if it is not the smallest share, that is the finding.
- 2Asynchronous and deferrable. Enrichment, summarisation, classification, backfills. Nobody is watching in real time, so the correct failure is to stop and resume later. This tier gets a tight cap on purpose, because it is the one that runs away.
- 3Experimental and internal. Evals, prototypes, the notebook someone is iterating in. Capped low enough that a bad loop costs a coffee, and structurally unable to touch the production ceiling.
The projects are cheap to create. The keys are the actual work: on the platform where this bit us, one key was in four services, and pulling it apart took longer than everything else combined. That is normal, and it is worth knowing before you quote it.
Once the split exists, the two error codes stop being trivia and start being triage. A project-scoped failure means one workload exhausted its own allowance while everything else is healthy — usually an incident about that job, not about the platform. An org-scoped failure means every project is out, and the only thing that matters is whether tier one has something to serve.
“We set the cap to protect the client from a runaway job. It worked. The runaway job stopped, and so did the product.”
The last line is the one worth arguing about internally. Deciding in advance what the product does with no model at all is a product decision, not an engineering one, and it is much easier to make in a planning meeting than at 16:12 on a Tuesday. For a support assistant it was keyword search over the same help centre with a one-line notice. For a document tool it was hiding the summarise button rather than letting it fail. Both took under a day to build, and both were only built because someone had been made to answer the question while nothing was on fire.
Degradation you have not fault-injected does not exist
Every team we have said this to agrees immediately and then does not test it, because the failure is annoying to reproduce: you cannot exhaust a real budget on demand, and nobody wants a staging project that has to actually burn money to enter the state you are testing. So the handler gets written, reviewed, merged, and never once executed before the day it matters.
The fix is unglamorous. Inject the error at the client boundary. A flag in the SDK wrapper that makes the next N calls throw a synthetic 429 carrying each spend-limit code in turn, plus a mode that fails only one project, gets you the whole matrix in an afternoon. We run it in CI on the revenue-critical path and manually in staging for the rest.
What that turns up is rarely the code you wrote deliberately. On the first platform we did this on, the degradation path itself worked and three other things did not: a queue consumer that treated the parked jobs as poison and dead-lettered them, a client-side retry that ignored the server's decision and re-fired the same request four times, and a status page component that read model availability from a health check which knew nothing about billing and therefore stayed green throughout. The last one is the one that mattered, because it meant support told customers everything was fine.
Set the alerting off thresholds below the cap rather than at it, and make the warning route to whoever can act on it. A notification that spend has reached the ceiling is not an alert, it is a receipt. Seventy percent with eight days remaining is an alert, because someone can still choose between raising the limit, pausing a job, or letting the tight tier stop on purpose.
What we do before turning a cap on now
- Split projects by blast radius — synchronous and revenue-critical, deferrable, experimental — and give each its own key before any limit is set.
- Write down what the product does with no model access at all, per surface, and get a product decision rather than an engineering guess.
- Distinguish spend-limit 429s from rate-limit 429s at the client boundary, because retrying the first one is guaranteed to be wrong.
- Fault-inject both error codes in CI on the revenue path, and check what the queue, the client retry and the status page do, not just the handler.
- Set warning thresholds well below the cap, routed to someone with authority to raise it, and treat reaching the cap itself as an incident rather than a notification.
- Reconcile provider spend against per-feature attribution monthly, so the tier that is quietly consuming the budget is visible before it is the thing that trips the switch.
If you only change one thing this week
Find out which single workload would take the product down if it ran away, and move it into its own project with its own key and its own limit. That one change converts an outage into a bad night for a batch job, and it is a couple of hours of work on most codebases.
Then ask the question that started all of this: if every model call returned an error for the next hour, what would a customer see? If the answer is a spinner, an error toast, or a support ticket, the cap is not protecting anyone yet. It is just deciding when the outage happens — and it will pick a worse moment than you would have.
