The price was wrong for nine days. A client's best-selling SKU dropped for a weekend promotion, went back up on the Monday, and Google Shopping carried on advertising the promotional number until a shopper screenshotted the ad next to their checkout total and asked which one was real. Nothing in our monitoring had turned red. For that window the sync job reported a 99.6% success rate, which is better than its monthly average.
The missing 0.4% was not noise. Those were calls still routed through Content API for Shopping, which Google sunset on 18 August 2026 and which, since 1 September, has been returning intermittent HTTP 410 Gone to projects without an extension. Our retry wrapper did exactly what it was written to do: it tried again, got a 200, logged a success and moved on. The second attempt succeeded against an endpoint scheduled for full decommissioning in early 2027 — so it succeeded at nothing that matters.
We have now run this cutover on three commerce stacks. The migration part is dull work: new endpoints, new auth scoping, renamed fields, a handful of resources that split in two. What was not dull was discovering that every piece of monitoring we had was built on an assumption a sunsetting API breaks — that a dependency either works or errors. This one does neither. It flickers. And a flickering dependency plus an eager retry equals product state that quietly stops matching what you believe you published.
Deprecation does not fail the way the rest of your stack fails
410 Gone is the clearest status code in the spec. A 500 says try later; a 503 says try later with a hint; a 410 says the resource is gone and is not coming back, do not ask again. Read in isolation, it is a status code that tells you precisely what to do.
Now read it the way it actually arrives during a staged decommission: mixed in with 200s, from the same endpoint, in the same minute, for the same SKU. Traffic is being moved off in slices rather than switched off, so the response you get depends on which slice your request landed in. The code is still a permanent-failure code. The pattern is an intermittent one. Every retry policy, alert threshold and circuit breaker we had was written for the second shape and reasoned about the first.
The 0.4% is the number worth sitting with. No threshold we would ever have set catches it, and no threshold we would ever have set should — at any other time in that integration's life, a 0.4% error rate with successful retries is a healthy system. The rate was never the signal. The signal was that some subset of our writes was landing nowhere while being counted as landed, and a rate cannot express that because a rate has already thrown away the thing that distinguishes the two.
The six producers were the finding nobody had budgeted for. We had assumed one sync service owned the merchant account, because that is what the architecture diagram said. What we found was the sync service, a nightly price job written before it, a local-inventory script running on a different server, a third-party feed tool somebody in marketing still had a login for, a Shopify app that pushed on product update, and a staging environment that had been pointed at production credentials in 2024 and never moved back. Four of the six were still on the sunsetting endpoint. Two of them nobody on the current team had written.
What we stopped doing
Most of what went wrong here was not a missing tool. It was four reasonable habits that stop being reasonable the moment a dependency enters a decommission window, and we have deleted all four from how we run these integrations.
Four monitors we retired, and what replaced each one
- Success rate on the sync job. Replaced by a drift count: rows where what the platform reports differs from what we last published. A rate describes requests; drift describes state, and only one of those is what a customer sees.
- Blanket retry with backoff. Replaced by per-status classification, with 410 as terminal. Retrying a 410 is not resilience, it is laundering — it converts the one honest signal you were given into a success line in a log.
- "Did the call return 2xx?" as the definition of published. Replaced by a read-back: we ask the API what it now holds for that offer and compare field by field. A write that returns 200 to a deprecated resource is a well-formatted no-op.
- One dashboard per integration. Replaced by one ledger per merchant account, because the account is the shared resource and the integrations are just the writers. Five dashboards all showed their own producer as healthy, which was true, and collectively missed that a sixth existed.
- Treating the cutover as done when the new client library shipped. Replaced by a dual-run window with a daily diff, held open until the old endpoint's traffic reaches zero in the platform's own logs rather than in ours.
The second one caused the most argument internally, so it is worth defending properly. Our retry wrapper was good code. It had jitter, it had a cap, it had a dead-letter queue, it had been correct for four years. It failed here for a reason that has nothing to do with its quality: it was built on the premise that a failed request and a successful retry describe one event with a happy ending. During a decommission, they describe two different endpoints, and the happy ending belongs to the one that is going away.
“A dependency being decommissioned does not go down. It goes quiet in slices, while every dashboard you own reports green — and your retry logic is the component turning the evidence into a success.”
The monitor is a diff, and it runs against the platform's answer
The replacement is unglamorous and took one engineer about three days. Before any write, record intent: offer id, feed, field, value, timestamp, which producer is asking. After the write, and then again on a schedule, read the platform's current state for that offer and store it. The monitor is the difference between those two tables, expressed in rupees at risk and in how long the oldest disagreement has been sitting there.
Two details in that query earned their place the hard way. The fifteen-minute grace period, because without it every normal propagation delay reads as drift and the monitor gets muted within a week. And IS DISTINCT FROM rather than an inequality, because the most common real disagreement we found was a field the platform reported as NULL where we believed we had sent out_of_stock — and a plain comparison treats that as equal by refusing to answer.
Grouping by producer is what turned the query from a monitor into an argument. It named the four producers still on the old endpoint, in order of money at risk, which is a far easier conversation to have with a client than a request to go and audit their integrations.
Alongside the diff, we rewrote how responses are classified. Four rules, and the third one is the one that would have caught our nine-day price on day one:
- 410 is terminal. No retry, no backoff, no dead-letter queue. Write a drift record naming the producer and the endpoint, and raise it as a migration task rather than an incident.
- 429 and 503 retry, with a cap on the window rather than the attempts. Attempt caps make a slow degradation look like a healthy system doing its job. A wall-clock cap turns it into something a human reads.
- A 200 that follows a 410 for the same offer is its own category. Not a success, not a failure — a _split_, and the single highest-value event in the whole log. Every one we found pointed at a producer we did not know we had.
- Absence is not a status code. An offer that no producer has touched for a week emits nothing at all. Nothing is exactly what a dropped update looks like, which is why the diff has to be driven by the catalogue and not by the request log.
Never let the comparison live inside the job that does the writing. If the sync service both publishes and judges whether publishing worked, a bug in its view of the world hides itself perfectly. Ours runs as a separate reader with its own credentials, on its own schedule, and it is allowed to disagree with the thing it is watching.
Two of the three cutovers found genuine money sitting in the diff within the first week — stale availability on roughly 2,000 offers in one case, a rounding mismatch on EU prices in the other. Neither had produced an error in any log, because neither was ever an error. They were writes that went somewhere that no longer counted.
If you only change one thing this week
Write the intent ledger. Not the migration, not the new client library, not the dashboard — the table that records what you believe you published, before you try to publish it. It is a day's work on most stacks, it is independent of which API version you are on, and it is the only thing in this post that keeps paying after the cutover is finished, because every future integration gets the same question asked of it: does the platform agree with us, and since when.
The deeper habit we are trying to break is reading status codes as though they described outcomes. They describe requests. Your customers experience state. For four years those two were close enough to the same thing that the difference never cost us anything — and then an endpoint started answering 410 to some of our traffic and 200 to the rest, and the gap between them was a wrong price, running for nine days, found by a shopper with a screenshot.
