Our release checklist for AI features had a line in it that said "evals green." It had been there for about a year, it had blocked two releases, and everybody treated it as the serious item on the list. On the Tuesday we read the deprecation notice, we sat down and worked out what that line actually pointed at: a dashboard we did not own, a set of graders defined in a web form, and a score history that existed in exactly one place and could not be copied out of it.
Nothing was down. Nothing had failed. No alert fired, and the suite was still passing that afternoon. We had just discovered that we could not say what passing had meant.
OpenAI's deprecation page puts two dates on its Evals platform: it goes read-only on 31 October 2026, and the dashboard and API switch off on 30 November 2026, with Promptfoo named as the place to go. That is a generous runway for sunsetting a product and a short one for moving a release gate, because a gate is not a tool you swap over a weekend. It is a thing other people's sign-offs hang on. So we audited ourselves, across every project with an AI feature in production. The results are below, and the headline is not the one we expected going in.
Read-only is the real deadline, not the shutdown
Teams read a two-date deprecation and plan against the second date, because that is the one where things stop working. For an eval harness the first date is the one that matters. Read-only means you can still look at yesterday; you cannot add a case for the bug you shipped this morning. The moment your suite stops accepting new cases is the moment it stops describing your product, and it will keep returning a confident green number while it does that.
This is not really a story about one vendor's sunset, and we would rather not have learned it from a calendar. Every managed eval product has this shape: the cases and the graders and the run history sit inside someone else's lifecycle decisions, and the thing you built on top of them is a gate that your own releases cannot pass without. Pricing changes, default model swaps and plan restructures do the same damage as a shutdown, just more quietly.
The test cases exported fine. The baselines did not.
We assumed the migration risk was the test cases, so that is what we checked first, and it was a non-event. Prompts and expected outputs are rows. A few hundred of them came out as CSV in an afternoon, went into the repo next to the code they test, and we have been glad of that ever since for reasons that have nothing to do with the deprecation.
The part that did not come out was the history. An eval score on its own is not a measurement of anything. Seventy-eight per cent is not good or bad; it is only interesting against the seventy-four per cent it scored before the prompt change, on the same cases, through the same grader, against the same model version. That comparison is the entire product. And the comparison is the asset that managed dashboards hold most tightly, because it lives in their run storage rather than in anything they hand you as a file.
“An eval score is not a measurement, it is a comparison. Move the harness without the history and you have kept the thermometer and thrown away the chart.”
Practically, this meant the four features we moved all started from a fresh baseline on the new harness. For three of them that was survivable. For the fourth it was expensive: it was the one where we had been slowly improving a retrieval step over several months, and the only evidence that the improvement was real had been a line going up on a chart that we can no longer render. We re-established a baseline by re-running the exported cases against the current build, which gives us today and tells us nothing about June.
If you take one operational habit from this post, take this one: whatever harness you are on, managed or self-hosted, write every run's results to storage you control, on the day you set it up. Not the summary — the per-case outcomes, the grader version, the model identifier, the timestamp. It is a few lines of plumbing and a cheap bucket, and it is the difference between a migration being an afternoon of CSV and a quarter of lost evidence.
Graders are code, and ours were living in a text box
The second finding was more embarrassing than the first. Our graders — the things that decide whether an answer is correct — had been written in a UI by whoever set up the suite, edited in place when they produced a result somebody disagreed with, and never reviewed by a second person. One of them had been changed four days before a release and we only know that because the dashboard showed a modified date. There was no diff, no author, no reason.
A grader is the specification of correctness for your AI feature. It deserves exactly the review a migration script gets. When it lives in a text box it gets none, and when it quietly changes, every score before and after the change is in a different unit while looking like the same number.
| What the gate depended on | Where it actually lived | What a cutover does to it |
|---|---|---|
| Test cases and expected outputs | Vendor project, editable by anyone with dashboard access. | Exports cleanly. The only genuinely portable layer, and the one everyone worries about. |
| Graders and rubrics | Web form, edited in place, no version history beyond a modified timestamp. | Has to be rewritten from reading the form. Semantics shift and you cannot prove they did not. |
| Run history and baselines | Vendor run storage. Visible as charts, not available as a file. | Gone. You restart from today's build and lose the ability to say whether anything improved. |
| Pass thresholds | A number in a settings panel, set once during setup, owner unknown. | Nothing carries it over, so somebody picks it again from memory — usually lower than before. |
| Test data captured from production | Pasted into cases by hand, with no snapshot of the system state it came from. | Cannot be regenerated or replayed. Cases silently rot as the underlying data changes. |
That last row is the one we are still fixing. Several of our strongest test cases were lifted from real production traffic, which is the right instinct, but we kept the prompt and the expected answer and threw away the context that made the expected answer correct. A case built on a document that has since been edited is not a test any more. It is a trap that fails a year later and sends somebody hunting through a model release note for a regression that does not exist.
The green aggregate was the worst thing we found
While we had everything open, we read the actual per-case results on one suite rather than its headline score for the first time in months. The suite was passing at ninety-two per cent. Underneath, the handful of cases covering the one journey that the feature exists to support had been failing for weeks. They were a small fraction of a large case count, and a large case count is very good at absorbing a small catastrophe.
A single aggregate across unevenly important cases is an average of things you do not care about equally, and it will always drift towards the cases that are cheap to write. Nobody did anything wrong here either: the suite had grown by accretion, each new case was reasonable on its own, and no one had ever gone back and asked which of them were allowed to fail.
So we split the suite. There is now a critical set — small, deliberately hard to add to, every case tied to a named user journey — where a single regression fails the build, and a broad set that reports a trend and blocks nothing. The broad set is still useful; it just no longer has a vote on whether we ship. Splitting it took half a day and found a real defect immediately, which tells you how long it had been possible to find.
“A threshold nobody owns is not a gate. It is a number that somebody will lower on a Friday, for a good reason, and never put back.”
Shadow-run both harnesses before you need to
The temptation under a deadline is to stand up the replacement, confirm it runs, and point the checklist at it. We did that on the first feature and it cost us a day of confusion, because the new harness returned a materially different number on cases we had not changed. Our first assumption was that the model had regressed. It had not. The graders were strict in slightly different places, which is exactly what you would expect from a rubric rewritten by hand from a screenshot of a form.
The cutover, in the order that worked
- Export the cases into the product repo first, before touching anything else. They are portable, it is an afternoon, and from that moment the migration is reversible.
- Start writing every run to your own storage on day one of the new harness — per-case outcomes, grader version, model identifier, timestamp. This is the step that makes the next migration boring.
- Rewrite graders in review, as code, with an author on each one. Treat a rubric change as a semver-major event for every score it has ever produced.
- Run both harnesses on the same cases, on the same build, for at least a week. Expect disagreement and go resolve each one — every disagreement is a grader you did not understand as well as you thought.
- Only move the release checklist after the two agree on the critical set. Until then the old gate is still the gate, read-only or not.
- Re-derive the thresholds deliberately, with a named owner, and write down what the number was chosen to protect. Do not carry over a figure nobody can explain.
Shadow-running also settled an argument we had been having abstractly for a year about whether to self-host the harness. Seeing the two side by side made the real trade-off obvious, and it is not the one usually advertised. The managed product was genuinely better at the parts we look at — the charts, the drill-down, the shared view a non-engineer can read. The self-hosted one was better at the part we depend on: it runs in our CI, on our pull requests, with its configuration in the same review as the code it gates, and its results in a bucket we pay for. We ended up keeping the reporting layer of the managed idea and rebuilding it over our own data rather than giving up the gate.
What we would tell a team with five days left
If you are reading this and your evals live in a dashboard with a read-only date on it this week, the honest advice is to be ruthless about the order. You do not have time to do this properly. You do have time to do the two things that cannot be done later.
- Export the cases today, and screenshot every grader— the cases are rows and they come out fast. The graders are prose in a form, and after the read-only date you can still read them, but after the shutdown you cannot. A folder of screenshots is an ugly artefact and it is vastly better than reconstructing a rubric from memory in December.
- Write down what the numbers currently are— not the chart, which you cannot have, but today’s per-case results, the thresholds, and one sentence per suite saying what it is supposed to protect. This is the cheapest possible substitute for a baseline, it takes an hour, and it is the only thing that will let you argue in February about whether something got worse.
- Leave the old gate in place until the new one agrees with it — a read-only suite still runs and still reports. Moving the checklist to an unvalidated harness in order to hit a vendor’s date is how you end up shipping against a green number that nobody has ever checked. The deprecation is a deadline for the tool, not for your release process.
The broader lesson is the one we keep relearning in a different costume. Anything a release cannot proceed without is part of the product, and it belongs where the product lives: in the repository, in review, in CI, with its outputs in storage you can query. We had moved deploys, migrations, secrets and feature flags into that discipline years ago. Evals arrived later, came with an excellent dashboard, and got a pass. A sunset notice is a slightly rude way to be told that, and it was still the cheapest version of the lesson available.
