Your AI can keep running while the evidence used to approve it disappears. Moving the test runner is only half the job.

The dashboard is going, but your approval history must stay

Your business may have approved an AI workflow using evidence that lives almost entirely inside somebody else's dashboard. That is convenient until the dashboard closes. The immediate problem is not whether a replacement tool can produce another green tick. It is whether you can still explain what was tested, which version passed, which failures were accepted and who authorised the release. A screenshot of an overall score rarely answers those questions.

OpenAI's current deprecations page lists 30 November 2026 as the shutdown date for the Evals dashboard, API and graders for evaluation workflows. It currently marks the service read-only and recommends migration to Promptfoo. This is a product-specific deadline, not an announcement that every OpenAI API or every evaluation library disappears. Check the exact service your supplier uses before deciding you are unaffected.

The useful business response is to separate two jobs. First, preserve the historical evidence behind decisions already made. Second, establish a working test process for future changes. Completing the second does not automatically complete the first. A fresh run in another system cannot recreate the exact outputs, reviewer comments or exceptions that supported a previous decision.

Imagine a customer-service assistant approved because it correctly refused to disclose another customer's order. If the test record disappears, a later reviewer cannot tell whether that case was checked, whether the source data was realistic or whether a human overrode a failed score. The assistant may still work, but the explanation for trusting it has become weaker.

Appoint an owner for the evidence before assigning a developer to the migration. Ask that owner to identify the last approved release and locate its supporting records. If that takes several conversations and a search through screenshots, treat that as a finding to fix now, not a reason to postpone the exercise.

Moving the test runner does not preserve the decisions made with the old one.

Preserve the decision, not just a collection of scores

Start with a compact evidence pack for each important workflow. It should identify the business task, the version of the application, the model and settings, the prompt, the test cases, the expected behaviour and the scoring rules. Include the actual outputs where retention is justified, the resulting scores and the human decision. Keep the relationship between those items intact. A folder full of unnamed exports is not a usable history.

The most easily lost element is often the explanation of a failure. An aggregate result may show that a release passed despite several unsuccessful cases. Unless the exception notes survive, nobody can distinguish an accepted cosmetic weakness from an unresolved privacy problem. Record the affected case, the person making the decision, the reason, any restriction on use and the date for reconsideration.

This is a recommendation for practical record keeping, not a claim that every UK company has a statutory duty to retain every AI response forever. Decide retention using the purpose, sensitivity, contractual commitments and applicable sector requirements. Do not copy customer names, medical information or access credentials into a general development repository merely because the material appeared in a test. Separate restricted evidence from reusable synthetic test fixtures.

In practice, choose one approved workflow and ask a colleague who did not build it to follow the evidence from release decision back to individual cases. Can they identify the exact question asked? Can they see the expected boundary? Can they understand why the outcome was acceptable? That exercise is more revealing than checking whether a download completed.

Preserve uncertainty honestly. Where the original model snapshot is unavailable or a retrieved document changed, say that exact reproduction is no longer possible. Historical evidence can remain useful without promising bit-for-bit replay. The important distinction is between a documented limitation and a gap concealed by a polished replacement report.

A score without its test cases and approval conditions is an incomplete business record.

Rebuild the checks without pretending the scores are identical

There is an existing open-source route, so most small teams do not need to commission a bespoke testing platform. OpenAI's migration guide describes manually recreating the prompts, provider, cases and assertions in Promptfoo. It explicitly does not depend on an OpenAI Evals export feature. It also says a newly created run is separate from previously completed hosted runs.

That difference matters commercially. Your supplier should not describe a recreated test suite as a complete historical migration unless the historical records have actually been preserved and checked. Ask for a list of what moves, what is recreated and what is unavailable. The deliverable should make those boundaries visible to an ordinary operational manager, not hide them in technical release notes.

Keep the migration comparison narrow at first. Use the same representative cases and the same acceptance intent. Where possible, hold the application and model configuration steady while checking the new runner. Changing the runner, prompt, model and scoring rules together makes it difficult to explain why results differ. If a necessary dependency has already disappeared, record that limitation rather than manufacturing a clean comparison.

Pay particular attention to model-based judges. A judge is another AI component with its own instructions and variability. Review examples where the old and new process disagree, then ask a person familiar with the business task to decide whether either assessment is reasonable. Exact numerical agreement is not the same as sound judgement.

For a hypothetical invoice assistant, a format check might validate that a total is a number. A separate business check must determine whether it is the correct total for the right invoice. A fluent explanation cannot compensate for a wrong account reference. Document those distinct acceptance conditions before setting a new overall pass threshold, and keep critical failures visible rather than averaging them away.

A newly recreated evaluation is a new run, not a recovered historical approval.

A managed service is fine if the evidence remains accessible

The strongest counterargument is reasonable: a small business should not have to operate its own evaluation infrastructure. Managed tools can reduce maintenance, make review easier and give non-technical colleagues a useful interface. The lesson is not to avoid hosted software. It is to avoid making one supplier's interface the only place where your acceptance rules and release decisions can be understood.

Ask the supplier to demonstrate the exit process on a small, non-sensitive sample. Request the test definitions, the scoring logic or an adequate description of it, individual results and approval notes in usable formats. Check whether the relationships between them survive. A PDF summary may be useful for a meeting, while still being insufficient for maintaining the checks in another tool.

There is a relevant UK industry signal. In its 28 September 2026 assurance roadmap event briefing, techUK describes work on professional skills, information access and credible measurement. It identifies the Centre for AI Measurement, led by NPL, as part of that developing ecosystem. This does not certify your particular workflow or create a universal retention rule. It reinforces why access to intelligible evidence matters when another person is asked to assess an AI system.

Government attention is also moving beyond product demonstrations. The 12 October 2026 announcement on AI risks describes independent scrutiny and preparation for incidents involving autonomous agents. These national measures are not an SME testing checklist, but they are a useful reminder that capability claims and operational assurance are different questions.

In practice, keep the operating model proportionate. A modest supplier-managed suite with readable definitions, named approvals and a tested evidence handover can be better than a sophisticated internal system nobody maintains. Buy the convenience, but retain the ability to explain your own decisions.

You can outsource the testing interface without outsourcing ownership of the acceptance decision.

Use the remaining window to prove continuity, not just finish an export

Work backwards from the affected service's deadline, but organise the work around business consequence. Identify which live processes depend on hosted evaluations to approve changes. Then distinguish customer-facing or sensitive workflows from low-risk internal drafting. If resources are tight, preserve and re-establish the checks for the most consequential workflows first. Do not spend the entire window perfecting the archive of an abandoned experiment.

Give each priority workflow three named owners: someone responsible for the technical move, someone who understands acceptable business behaviour and someone authorised to approve continued use. In a small company one person may hold more than one role. The point is clarity about decisions, not an elaborate committee or an extra layer of administration.

Define completion before work starts. The historical evidence must be readable by an authorised colleague. The replacement checks must execute successfully against the intended application. Known critical failures must remain visible. The acceptance owner must review disagreements and approve the new process. Finally, access and retention controls must still make sense after the records move.

Use deliberately failing cases to test the new process. A suite that runs is not necessarily a suite that can stop an unsafe release. Introduce a synthetic wrong customer reference, an unsupported answer or another failure relevant to the workflow. Confirm the check catches it and that the result reaches the person who can act. Avoid doing this against live customer records or allowing the test to send external messages.

After the move, record a baseline with its limitations and a clear cutover date. Keep old and new results distinguishable so future reviewers do not mistake a new scoring method for a sudden improvement in the application. Schedule a small follow-up review after the next real change.

The final management question is simple: if the hosted dashboard disappeared tomorrow, could a colleague still explain why the current AI workflow was approved and demonstrate that the next change will be checked? That is a more useful completion test than the number of files downloaded.

A successful cutover preserves yesterday's explanation and tomorrow's ability to reject a bad change.

Key Takeaways

  • Identify which live approvals depend on hosted Evals before the 30 November 2026 shutdown.
  • Preserve case-level outputs, scoring rules, release versions and exception decisions with appropriate access controls.
  • Use an existing evaluation tool such as Promptfoo rather than building a bespoke runner by default.
  • Review disagreements between old and new scoring before setting a replacement acceptance threshold.
  • Prove a deliberately failing synthetic case can still block a release after the cutover.