AI Agent Evaluation Is Becoming A Pre-Production Requirement For UK Businesses

Most UK businesses deploying AI agents this year have never run a single formal evaluation against them before going live.

For most of the last two years, testing an AI agent meant a developer running it against a handful of prompts, eyeballing the outputs, and shipping if nothing looked obviously wrong. That approach does not survive contact with an agent that can send emails, update customer records, or approve transactions on its own. Gartner's Anushree Verma put the scale of the problem plainly in June 2026: 99% of organisations do not evaluate any AI agents before they go into production. That is not a rounding error. It means the overwhelming majority of agents now handling live customer conversations, internal approvals, and data lookups were never systematically checked for how they fail, only whether they occasionally work.

The reason this has moved from an engineering concern to a board concern is straightforward: agents fail differently to traditional software, and often invisibly. Microsoft, releasing its own open-source evaluation framework in mid-2026, described the pattern well - agents drift from policy, produce unsafe outputs in edge cases, and behave differently in production than they did in testing. A chatbot that occasionally gives a slightly wrong answer is an annoyance. An agent that quietly starts approving refunds outside policy, or leaking one customer's data into another's session, is an incident. For UK firms already building AI assurance evidence and audit trails into procurement, evaluation is the missing layer that proves those controls actually hold under real conditions, not just on paper.

The evaluation tooling market matured fast enough this year to split cleanly into two tiers. Commercial platforms provide managed infrastructure - trace ingestion, annotation queues, dataset versioning and enterprise SLAs - in exchange for a subscription. Braintrust closed an $80m Series B in February 2026 at an $800m valuation, with customers including Notion, Replit, Cloudflare and Ramp, and it is the only major platform offering sandboxed Python custom scorers rather than fixed metric templates. LangSmith remains the default choice for teams built on LangChain and LangGraph, backed by 80 million monthly downloads of the underlying library. Helicone takes a different approach entirely, wrapping evaluation around your existing API calls as a proxy with no SDK changes required, which makes it the fastest to bolt on to an agent that is already live.

Then there is Promptfoo. It was the security and red-teaming leader in this space, used by more than a quarter of Fortune 500 companies for pre-deployment testing across 50-plus vulnerability categories, right up until OpenAI acquired it on 9 March 2026 for a reported $86m. That acquisition is the single most consequential event in this market this year, and it changes the calculus for any UK business currently using or considering Promptfoo, which we come back to below. On the open-source side, three standards now cover most of what a UK team needs without a subscription at all: OpenAI Evals for registry-style benchmark testing, DeepEval for pytest-native unit-level checks that slot into a normal CI pipeline, and Inspect AI, built and maintained by the UK AI Security Institute.

Buried inside a market dominated by Silicon Valley funding rounds is a framework built by a UK government body that most business buyers have never encountered. Inspect AI was open-sourced in May 2024 by what is now the AI Security Institute (AISI, formerly the AI Safety Institute), and it has since attracted contributions from more than 50 organisations, including other national AI safety institutes and frontier model labs. In November 2024 AISI extended it with Inspect Evals, a companion repository of community-contributed benchmark evaluations covering coding, mathematics, cybersecurity, safeguards, reasoning and general knowledge, built in partnership with Arcadia Impact and the Vector Institute. By May 2026 the framework had reached version 0.3.225 and is now regularly cited alongside the big commercial names in independent comparisons of agent evaluation tooling.

The practical appeal for UK firms is threefold. First, it is genuinely free and self-hosted, so evaluation data - which often includes real customer transcripts or synthetic versions of sensitive workflows - never leaves your infrastructure. Second, because it was built by a UK public body specifically to assess frontier model capabilities and safety, it is a credible, citable reference point in a procurement pack or a regulator conversation in a way that a vendor's own marketing claims are not. Third, it runs from a standard Python package with VS Code support, so it does not require the kind of platform lock-in that a hosted SaaS eval tool tends to create over time. For firms already leaning towards sovereign or self-hosted AI infrastructure for other reasons, Inspect AI is the natural evaluation layer to sit alongside that decision, rather than adding another external vendor to the supplier register.

The instinct when a business first hears 'evaluate the agent' is to think of it as a bigger version of QA testing - run more test cases, check more outputs are correct. That is necessary but not sufficient, and it is exactly the gap Microsoft's ASSERT framework and the wider 2026 tooling landscape are trying to close. A proper evaluation programme for an agent that touches live business processes needs to check at least four distinct things: task accuracy (did it do the thing correctly), policy adherence (did it stay within the rules it was given, including in edge cases nobody explicitly tested for), safety under adversarial input (what happens when a user, deliberately or not, tries to get it to do something it should not), and behavioural drift over time (does the same agent, on the same task, still behave the same way three months and two model updates later).

That last point matters more than most UK businesses currently appreciate. An agent evaluated once at launch and never again is not evaluated in any meaningful sense - it is evaluated at a single point in time that stops being representative the moment the underlying model is updated, a connector changes, or usage patterns shift. Gartner's forecast on this is stark: by 2029, more than 75% of domain-specific agents deployed in regulated industries without proper agentic simulation and evaluation will fail to deliver value. Forrester's current data shows 45% of organisations already have agents live and a further 25% piloting them, which means most of that eventual failure is already being built into production systems right now, this year, without anyone measuring for it.

OpenAI acquiring the market's leading independent security and red-teaming eval tool raises a question UK buyers cannot responsibly ignore: can a testing framework now owned by one of the model vendors it evaluates remain genuinely neutral? Promptfoo built its reputation on catching vulnerabilities and failure modes across models from every provider, including OpenAI's own. That independence was the product. Once the acquiring company is also the entity whose models are being graded, even if nothing about the tool's behaviour changes on day one, the incentive structure has changed, and procurement teams evaluating multi-model or non-OpenAI stacks now have a legitimate reason to ask harder questions before continuing to rely on it as their sole evaluation layer.

This is not a reason to panic or to rip out existing Promptfoo deployments overnight. It is a reason to do what good procurement discipline already recommends for any single-vendor dependency: avoid having your only evaluation layer be owned by the same company whose model you are trying to keep honest. UK firms running multi-model strategies, which are increasingly the default rather than the exception, have a practical answer available in the open-source tier - pairing a vendor-neutral framework such as Inspect AI or DeepEval for the core safety and policy checks with whichever commercial platform suits day-to-day trace monitoring. That combination keeps the ownership of the 'was this agent actually safe' judgement outside any single model vendor's balance sheet.

Translating this into something a UK business can actually action does not require hiring a research team. It requires a short, non-negotiable checklist that sits alongside the existing sign-off process for any agent handling customer data, money, or system access. Before any agent goes live, require: a written evaluation dataset covering both normal tasks and known edge cases specific to your workflow, not a generic benchmark; a documented pass threshold agreed before testing starts, not adjusted after seeing the results; a red-team pass checking adversarial prompts and prompt injection attempts, particularly for anything with browser or connector access; and a re-evaluation trigger tied to model updates, connector changes, or a defined time interval, whichever comes first.

The tooling choice matters less than the discipline. A small UK business with one customer-facing agent can run a meaningful evaluation suite using DeepEval inside an existing CI pipeline for close to zero additional cost, spending a day building a representative test dataset from real support tickets or transcripts. A larger firm running multiple agents across departments will get more value from a commercial platform's dataset versioning and annotation workflow, paired with an open-source framework like Inspect AI for the safety-critical checks that should not depend on a single vendor's continued goodwill. Either way, the evidence this produces - the dataset, the pass criteria, the results, the re-test schedule - is exactly what an assurance data room or an AI audit trail needs to contain. Evaluation is not a separate initiative from the governance work UK firms are already doing. It is the missing proof that the governance actually works.