AI Agent Evaluation Is Becoming A Pre-Production Requirement For UK Businesses

Tools & Technical Tutorials

4 August 2026 | By Ashley Marshall

Most UK businesses deploying AI agents this year have never run a single formal evaluation against them before going live.

Evaluation has quietly become a board issue, not a developer task

For most of the last two years, testing an AI agent meant a developer running it against a handful of prompts, eyeballing the outputs, and shipping if nothing looked obviously wrong. That approach does not survive contact with an agent that can send emails, update customer records, or approve transactions on its own. Gartner's Anushree Verma put the scale of the problem plainly in June 2026: 99% of organisations do not evaluate any AI agents before they go into production. That is not a rounding error. It means the overwhelming majority of agents now handling live customer conversations, internal approvals, and data lookups were never systematically checked for how they fail, only whether they occasionally work.

The reason this has moved from an engineering concern to a board concern is straightforward: agents fail differently to traditional software, and often invisibly. Microsoft, releasing its own open-source evaluation framework in mid-2026, described the pattern well - agents drift from policy, produce unsafe outputs in edge cases, and behave differently in production than they did in testing. A chatbot that occasionally gives a slightly wrong answer is an annoyance. An agent that quietly starts approving refunds outside policy, or leaking one customer's data into another's session, is an incident. For UK firms already building AI assurance evidence and audit trails into procurement, evaluation is the missing layer that proves those controls actually hold under real conditions, not just on paper.

The 2026 landscape: five commercial platforms, three open-source standards

The evaluation tooling market matured fast enough this year to split cleanly into two tiers. Commercial platforms provide managed infrastructure - trace ingestion, annotation queues, dataset versioning and enterprise SLAs - in exchange for a subscription. Braintrust closed an $80m Series B in February 2026 at an $800m valuation, with customers including Notion, Replit, Cloudflare and Ramp, and it is the only major platform offering sandboxed Python custom scorers rather than fixed metric templates. LangSmith remains the default choice for teams built on LangChain and LangGraph, backed by 80 million monthly downloads of the underlying library. Helicone takes a different approach entirely, wrapping evaluation around your existing API calls as a proxy with no SDK changes required, which makes it the fastest to bolt on to an agent that is already live.

Then there is Promptfoo. It was the security and red-teaming leader in this space, used by more than a quarter of Fortune 500 companies for pre-deployment testing across 50-plus vulnerability categories, right up until OpenAI acquired it on 9 March 2026 for a reported $86m. That acquisition is the single most consequential event in this market this year, and it changes the calculus for any UK business currently using or considering Promptfoo, which we come back to below. On the open-source side, three standards now cover most of what a UK team needs without a subscription at all: OpenAI Evals for registry-style benchmark testing, DeepEval for pytest-native unit-level checks that slot into a normal CI pipeline, and Inspect AI, built and maintained by the UK AI Security Institute.

Inspect AI: the UK-built open standard most UK firms have not heard of

Buried inside a market dominated by Silicon Valley funding rounds is a framework built by a UK government body that most business buyers have never encountered. Inspect AI was open-sourced in May 2024 by what is now the AI Security Institute (AISI, formerly the AI Safety Institute), and it has since attracted contributions from more than 50 organisations, including other national AI safety institutes and frontier model labs. In November 2024 AISI extended it with Inspect Evals, a companion repository of community-contributed benchmark evaluations covering coding, mathematics, cybersecurity, safeguards, reasoning and general knowledge, built in partnership with Arcadia Impact and the Vector Institute. By May 2026 the framework had reached version 0.3.225 and is now regularly cited alongside the big commercial names in independent comparisons of agent evaluation tooling.

The practical appeal for UK firms is threefold. First, it is genuinely free and self-hosted, so evaluation data - which often includes real customer transcripts or synthetic versions of sensitive workflows - never leaves your infrastructure. Second, because it was built by a UK public body specifically to assess frontier model capabilities and safety, it is a credible, citable reference point in a procurement pack or a regulator conversation in a way that a vendor's own marketing claims are not. Third, it runs from a standard Python package with VS Code support, so it does not require the kind of platform lock-in that a hosted SaaS eval tool tends to create over time. For firms already leaning towards sovereign or self-hosted AI infrastructure for other reasons, Inspect AI is the natural evaluation layer to sit alongside that decision, rather than adding another external vendor to the supplier register.

What 'good' evaluation actually covers, beyond accuracy

The instinct when a business first hears 'evaluate the agent' is to think of it as a bigger version of QA testing - run more test cases, check more outputs are correct. That is necessary but not sufficient, and it is exactly the gap Microsoft's ASSERT framework and the wider 2026 tooling landscape are trying to close. A proper evaluation programme for an agent that touches live business processes needs to check at least four distinct things: task accuracy (did it do the thing correctly), policy adherence (did it stay within the rules it was given, including in edge cases nobody explicitly tested for), safety under adversarial input (what happens when a user, deliberately or not, tries to get it to do something it should not), and behavioural drift over time (does the same agent, on the same task, still behave the same way three months and two model updates later).

That last point matters more than most UK businesses currently appreciate. An agent evaluated once at launch and never again is not evaluated in any meaningful sense - it is evaluated at a single point in time that stops being representative the moment the underlying model is updated, a connector changes, or usage patterns shift. Gartner's forecast on this is stark: by 2029, more than 75% of domain-specific agents deployed in regulated industries without proper agentic simulation and evaluation will fail to deliver value. Forrester's current data shows 45% of organisations already have agents live and a further 25% piloting them, which means most of that eventual failure is already being built into production systems right now, this year, without anyone measuring for it.

The vendor-objectivity problem the Promptfoo acquisition created

OpenAI acquiring the market's leading independent security and red-teaming eval tool raises a question UK buyers cannot responsibly ignore: can a testing framework now owned by one of the model vendors it evaluates remain genuinely neutral? Promptfoo built its reputation on catching vulnerabilities and failure modes across models from every provider, including OpenAI's own. That independence was the product. Once the acquiring company is also the entity whose models are being graded, even if nothing about the tool's behaviour changes on day one, the incentive structure has changed, and procurement teams evaluating multi-model or non-OpenAI stacks now have a legitimate reason to ask harder questions before continuing to rely on it as their sole evaluation layer.

This is not a reason to panic or to rip out existing Promptfoo deployments overnight. It is a reason to do what good procurement discipline already recommends for any single-vendor dependency: avoid having your only evaluation layer be owned by the same company whose model you are trying to keep honest. UK firms running multi-model strategies, which are increasingly the default rather than the exception, have a practical answer available in the open-source tier - pairing a vendor-neutral framework such as Inspect AI or DeepEval for the core safety and policy checks with whichever commercial platform suits day-to-day trace monitoring. That combination keeps the ownership of the 'was this agent actually safe' judgement outside any single model vendor's balance sheet.

What to require before an agent reaches production, in practice

Translating this into something a UK business can actually action does not require hiring a research team. It requires a short, non-negotiable checklist that sits alongside the existing sign-off process for any agent handling customer data, money, or system access. Before any agent goes live, require: a written evaluation dataset covering both normal tasks and known edge cases specific to your workflow, not a generic benchmark; a documented pass threshold agreed before testing starts, not adjusted after seeing the results; a red-team pass checking adversarial prompts and prompt injection attempts, particularly for anything with browser or connector access; and a re-evaluation trigger tied to model updates, connector changes, or a defined time interval, whichever comes first.

The tooling choice matters less than the discipline. A small UK business with one customer-facing agent can run a meaningful evaluation suite using DeepEval inside an existing CI pipeline for close to zero additional cost, spending a day building a representative test dataset from real support tickets or transcripts. A larger firm running multiple agents across departments will get more value from a commercial platform's dataset versioning and annotation workflow, paired with an open-source framework like Inspect AI for the safety-critical checks that should not depend on a single vendor's continued goodwill. Either way, the evidence this produces - the dataset, the pass criteria, the results, the re-test schedule - is exactly what an assurance data room or an AI audit trail needs to contain. Evaluation is not a separate initiative from the governance work UK firms are already doing. It is the missing proof that the governance actually works.

Frequently Asked Questions

What is AI agent evaluation, and how is it different from testing a chatbot?

Chatbot testing usually checks whether a single response is accurate. Agent evaluation checks a full chain of decisions and actions - whether the agent stayed within policy, handled edge cases and adversarial input safely, and behaved consistently over multiple steps, not just whether one output looked right.

Do we need a paid evaluation platform, or is open source enough?

For most single-agent UK small and mid-market deployments, an open-source framework like DeepEval or Inspect AI run inside your existing CI pipeline is sufficient and free. Paid platforms such as Braintrust or LangSmith earn their cost once you are running multiple agents and need dataset versioning, annotation workflows, and team collaboration at scale.

What is Inspect AI and why should a UK business care about it specifically?

Inspect AI is an open-source evaluation framework built and maintained by the UK AI Security Institute, first released in 2024 and now on version 0.3.225 with over 50 contributing organisations. It runs entirely on your own infrastructure, so evaluation data never leaves your environment, and it carries the credibility of a UK public body rather than a commercial vendor's marketing claims.

Does OpenAI's acquisition of Promptfoo affect UK businesses that already use it?

It does not break anything on day one, but it does mean Promptfoo is now owned by a model vendor it also evaluates. Firms running multi-model or non-OpenAI stacks should not treat it as their sole evaluation layer any longer and should pair it with a vendor-neutral framework for safety-critical checks.

How much does it cost to set up basic agent evaluation?

Using an open-source framework, the main cost is time, not licensing: typically one to two days to build a representative test dataset from real transcripts or tickets, plus ongoing CI compute, which is usually negligible. Commercial platforms range from roughly £30 to £250+ per seat per month depending on scale and features.

What should be in an evaluation before a customer-facing agent goes live?

At minimum: a written test dataset covering normal tasks and known edge cases, an agreed pass threshold set before testing, a red-team pass for adversarial prompts and prompt injection, and a defined trigger for re-testing after model or connector updates.

How often should an agent be re-evaluated after launch?

At every model update or connector change, and at a fixed interval regardless - three months is a sensible default for most business use cases. An agent evaluated only once at launch is effectively unmonitored within weeks, given how frequently underlying models and integrations change.

Is agent evaluation a formal compliance requirement in the UK or EU?

It is not yet a named line-item requirement under UK guidance, but it directly supports the evidence UK firms need for EU AI Act compliance, NCSC AI Cyber Security Code alignment, and any AI assurance data room a board or auditor might ask to see. Treat it as the practical proof behind those governance obligations, not a separate box to tick.