Your AI Pilot Needs a Comparison Group, Not Just a Before-and-After Chart

Your team got faster after the AI pilot. Did the AI cause it, or did the work get easier?

A better month is not proof that AI caused it

Your team completes more work after switching on an AI assistant. The dashboard turns green, the supplier calls the pilot successful, and somebody multiplies the improvement across the entire workforce. There is one unanswered question: what would have happened over the same period without the assistant? A quieter inbox, an experienced colleague returning from leave or a new document template could produce a better month too. The commercial risk is not merely an exaggerated presentation. It is buying more licences for an improvement you cannot reliably reproduce.

The UK AI adoption survey published on 29 October 2026 gives useful context. It reports that 32% of businesses used at least one AI technology, compared with 16% in 2025. Improving efficiency and productivity was an adoption driver for 71%. Those findings describe adoption and business perceptions. They do not establish the return your organisation will get from a particular workflow, supplier or training programme.

A before-and-after comparison is a sensible starting description, but it mixes the intervention with everything else changing around it. If you simplify a process and introduce Microsoft 365 Copilot in the same week, you may have evidence that the combined package helps. You do not yet know how much of the improvement requires Copilot. That distinction matters when the renewal invoice arrives, or when another department proposes buying the tool without making the process change.

Write the question before choosing the chart. For example: does AI-assisted preparation reduce staff time per accepted customer proposal, without increasing corrections or slowing approval, compared with our existing process during the same period? That wording identifies the work, outcome, quality limit and comparison. It also leaves room for an honest answer: the pilot may be promising while the contribution of AI itself remains uncertain.

A better month is an observation. An AI effect needs a credible comparison.

Choose comparable work, not your keenest volunteers

A useful comparison group handles similar work at a similar time without the new AI assistance. It should not be a punishment group or an excuse to deny an accessibility adjustment. The point is to understand the intervention, not to rank employees. In a small business, comparable batches of cases may be more practical than separate departments. Allocate eligible low-risk cases between the existing workflow and the AI-assisted workflow using a pre-agreed rule, while retaining the same required human checks.

Where practical, random allocation makes it harder to place easy work in the AI batch and difficult work elsewhere. Where that is not appropriate, match cases on the features that genuinely affect effort: length, missing information, service type, customer complexity and staff experience. Keep the rule stable. A pilot that allows people to move awkward cases out of the AI group will quietly manufacture a success story. Record exclusions and the reasons for them, rather than deleting inconvenient observations.

The HMRC Copilot evaluation published in July 2026 is an older but useful methodological example. It describes 3,000 randomly allocated licences and a primary survey completed by 1,364 users, a 45% response rate. Random licence allocation did not turn the reported time savings into perfect objective measurement: the report also discusses self-reporting and adjustments for participation. Read the method, not just the headline result.

What this means in practice: do not compare your most enthusiastic office manager using AI with an overloaded colleague handling escalations. Either compare like-for-like cases across both workflows, or let staff use both methods in a planned sequence and acknowledge learning and carry-over effects. Involve the people doing the work when defining comparable cases. If a fair comparison is impossible, describe the results as observational rather than pretending that a spreadsheet has removed the uncertainty.

Your keenest volunteers are a useful discovery team, not automatically a fair comparison group.

Measure the whole accepted job and the background change

Choose an outcome that survives contact with the next person in the process. Drafting speed alone is not enough if a manager spends the saved time repairing the draft. For a quotation workflow, measure total active staff minutes from eligible enquiry to an approved, usable quotation. Track elapsed turnaround separately, because waiting for a customer or director is not the same as staff effort. Keep acceptance criteria consistent between the AI and existing workflows, including the checks required before anything reaches a customer.

Here is a deliberately simplified illustration, not a client result. Before the pilot, both comparable groups take 30 active minutes per accepted case. During the pilot, the AI group falls to 24 minutes and the comparison group falls to 27. The raw improvement in the AI group is six minutes, but three minutes also disappeared without the intervention. The additional improvement associated with the AI workflow is therefore three minutes per accepted case. That is the figure to investigate, not automatically a proven causal effect.

This approach is often called a difference-in-differences comparison. Its usefulness depends on assumptions, especially whether the groups would have followed similar trends without the change. Check earlier periods where you have them. A comparison group with a different seasonal pattern, a new supervisor or a different customer mix may be misleading. Small samples and volatile work can leave a result too uncertain to support a rollout decision even when the arithmetic looks neat.

The ECB worker-survey analysis of 26 August 2026 reports median perceived savings of three hours a week among AI users, while stressing that employers need to put freed capacity to use. It is euro area evidence, not a UK performance target. Use external studies to challenge your assumptions, not to replace measurement of your own accepted work. Report quality failures, abandoned cases and uncertain observations alongside the average.

If the existing workflow improves too, do not credit AI with the whole difference.

Test the package you are actually planning to buy

An AI pilot rarely changes just one thing. Staff receive training, managers pay more attention, templates improve and a supplier helps tidy source documents. Those changes may be valuable. The mistake is claiming that every benefit belongs to the model, then budgeting for licences alone. Decide whether you are evaluating an AI tool in isolation or an implementation package that includes process redesign, training and continuing support. Both questions are legitimate, but they require different comparisons and lead to different purchasing decisions.

The strongest practical counterargument is that a small business cannot run a laboratory every time it changes a tool. That is fair. You do not need an elaborate trial for a reversible, low-cost experiment. You do need a proportionate way to avoid fooling yourself before making a larger commitment. Start with one repeatable workflow, an existing spreadsheet and a short record of case type, method, active time, acceptance and corrections. Do not build a custom analytics platform before testing whether your existing records can answer the question.

Keep training and attention reasonably balanced if you want to estimate the additional contribution of the AI tool. If the comparison group gets no process help while the AI group gets daily coaching, report that you tested a supported AI package against business as usual. Log model changes, prompt changes and altered approval rules during the pilot. A system that changes halfway through needs its results separated or explained, not blended into one tidy average.

The Bank of England November 2026 Monetary Policy Report distinguishes modest evidence of realised AI-related productivity gains from larger expectations for the future. That is the right discipline at company level too: observed results, expected benefits and rollout assumptions belong in separate columns. In practice, ask a supplier to state which parts of its proposed service produced the pilot result, which costs continue after launch, and what evidence would change its recommendation. A supplier willing to discuss a disappointing outcome is more useful than one selling certainty.

A supported AI package can be worthwhile without proving that the model caused every gain.

Agree the decision rule before you see the result

Before the pilot starts, decide what result would justify expanding, continuing, changing or stopping it. Set a minimum useful improvement linked to the real workflow, not a borrowed industry percentage. Include a quality condition, an operational owner and a review date. For a small sample, the honest conclusion may be that the range of plausible outcomes is too wide. Extending the pilot with a specific evidence question is different from repeatedly extending it until an attractive result appears.

Keep financial claims separate from measured effort. Three fewer active minutes per accepted case may provide capacity, but it does not automatically reduce payroll. Cash savings require an actual spending change, such as less paid overtime or a reduced external processing bill. Additional output needs evidence of demand and delivery capacity. Do not count the same saved hour once as salary reduction and again as extra billable work. Include licences, support, training, checking and implementation costs when deciding whether the measured improvement is worth buying.

The UK Digital and Data Benefits framework is government appraisal guidance rather than a requirement for an ordinary small-business pilot. Its existence is a useful reminder that benefits need an explicit method, not just a persuasive headline. Keep your own method understandable: what changed, compared with what, for which work, at what cost and with what uncertainty? Preserve the original decision rule beside the results so later enthusiasm cannot quietly rewrite the test.

What this means in practice: take a one-page decision note to the person signing the renewal. Show the concurrent comparison, case mix, accepted-work outcome, failures, full costs and limits of the evidence. Explain whether you tested the tool or the whole service. If you expand, keep a small ongoing check for deteriorating quality or changing workload. For related implementation detail, our guide to private business acceptance tests explains why generic model scores cannot decide whether your actual work is ready. Buy the improvement you can defend, not the chart you most want to believe.

Set the buying rule before the dashboard tells you the story you hoped to hear.