Before You Buy AI Forecasting, Make It Beat a Simple Rule

Tools & Technical Tutorials

28 October 2026 | By Ashley Marshall

A clever forecast is worthless if last year's numbers would make the same decision for less. Make the model prove its value against a simple rule first.

Start with the decision, not the model

A forecast is only useful if it changes a decision. That sounds obvious, yet many AI forecasting pilots begin with a demonstration of charts rather than a record of what the business would do differently. A retailer might need a weekly buying decision, a service firm a staffing rota, and a manufacturer a reorder point. Those are different problems even if each supplier calls its product predictive AI. Before purchasing anything, write down the decision owner, the decision date, the cost of being wrong in either direction, and the action available at that date. A highly accurate prediction that arrives after the purchasing window has closed has no operational value.

The pressure to move quickly is real. The Office for National Statistics reported in its October 2026 Business Insights bulletin that 30% of UK businesses used at least one AI technology in September 2026, up from 9% when it first asked in September 2023. But adoption is not evidence that a particular forecasting model improves a particular decision. The same bulletin reports that 72% of businesses expressed some concern about energy prices and 73% about fuel prices in late September. These are reminders that costs and demand drivers move. A model fitted to a calm period can look impressive and then miss the shift that matters most to the finance team. See the ONS bulletin and methodology.

What this means in practice is to define one decision and one horizon before choosing a tool. For example: each Thursday, how many units of a slow-moving product should we order for delivery in three weeks? Decide how you will score excess stock, missed sales and working capital. Keep a record of the human decision currently made and the information the decision maker had at the time. Without that record you cannot tell whether AI improved judgement, merely whether its chart looks sophisticated. The first deliverable is a one-page decision specification, not a model procurement shortlist.

Build the simple comparison first

A naive baseline is a deliberately simple forecast that a reasonable manager could produce without AI. It might be last week's sales, the same week last year, or the average of the previous four comparable weeks. For a highly seasonal business, last year may be better than last week. For a newly launched service, a rolling average may be more honest than a seasonality pattern that does not yet exist. Choose the rule before you see the test results, record why it fits the business, and keep it fixed during the first evaluation. The AI proposal must beat that rule on the outcomes that actually cost money.

Run a backtest using only information that genuinely existed at each historical decision date. If a September forecast quietly includes an October stock adjustment, a revised CRM field or a promotion outcome entered later, you have leaked the answer into the test. This is a common route to attractive but misleading accuracy. Work through consecutive historical weeks rather than random rows: train on earlier periods, predict the next period, move the window forward, and repeat. Include quiet weeks, promotions, supplier delays and exceptional periods. Do not remove the difficult weeks merely because the model looks worse. Those are often the weeks when the business most needs a useful warning.

Compare at least three lines: the simple rule, the current human process, and the proposed AI model. For each, show absolute error, costly over-forecasts, costly under-forecasts, and the decisions each would have prompted. A model can reduce average error yet increase stockouts on the few products that matter most. Conversely, a simple rule may be entirely adequate for stable lines while a better model earns its place on volatile lines. What this means in practice is that selective deployment can be better than a company-wide licence. If the model cannot clear the baseline after an honest test, improve the data or decision process before paying for more complexity.

Use an error budget that a finance director recognises

Forecast accuracy is not a single number. Suppose two methods each miss demand by 100 units across a month. One over-orders low-value stock that can be sold later; the other repeatedly under-orders a high-margin item and loses customers. The nominal error is equal, but the commercial consequence is not. Ask finance and operations to put a provisional pound value on a missed sale, emergency delivery, wasted labour hour, expired stock and avoidable storage. State the assumptions openly. If a value is uncertain, show a range rather than manufacturing precision. Then compare decisions made using each forecast against the same cost rules.

A practical pilot can start with one product family or one operational queue, perhaps 8 to 12 weeks of live shadow forecasting after the historical backtest. During shadow mode, the model advises but does not automatically order stock or set shifts. Record what the manager would have done, what the model recommended, what was actually done, and what happened. Predefine the improvement needed to justify roll-out, such as fewer urgent orders without an unacceptable rise in unsold inventory. These example thresholds must come from the organisation's economics, not a vendor case study. The cost of preparing data, monitoring drift, reviewing exceptions and training staff belongs in the return calculation too.

London City Hall announced a £12 million support package in September 2026 aimed at helping small businesses improve productivity through AI readiness assessment, mentoring and tailored adoption guidance. That is a useful signal about the shape of a sensible adoption programme: assess readiness and operational value before treating technology as the result. It is not proof that a forecasting model will save your firm money. See City Hall announcement. The counterargument is that a business might miss an opportunity by testing slowly. That is fair if the decision window is short. The answer is a tightly scoped shadow test with a deadline, not a blanket purchase without a comparison.

Fix the data and ownership boundaries

Forecasting failures often begin upstream. Point-of-sale data can confuse returns with new sales. CRM opportunity values may represent ambition rather than committed revenue. Staff may enter a service completion date days after the work happened. A model cannot repair these meanings by being more powerful. Give each input a named owner, a definition, a refresh cadence and a rule for corrections. Mark exceptional events such as closures and one-off promotions in a way the future team can understand. Keep a version of the dataset used for each forecast so that a surprising decision can be reconstructed later.

If personal data is involved, test whether it is necessary at all. Weekly product-level demand may not require customer identities. Workforce forecasts may need aggregated capacity rather than named employee performance histories. The ICO's October 2026 update says ten foundation model developers made or committed to data protection improvements, while its agentic AI work examines accountability, safeguards and lawful data use. That report is about foundation models and agents, not an endorsement of any forecasting product. The relevant buyer lesson is narrower: supplier promises do not replace your own data protection assessment or a clear account of who can access the information. Read the ICO update.

What this means in practice is to create an input register before connecting software: source system, field, purpose, retention rule, access group, quality check and export route. Ask the vendor whether customer data trains shared models, where processing occurs, what audit logs are retained, and how data can be deleted. Have a human sign off consequential changes until the forecast has a reliable live record. If an automated reorder or staff allocation could materially affect people, customers or contractual commitments, the approval boundary needs to be explicit. Neither a dashboard nor a model confidence score is a substitute for assigning responsibility.

Train decision makers to challenge the forecast

The hardest skill in a forecasting programme may be knowing when not to trust the output. A category manager who spots a one-off corporate order, a transport delay or a new competitor can possess information absent from the training data. Rather than treating that intervention as disobedience, capture it as an override with a reason. Review overrides monthly: were they consistently helpful, or did they merely make people feel more comfortable? Both people and models can be wrong. A strong process compares them without assuming either side must always win. Train managers to ask what changed since the model's last run and which input is driving the recommendation.

Skills England's October 2026 annual report argues that workers need cross-cutting skills including critical thinking and analytical skills as AI adoption accelerates. Its analysis also says more than a quarter of job vacancies are hard to fill because of skills shortages. Those are national observations, not a forecast of your own headcount. For a business pilot, the useful implication is to make critical challenge part of the operating model rather than buying a dashboard and assuming people will know how to interrogate it. The report is available from Skills England.

Give the forecast owner a short weekly review: did the inputs arrive, has the business context changed, where is uncertainty highest, and did last week's recommendation help? Escalate if an error threshold is breached or a major new event appears. Revisit the naive baseline each quarter, because a model that once justified its cost can fall behind a simple rule after a business changes. The counterargument is that this sounds like too much process for a small firm. In fact, a five-minute review and a simple spreadsheet may be sufficient. The point is to preserve a clear decision loop. If the process costs more than the benefit it creates, stop or simplify it.

Frequently Asked Questions

What counts as a naive forecasting baseline?

A fixed, simple rule such as last week, the same week last year or a rolling average. Choose it before evaluating the AI model.

How long should a shadow forecasting pilot run?

Start with enough cycles to include normal variation. An 8 to 12 week live pilot is a practical starting point for weekly decisions, but seasonal businesses may need longer.

Should we measure accuracy or financial return?

Both. Accuracy diagnoses performance, while the pound cost of over- and under-forecasting shows whether the result improves the decision.

Can a model win on average and still be worse for the business?

Yes. It may improve common low-value items while making costly errors on high-margin or critical lines. Segment the result.

Do small firms need specialist forecasting software?

Not necessarily. A spreadsheet baseline and a clear decision record may be enough. Buy a specialist tool only when it proves additional value.

What data should we give a forecasting supplier?

Only the fields needed for the stated purpose. Prefer aggregated data where possible and document access, retention, processing and deletion.

Should AI forecasts automatically place orders?

Not at the start. Run a shadow pilot and define a human approval boundary before allowing consequential actions.

When should we stop using the model?

If it no longer beats the simple baseline on the agreed business costs, or if monitoring costs exceed the benefit, simplify or stop the system.