AI Usage Telemetry Is Now The Missing Link Between Adoption And ROI

Tools & Technical Tutorials

14 August 2026 | By Ashley Marshall

Most AI programmes can now show usage. Far fewer can show whether that usage changed the work, reduced risk, or paid back the business case.

Adoption Has Outrun Measurement

The headline story is that AI adoption is moving quickly. The more useful story is that most organisations still cannot tell whether adoption is becoming operational capability. The Office for National Statistics reported that use of at least one AI technology among UK businesses with 10 or more employees rose from around 12% in late 2023 to around 35% by June 2026. That is a material shift, but the same ONS article also found that the average number of AI technologies used per adopting business moved only from around 1.4 to 1.6 over the same period. In other words, many firms have started using AI, but the depth of adoption is still thin.

That is exactly why usage telemetry matters. Counting licences, active users or prompt volume tells you that people have found the tools. It does not tell you whether a claims handler resolved more cases, whether a sales team reduced admin time, whether a support assistant introduced privacy risk, or whether a finance analyst simply moved work from one screen to another. For UK leaders, the important question is no longer whether AI exists in the business. It is whether the business can see how AI changes work.

What this means in practice is simple: every serious AI workflow needs an instrumentation plan before scale. That plan should define which events are captured, which outcomes are measured, which costs are attributed, which risks are flagged and which human decisions remain accountable. Without that layer, AI governance becomes anecdotal and ROI becomes a quarterly debate built on weak evidence.

The Dashboard Needs To Track Work, Not Curiosity

The easiest AI dashboard to build is also the least useful: users, tokens, requests and spend. Those metrics have their place, but they mostly describe curiosity and consumption. A better dashboard tracks the work itself. For a customer service assistant, that might mean first-contact resolution, escalation rate, manual correction rate, policy exception count and average handling time. For a procurement agent, it might mean supplier checks completed, missing evidence found, human approvals requested, blocked actions and decisions reversed. For an internal coding assistant, it might mean cycle time, defect leakage, security findings and review effort.

The technical pattern is familiar from software operations. Teams already monitor latency, errors, saturation and cost for applications. AI workflows need a similar operating view, but with additional fields that reflect model behaviour and human oversight. That includes model name, prompt template version, retrieval source, tool calls, token usage, response latency, safety filter events, confidence or evaluation scores, reviewer identity, final disposition and business outcome. The OpenTelemetry community has been pushing towards common generative AI semantic conventions so teams can capture model attributes, usage and latency in a more portable way, even though wider evaluation and business outcome measurement still need local design.

The counterargument is that this sounds heavy for early AI pilots. It can be, if the organisation tries to build a full observability platform before proving value. The better approach is a narrow event schema for each workflow: five to ten events that describe the journey from request to outcome. If the workflow later scales, that instrumentation becomes the spine of ROI reporting, incident review and vendor comparison.

Security And Accountability Depend On Visibility

Telemetry is not only a finance tool. It is becoming a security and accountability requirement for agentic systems. The National Cyber Security Centre's 2026 guidance on agentic AI is blunt: agents can access data sources, remember context, make decisions, use tools and take actions in pursuit of a goal. That creates wider access, less predictable behaviour and faster failure modes than a standard chatbot. The NCSC advises organisations to start small, use agents for low-risk tasks, apply least privilege, monitor behaviour, threat-model deployments and plan for incidents.

Those recommendations are difficult to implement without useful logs. If an agent updated a CRM record, changed a supplier field, emailed a customer or triggered a payment workflow, the organisation needs to know what instruction led to the action, which data the agent used, which tool permissions were active, which policy checks ran, who approved the action and whether the result was later corrected. This is not about recording everything forever. It is about capturing enough evidence to understand what happened and improve the control design.

The practical implication for UK firms is that security, risk and operations teams should agree the telemetry contract before connecting agents to live systems. That contract should include least-privilege permissions, short-lived credentials where possible, traceable tool calls, alerting for unusual behaviour and a clear owner who can pause the workflow. The NCSC says humans remain accountable for the decision to deploy an agent, the access it was granted, the safeguards around it and the consequences of its operation. Telemetry is how that accountability becomes observable rather than theoretical.

Regulatory Confidence Needs Evidence, Not Intent

UK regulators are also moving towards clearer expectations around AI evidence. In May 2026, the Information Commissioner's Office said its 2026/27 work would include further activity to build consumer trust in AI innovation and give businesses greater regulatory certainty on how data protection law applies to AI development and deployment. It specifically pointed to an AI code of practice, dedicated guidance on agentic AI and support for consumers navigating increasingly personalised AI systems.

That matters because many AI rollouts touch personal data long before they feel like regulated systems. Meeting summaries include employee opinions. Sales assistants process prospects' personal data. Recruitment tools can influence candidate screening. Customer service agents may infer vulnerability, complaint severity or financial distress. If the business cannot show what data was processed, why it was needed, what the AI produced, what humans reviewed and what decision followed, it will struggle to explain the system when challenged.

What this means in practice is that AI telemetry should include data protection fields where personal data is involved. Teams should log data source classes, retention category, lawful basis reference, automated decision flag, human review status, model or supplier used, and whether the output was accepted, amended or rejected. The goal is not to turn every product team into a legal department. The goal is to make the evidence routine enough that privacy, security and business owners can review the same facts. For most UK businesses, that shared evidence layer will be more useful than a thick policy document that nobody can connect to live behaviour.

Standards Will Make Telemetry Easier To Buy

One reason leaders hesitate is that AI tooling still feels fragmented. Copilot dashboards, model gateway logs, CRM automation history, vector database traces and finance reports often live in different places. That fragmentation is real, but it is not a reason to wait. The UK government's Digital Standards Strategy for 2026 to 2030 argues that digital standards provide guidance and confidence for businesses, improve interoperability, complement regulation and help new technologies such as AI scale safely. It also notes that standards have strong economic importance, citing analysis that around 23% of UK GDP growth since 2000 can be attributed to standards, and that the digital and technologies sector contributed an estimated 207 billion pounds in GVA in 2023.

For AI telemetry, standards matter because they reduce translation work. If models, gateways, orchestration layers and observability tools use consistent names for model requests, token usage, latency, tool calls and errors, buyers can compare systems more easily and avoid being trapped in supplier-specific dashboards. That is especially important for UK firms trying to run multi-model strategies across Microsoft, Google, OpenAI, Anthropic, AWS, Azure or private models.

The common misconception is that standards are a compliance topic rather than an implementation accelerator. In practice, standards are often what make pragmatic buying possible. A business does not need to wait for perfect maturity. It can start by requiring suppliers to export logs, support common telemetry formats, document event schemas, separate personal data from operational metrics and provide APIs for evidence review. Those requirements belong in procurement now, before AI usage spreads into too many disconnected systems.

Start With A Small Evidence Layer

The sensible starting point is not a grand AI command centre. It is a small evidence layer for the highest-value or highest-risk workflows. Pick one workflow where AI is already being used or is about to move from pilot to production. Define the business outcome, the user journey, the model interactions, the tool permissions, the review points, the exception rules and the cost unit. Then instrument enough events to answer four questions: what happened, what did it cost, what changed, and who was accountable?

A practical first version might include a model gateway such as LiteLLM, Portkey or an enterprise AI gateway, application traces through OpenTelemetry, workflow logs from the orchestration layer, review outcomes from the operational system, and a weekly dashboard in Power BI, Looker Studio or a simple internal admin panel. The dashboard should show completed tasks, exception rate, manual override rate, average cost per useful outcome, latency, failed retrievals, blocked tool calls, sensitive-data events and user feedback. It should also separate experimentation from production use, because mixing the two makes both governance and ROI reporting unreliable.

The payoff is discipline. Once telemetry exists, leaders can stop asking whether people are using AI and start asking whether AI is improving the operating model. They can retire weak pilots, fund the workflows with measurable gains, renegotiate suppliers using real usage data, and spot risk before it becomes a customer, regulator or board issue. AI usage telemetry is not glamorous, but it is the difference between adoption theatre and operational management.

Frequently Asked Questions

What is AI usage telemetry?

AI usage telemetry is the structured capture of events from AI workflows, including model calls, prompts, tool use, human review, cost, errors, risk flags and business outcomes.

Is this different from an AI dashboard?

Yes. A dashboard is the visible report. Telemetry is the underlying evidence that makes the report reliable, traceable and useful for decision-making.

Which AI metrics should UK businesses track first?

Start with completed tasks, exception rate, human override rate, cost per useful outcome, latency, model or supplier used, data source, review status and any blocked or failed tool actions.

Do small businesses need AI telemetry?

Small businesses do not need a complex observability stack, but they still need a lightweight record of which AI workflows are being used, what they cost and whether they improve the work.

How does telemetry support AI governance?

It gives governance teams factual evidence about usage, access, approvals, incidents and outcomes, instead of relying on policy statements or self-reported behaviour.

Does AI telemetry create privacy risk?

It can if designed badly. Teams should avoid logging unnecessary personal data, separate operational metrics from sensitive content, apply retention rules and restrict access to logs.

What tools can help with AI telemetry?

Common building blocks include model gateways, OpenTelemetry traces, application logs, workflow orchestration logs, review queues and BI dashboards such as Power BI or Looker Studio.

When should telemetry be added to an AI pilot?

Add a basic event schema before the pilot touches live work. It is much harder to prove value, investigate incidents or compare suppliers after usage has already spread.