OpenTelemetry Should Be Your First AI Agent Control Before Scaling

Tools & Technical Tutorials

26 August 2026 | By Ashley Marshall

AI agents do not just need better prompts. They need traces that show which model acted, which tool ran and where the risk entered the workflow.

Why agent telemetry has moved from nice-to-have to control evidence

Most UK teams are still treating AI observability as a developer convenience: useful when an answer is slow, optional when the pilot appears to work. That view is now too narrow. Agentic systems make tool calls, retrieve documents, choose models, retry failed steps and sometimes act across SaaS accounts. If those actions are not traceable, leaders are left judging performance from the final answer rather than from the chain of decisions that produced it. The practical risk is simple: an agent can look productive while making choices that security, compliance or finance cannot explain afterwards.

The NCSC's August 2026 advice on managing the cyber risk of agentic AI says organisations should consider how these systems are deployed, constrained, observed and responded to. It specifically calls for observability, operational monitoring and response procedures, and warns that the greater the autonomy, the greater the need for controls. That matters because many AI pilots are moving from assisted drafting into real workflow access: CRM updates, procurement summaries, finance reconciliations, research packs and service desk triage.

OpenTelemetry is useful here because it gives teams a familiar control surface. Instead of inventing a new dashboard for every model vendor or agent framework, teams can emit traces, metrics and logs into an existing telemetry pipeline. The first business benefit is not a prettier chart. It is the ability to answer basic governance questions: which model was called, which tool was used, how long did it take, how many tokens were consumed, what data source was retrieved, and where did a workflow stop. That is the evidence trail leaders need before scaling AI agents beyond controlled pilots.

What OpenTelemetry records for AI agents

OpenTelemetry's GenAI semantic conventions standardise the shape of AI telemetry. The May 2026 OpenTelemetry walkthrough, Inside the LLM Call, shows a trace tree with a top-level agent invocation, child chat spans for model calls and execute_tool spans for tool invocations. Span attributes include the requested model, input tokens, output tokens and finish reasons. Those details sound technical, but they map directly to questions business leaders keep asking when AI spend and AI risk become visible.

For example, finance can separate expensive prompts from useful outcomes because token usage is attached to the actual call. Engineering can see whether a slow workflow was caused by the model, a retrieval step, a tool call or a retry loop. Security can inspect whether a tool was invoked outside an expected path. Product owners can compare model choices without losing the thread of the user request that triggered them. The same trace can connect a customer-facing answer to the retrieval source and the operational systems touched during the workflow.

The useful point for UK organisations is that OpenTelemetry is not another AI platform to buy before learning anything. Many teams already use OTLP-compatible backends, collectors or APM tools. A sensible first implementation is therefore small: instrument one agent workflow, capture metadata by default, disable sensitive content capture unless there is a clear legal and security basis, and build a dashboard around four signals: latency, token use, tool calls and error paths. That gives a baseline without over-collecting personal data or confidential prompts.

The UK governance link: logs, DPIAs and accountability

Observability also connects directly to UK data protection practice. The ICO's guidance on AI and data protection frames AI governance around accountability, transparency, lawfulness, accuracy and fairness. It highlights Data Protection Impact Assessment considerations and fairness across the AI lifecycle. Telemetry does not solve those obligations by itself, but it provides evidence that an organisation can review when a system is challenged, audited or improved.

This is where the counterargument deserves a fair hearing. Some teams worry that logging prompts, tool arguments and model outputs creates a bigger privacy and confidentiality risk than the agent itself. They are right to worry. Full content capture can include personal data, commercial information, credentials accidentally pasted into prompts, or sensitive customer context. The answer is not to avoid observability. The answer is to design it like a controlled data product: define retention, minimise content capture, redact where possible, restrict access, and record enough metadata to investigate behaviour without turning every prompt into a permanent compliance liability.

What this means in practice is that DPIA work should include telemetry decisions. Decide which spans are mandatory, which attributes are prohibited, who can view traces, how long logs are retained, and when prompt content can be captured for debugging. For higher-risk agents, especially those touching customer records, HR data, regulated advice or financial workflows, the observability plan should sit beside the risk assessment rather than behind the engineering backlog. A trace that cannot be shared with governance colleagues is not governance evidence. It is just operational noise.

Where OpenTelemetry stops and evaluation must begin

A common misconception is that observability proves an AI system is good. It does not. Observability tells you what happened. Evaluation tells you whether what happened met the standard. The distinction matters because an LLM call can complete quickly, consume a sensible number of tokens, return no technical error and still produce an answer that is irrelevant, ungrounded or unsafe. OpenTelemetry gives you the trace context needed to investigate. It does not replace test sets, human review, policy checks or model quality scoring.

Recent industry analysis from Fiddler's July 2026 article on OpenTelemetry for AI observability makes this boundary clear: OTel can record that an LLM returned 1,200 tokens in 850ms, but it cannot decide whether those tokens contradicted the source documents. Greptime's May 2026 guide to OpenTelemetry GenAI semantic conventions also notes that the conventions now span model calls, agent orchestration, MCP tool calling, content capture and evaluation events, while parts of the GenAI and MCP conventions remain in development.

The practical operating model is therefore layered. Use OpenTelemetry to make the workflow visible. Use evaluation datasets to test accuracy and groundedness. Use guardrails to block known unsafe behaviours. Use incident response to decide what happens when a trace shows an unacceptable action. This also makes procurement more mature. When a vendor says their agent is observable, ask whether they emit standard GenAI telemetry, whether traces include tool calls, whether metadata can be exported, and how evaluation results are linked back to the same workflow ID.

A first implementation pattern for UK teams

The best starting point is one workflow, not an estate-wide observability programme. Pick an agent that already has business value but limited blast radius: internal research summaries, service desk categorisation, sales call note processing or policy document search. Instrument the orchestrator, model calls, retrieval steps and tool invocations. Keep content capture off by default. Record model name, provider, latency, token usage, finish reason, tool name, tool result status, retrieval source identifier and user-facing outcome. Add a correlation ID so governance, support and engineering can all discuss the same run.

For tooling, do not overcomplicate the first pass. OpenTelemetry's own example uses the free Aspire Dashboard as a local viewer. Production teams may export to Grafana Tempo, Datadog, Honeycomb, New Relic, Azure Monitor, Google Cloud Trace or another OTLP-compatible backend already approved internally. The important decision is to avoid a telemetry design that only works inside one agent framework. LangGraph, Semantic Kernel, CrewAI, PydanticAI, custom Python services and coding assistants will keep changing. A standard trace layer gives the organisation a better chance of comparing them without rebuilding the evidence trail each time.

What this means in practice: make an AI observability acceptance test part of release readiness. A workflow should not move from pilot to production unless a reviewer can open a trace, identify each model call, see each tool invocation, inspect cost and latency, and connect failures to an owner. That acceptance test is small enough to run quickly, but strong enough to expose whether the agent is operationally mature. It also gives boards and senior leaders a more realistic view of AI progress: not just how impressive the demo looked, but whether the system can be monitored, contained and improved.

The leadership decision: standardise before the second platform arrives

The moment to standardise AI telemetry is before the second or third agent platform arrives. Once separate teams have chosen separate vendors, each with its own tracing format and dashboard, governance becomes a spreadsheet exercise. Leaders end up asking teams to export screenshots, explain missing logs and manually reconcile incidents across systems. That is exactly the fragmentation OpenTelemetry was designed to reduce. It does not force a single AI stack. It creates a common evidence layer above diverse tools.

For UK leaders, the decision is less about observability fashion and more about operating discipline. If agents are going to touch production systems, customer data or regulated processes, the organisation needs to know what they did. If AI budgets are going to grow, finance needs cost and usage tied to outcomes. If employees are going to rely on agents for daily work, support teams need failure patterns that can be investigated. If suppliers are going to provide AI functionality, procurement needs exportable telemetry requirements in contracts. These are business controls, not developer preferences.

The recommendation is straightforward: put OpenTelemetry GenAI compatibility into the next AI platform checklist, even if the first implementation is modest. Ask vendors whether they emit OTLP traces and metrics, whether GenAI semantic conventions are supported, how content capture is controlled, and whether tool calls and retrieval steps are visible. For internal builds, create a lightweight reference implementation that teams can copy. The organisations that do this early will have a quieter scaling path. They will still have AI incidents and quality issues, but they will be able to see them, explain them and fix them with evidence.

Frequently Asked Questions

Is OpenTelemetry an AI observability platform?

No. It is a vendor-neutral telemetry standard and pipeline. You still need a backend such as Grafana, Datadog, Honeycomb, New Relic, Azure Monitor or another OTLP-compatible tool to store and inspect the data.

Should we capture full prompts and model responses?

Usually not by default. Start with metadata such as model, latency, token use and tool calls. Capture content only when you have a clear debugging need, a lawful basis, access controls and retention rules.

Does OpenTelemetry prove an AI answer is accurate?

No. It shows what happened in the workflow. Accuracy, groundedness, policy compliance and relevance need evaluation datasets, human review, guardrails or specialist AI evaluation tools.

Which AI signals should a first dashboard show?

Start with latency, input tokens, output tokens, model name, tool invocations, retrieval source, error paths and cost estimates. Add quality and safety scores once evaluation is in place.

How does this help with UK GDPR and ICO expectations?

It gives accountable evidence about how an AI system operated, but it must be designed carefully. Telemetry choices should be included in DPIAs, with minimisation, redaction, access control and retention agreed upfront.

Can this work with different agent frameworks?

Yes, that is the point. The ecosystem is still maturing, but OpenTelemetry reduces dependence on one framework or observability vendor by giving teams a standard export format.

What is the main implementation risk?

Over-collection. If teams capture every prompt and output without controls, telemetry can become a new data protection and confidentiality risk. Metadata-first instrumentation is the safer starting point.

When should a business invest in this?

Before agents move from pilots into production workflows, especially where they can access customer data, internal systems, regulated content or spend-generating tools.