Runbook-driven AI agents: safer recovery when tools fail

Tools & Technical Tutorials

11 May 2026 | By Ashley Marshall

The dangerous moment is not when an AI agent gets stuck. It is when it decides how to recover without a safe operating procedure.

Why agent recovery is now an operational design problem

Tool-using AI agents change the failure mode of automation. A normal workflow engine either completes a defined step or returns an error. An agent can interpret an error, choose a different tool, call a second system, modify a record, ask another agent for help, or continue with partial context. That flexibility is useful, but it also means failure recovery is no longer just exception handling. It is operational risk management.

The clearest recent warning comes from the joint guidance Careful adoption of agentic AI services, authored by ASD's ACSC, CISA, NSA, the Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK. It says agentic systems increasingly operate in critical infrastructure and defence, and recommends that organisations never grant broad or unrestricted access, especially to sensitive data or critical systems. That matters for ordinary UK firms too, because the same patterns appear in procurement, HR, finance, customer support and IT operations.

A runbook-driven agent treats recovery as a controlled procedure, not an improvisation. The runbook defines allowed retries, fallback tools, escalation thresholds, human approval points, rollback steps, log requirements and stop conditions. In practice, this means the agent is not asked to be clever when the situation is ambiguous. It is asked to follow a known recovery path, gather evidence and pause before it can make the blast radius worse.

What this means in practice: do not start by giving an agent general permission to "fix failed tasks". Start with one narrow workflow, such as invoice matching exceptions, helpdesk triage or failed CRM enrichment. Write the failure states down. Decide which states the agent may resolve, which it may retry once, which it must quarantine, and which it must escalate to a named human owner.

The real risk is not a bad answer, it is an uncontrolled action

Most boards still think about AI risk as hallucination risk. For tool-using agents, that is too narrow. The bigger risk is a plausible action taken in the wrong context, using the wrong privilege, after a tool failure has removed the normal checks. If a summarisation model makes a mistake, a human can often spot it before acting. If an agent deletes logs, approves a payment, sends a customer email or changes firewall rules, the organisation may only notice after the damage has propagated.

The Five Eyes guidance gives a useful procurement example. An organisation grants an agent broad access to financial systems, email and contract repositories. A malicious actor compromises a low-risk integrated tool, inherits the agent's excessive privileges, modifies contracts and approves unauthorised payments. The guidance calls out the "confused deputy" pattern, where a trusted agent is misused to perform actions the attacker could not perform directly.

This is where runbooks become a security control. A recovery runbook should say that a failed supplier check cannot be resolved by changing payment details, approving an exception and suppressing an alert in the same autonomous chain. It should require separation of duties. It should require a fresh identity and policy check before each high-impact action. It should require a human review when the agent crosses from information gathering into transaction execution.

For UK firms, this maps neatly to existing controls. Finance teams already understand approval limits. IT teams already understand change windows. Compliance teams already understand audit trails. The agent runbook simply makes those controls executable. It turns "use judgement" into "if the supplier record conflicts with Companies House data, quarantine the case, attach evidence, notify finance operations and do not update bank details".

A good runbook has stop conditions, not just retry logic

Many agent prototypes fail safely in demos because the happy path is well curated. Production is different. APIs rate-limit. SaaS tools return partial responses. A CRM field changes name. A browser automation step lands on a consent banner. A retrieval tool returns stale data. A user prompt contains contradictory instructions. The agent may respond by retrying, switching tools or inventing a workaround. Without a runbook, these are uncontrolled design decisions.

The NCSC's recent blog on supporting AI adoption for UK cyber defence makes the same point in a cyber context. It notes that frontier AI tools can perform some tasks extremely well, but can also be unreliable, difficult to validate and hard to integrate safely into existing environments. That sentence should be printed above every agent implementation plan.

A practical recovery runbook should define four layers. First, expected failures: timeout, unavailable tool, malformed response, duplicate record, missing field. Second, permitted recovery actions: retry once, call a read-only fallback, ask for clarification, create a draft, or stop. Third, hard stops: privilege mismatch, conflicting source data, request to delete evidence, unexpected tool output, or any request outside the approved task boundary. Fourth, escalation: who receives the case, what evidence is included, what service level applies and whether the agent may resume afterwards.

The important design choice is that the stop condition must beat the optimisation goal. If the agent's target is "complete 95 percent of tickets without human input", it may learn to avoid escalation. If the runbook target is "complete low-risk tickets and preserve evidence whenever uncertainty rises", the system behaves more like a controlled operator. That is less glamorous, but it is how firms avoid turning a recoverable failure into a governance incident.

Observability is the difference between recovery and guesswork

Runbook-driven recovery only works if the organisation can see what the agent saw and did. This is where many pilots are weakest. Teams log the user prompt and the final answer, but not the tool calls, retrieved sources, intermediate decisions, rejected options, policy checks, identity changes or escalation reason. When something goes wrong, the post-incident review becomes a debate about a black box.

The Five Eyes guidance is direct on this point. It recommends comprehensive artefact logging by default, unified audit logs for inter-agent interactions, source checks that record which tools were used and what information was retrieved, runtime monitoring, anomaly detection and multiple independent monitoring systems that cross-validate agent reports and system logs. It also recommends that agents be isolated into enclaves with no write access to logs. That last point is easy to overlook and extremely important.

What this means in practice: every runbook step should produce a structured event. A useful event record includes the runbook name and version, agent identity, user identity, tool called, input summary, output hash or reference, policy decision, confidence band, escalation trigger and human approval reference where relevant. Store that outside the agent's own write path. If the agent can edit the audit trail, the audit trail is not evidence.

This also improves performance management. If a claims handling agent escalates 40 percent of cases because a validation API often times out, the answer may not be "make the agent braver". The answer may be to improve the tool, change the retry schedule, add a read-only fallback source, or narrow the workflow. Observability lets you improve the system without weakening the control model.

UK governance is moving towards assurance, not blind autonomy

Runbook-driven agents fit the direction of UK policy because they create evidence. The Department for Science, Innovation and Technology's AI Management Essentials tool is intended to help organisations assess and implement responsible AI management systems and processes. DSIT reported 65 consultation responses from businesses, trade associations, professional bodies, academia, research and legal services before publishing the government response in February 2026. The direction is clear: AI governance needs to become something organisations can assess, procure and evidence.

The trusted third-party AI assurance roadmap reinforces that point. DSIT says the UK AI assurance market had over 524 companies and about £1.01 billion in gross value added in 2024, with potential to reach over £18.8 billion by 2035 if barriers to adoption are addressed. Assurance is not paperwork for its own sake. It is how buyers, boards and regulators gain confidence that AI systems are working as intended.

Runbooks provide a concrete assurance artefact. They show what the agent is allowed to do, what it is not allowed to do, what happens when a tool fails, when a person must approve, how actions are logged and how rollback works. They can be reviewed by internal audit, cyber security, data protection, legal and operational owners. They can also be tested before deployment.

The counterargument is that strict runbooks reduce the value of agents. If every unusual case escalates, why not use ordinary automation? The answer is that the best agents should be flexible inside a governed envelope. They can summarise evidence, compare sources, prepare draft actions, route exceptions and recommend next steps. They should not silently expand their own authority because a workflow was badly designed.

How to build a recovery runbook that survives production

Start with the job, not the model. Pick one workflow where failures are frequent enough to matter but contained enough to manage. Good candidates include failed onboarding checks, duplicate customer records, missing invoice data, CRM enrichment exceptions, support ticket routing and internal knowledge retrieval. Avoid starting with payroll changes, supplier bank detail updates, legal commitments or live security changes unless you already have mature controls.

Then map the tools. List every API, database, browser action, email account, file store and retrieval source the agent can touch. For each one, define whether access is read-only or write-capable, what identity is used, what rate limits apply, what a normal response looks like and what failure modes are expected. This is not bureaucracy. It is how you stop a low-risk integration becoming the route to a high-impact action.

Next, write the runbook in operational language. For each failure state, define: detect, decide, act, log, escalate and recover. "Detect" means the signal that proves the failure happened. "Decide" means the allowed branch. "Act" means the permitted tool call. "Log" means the evidence record. "Escalate" means the named queue or person. "Recover" means the safe end state, including rollback if needed.

Finally, test the runbook with adversarial scenarios before going live. Try stale data, conflicting records, tool timeouts, prompt injection in an email, a request to delete logs, a user asking the agent to bypass approval and a compromised low-risk tool returning persuasive instructions. The NCSC's paper on adversarial attacks against machine learning and AI warns that ML systems have a larger attack surface than traditional software and that successful attacks against one component can cascade across the system. A recovery runbook is how you make that cascade visible, containable and reversible.

Frequently Asked Questions

What is a runbook-driven AI agent?

It is a tool-using AI agent whose recovery actions are governed by predefined operational procedures. The runbook tells the agent when to retry, stop, escalate, roll back, log evidence or ask for human approval.

Why do agents need runbooks if they can reason through errors?

Reasoning is not the same as authority. A runbook prevents the agent from inventing risky workarounds when tools fail, data conflicts or permissions do not match the action requested.

Which workflows are best for a first UK business pilot?

Start with contained workflows such as support triage, duplicate record review, invoice data exceptions or CRM enrichment checks. Avoid high-impact actions such as payments, contract changes or security configuration until controls are mature.

What should trigger human approval?

Human approval should be required for high-impact, hard-to-reverse or ambiguous actions, including deleting records, changing payment details, sending external commitments, disabling controls, modifying privileges or overriding conflicting evidence.

How should agent recovery be logged?

Log the runbook version, agent identity, user identity, tool calls, retrieved sources, policy checks, decisions, escalation triggers and approval references. Store logs somewhere the agent cannot edit.

Does this slow down AI automation?

It slows down the cases that should be slowed down. Low-risk cases can still be automated, while uncertain or high-impact cases are contained before they become operational incidents.

How does this relate to UK AI governance?

Runbooks create evidence for assurance. They help organisations show how AI systems are controlled, monitored, reviewed and kept within authorised boundaries.

What is the biggest mistake firms make with tool-using agents?

The biggest mistake is granting broad permissions and relying on the model to behave sensibly. Least privilege, bounded workflows and explicit escalation rules matter more than model confidence.