Build An AI Incident Response Runbook Before Autonomous Workflows Go Live

Tools & Technical Tutorials

17 June 2026 | By Ashley Marshall

The risky moment is not when an AI agent gives a bad answer. It is when it can quietly act on that answer inside a live system.

Start With The Failure Modes, Not The Workflow Diagram

Most AI workflow planning still starts with a neat automation map: trigger, model call, tool call, human approval, update the system of record. Incident response needs to start somewhere less comfortable. What would happen if the agent picked up a poisoned instruction, used the wrong customer record, retried an action until it created duplicate work, or continued operating after a downstream system changed its permissions? That is the point of an AI incident response runbook. It turns vague unease about autonomy into named failure modes, clear thresholds and rehearsed decisions.

UK guidance is already pointing in this direction. The NCSC says AI systems are subject to novel security vulnerabilities that have to be considered alongside standard cyber security threats, and its secure AI system development guidance explicitly includes developing incident management processes during secure deployment. The UK government and DSIT also frame AI security as a lifecycle issue, not a one-off model assessment. Their AI Cyber Security Code of Practice covers secure design, deployment, maintenance and end of life, and includes principles such as human responsibility, asset tracking, monitoring system behaviour and documenting data, models and prompts.

What this means in practice is simple: before an autonomous workflow touches Salesforce, HubSpot, ServiceNow, Xero, Microsoft 365, a warehouse system or any live database, the runbook should define the incident classes you expect. For example: unauthorised tool use, data leakage through a prompt or retrieval system, prompt injection, memory poisoning, excessive retries, wrong-recipient communications, unexpected privilege escalation, bad model output accepted by an approver, and loss of auditability. Each class needs a severity level, an owner, a containment action and evidence to preserve. If you cannot name the failure mode, you cannot measure it, escalate it or learn from it.

Build The Runbook Around Control Points

A useful runbook is not a PDF that security owns and operations forgets. It is a set of control points sitting beside the actual workflow: where the agent gets credentials, where it reads context, where it calls tools, where it writes to systems, where a human approves an action, and where the organisation can pause, roll back or revoke access. Autonomous workflows are different from ordinary integrations because intent can be inferred at run time. The same agent might summarise a record, draft an email, update a field and trigger a follow-up sequence. The incident plan has to separate those activities rather than treating the agent as either trusted or blocked.

That distinction matters because the leading governance problem is often over-simplification. In May 2026, ITPro reported Gartner analysis warning that governance failures could lead to four-in-ten organisations demoting or decommissioning autonomous AI agents within a year, partly because companies were treating agent governance as binary. The article describes four practical levels: observe, advise, act with approval and fully autonomous. For an incident runbook, those levels become response levers. You might leave an observe agent running while freezing write access for an act-with-approval agent, or keep an advisory assistant available while disabling external email sending.

The minimum control points are identity, access, logging, approval, rate limits and rollback. Identity means named service accounts, not shared admin credentials. Access means scoped permissions per tool and per workflow, not a general API token that can do everything. Logging means prompt inputs, retrieval sources, tool calls, approvals, outputs and error paths captured in a tamper-resistant place such as Microsoft Sentinel, Splunk, Datadog, Elastic or your SIEM of choice. Approval means a real decision point with enough context for a person to challenge the action. Rate limits and circuit breakers stop a bad instruction turning into hundreds of live changes. Rollback means the team knows how to reverse a CRM update, withdraw an email, disable a queue, restore a vector index or rotate a token quickly.

Write AI-Specific Triage Questions

Traditional incident triage asks what system is affected, what data is involved, who has access and whether the threat is still active. AI incidents need those questions, but they also need a second layer. Was the output wrong because the model hallucinated, because retrieval returned the wrong document, because a prompt injection changed the instruction, because memory was poisoned, because a tool had too much access, or because a human approved something they did not understand? Those are different incidents, even if the visible symptom is the same.

The Coalition for Secure AI describes AI incident response as a discipline that needs AI-specific telemetry: prompt logs, model inference activity, tool executions and memory state changes. Its framework highlights detection of unusual prompt patterns, model drift, unexpected retrieval behaviour in RAG systems, memory injection attacks, RAG poisoning and cloud credential abuse. Those examples are useful because they shift the team away from a generic question such as "did the chatbot go wrong?" and towards a forensic question: which component made the unsafe action possible?

What this means in practice is that your first-hour checklist should be written in plain operational language. Freeze the affected workflow. Preserve the conversation, prompt, retrieval results, tool call trace, approval record and downstream system logs. Identify whether personal data, special category data, regulated records, payment data, commercially sensitive information or customer communications were involved. Check whether the incident is ongoing or whether it stopped when the workflow paused. Decide whether to rotate credentials, purge memory, rebuild a vector store, roll back a model version, disable a plugin, quarantine documents or notify affected business owners. If you use LangSmith, Helicone, OpenTelemetry, Azure AI Foundry tracing, AWS CloudTrail, Google Cloud Audit Logs or a vendor console, the runbook should say exactly where the responder goes and what evidence to export.

Treat Regulatory Clocks As Part Of The Design

For UK organisations, an AI incident runbook is not only a technical recovery document. It is also a compliance timing document. If an autonomous workflow exposes, alters, deletes or makes unavailable personal data, the question quickly becomes whether there has been a personal data breach under UK GDPR. The ICO defines a personal data breach as a security breach leading to accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data. The ICO also says organisations must notify a notifiable breach without undue delay and no later than 72 hours after becoming aware of it.

That 72-hour clock is why the runbook should not wait for perfect forensic certainty. It should define who decides whether the organisation has become aware, who performs the risk assessment, who signs off notification, and who informs individuals if the risk to their rights and freedoms is high. The ICO data security incident trends page reinforces the same requirement and notes that reported figures only cover incidents discovered and reported to the ICO, so they are not a complete picture of every incident affecting UK organisations.

The runbook should also connect to sector obligations. A law firm may need to consider SRA expectations and client confidentiality. A financial services firm may need FCA operational resilience and outsourcing considerations. A health or care organisation may have clinical safety, NHS DSPT and patient confidentiality issues. A supplier to government or critical infrastructure may need contractual, NCSC or customer notification routes. The practical lesson is to build a regulatory contact tree before the incident. Include legal, data protection, cyber, operations, communications, the system owner and vendor contacts. If the first time those people meet is during an AI incident, the organisation has already lost time it cannot recover.

Rehearse The Human Decisions Before Pressure Hits

The common misconception is that a human approval step solves the autonomy problem. It helps, but only if the person has time, context, authority and a realistic chance of saying no. In busy operational teams, approval can degrade into a reflex. If the agent has drafted thirty account changes, flagged them as routine and presented them in a cramped queue, the approver may become a throughput mechanism rather than a control. That is why the runbook has to define what a meaningful approval looks like and what happens when approval quality itself is part of the incident.

The NCSC Annual Review 2025 shows why rehearsal matters. Its incident management team received 1,727 incident tips in 2024 to 2025, triaged 429 incidents requiring NCSC support, and classified 204 as nationally significant, up from 89 the previous year. The same review says 18 incidents were highly significant, a 50 percent increase and the third consecutive annual rise. These are not AI-agent numbers, but they are a useful warning: incident response is already under pressure before organisations add workflows that can act at machine speed.

Run a tabletop exercise before live deployment. Give the team a scenario: a support agent connected to Zendesk and Salesforce receives an indirect prompt injection in a customer attachment, updates the wrong account tier and sends a misleading refund email to twenty customers. Ask who pauses the workflow, who owns customer communication, who checks the CRM audit trail, who assesses personal data impact, who informs the vendor, and who decides whether the agent can be restarted. Then repeat with a finance workflow, an HR workflow or a software delivery workflow using GitHub Actions. The point is not theatre. It is to expose missing access controls, unclear ownership, logging gaps and decision delays while the stakes are still artificial.

Keep The Runbook Small Enough To Use And Strong Enough To Audit

A good AI incident response runbook is usually shorter than people expect. The hard work is not producing a long document. It is agreeing the thresholds and wiring them into operations. For each autonomous workflow, document the owner, business purpose, connected systems, data classes, credentials, model provider, retrieval sources, memory behaviour, tool permissions, logging location, human control points, kill switch and recovery method. Then add a severity matrix: advisory output issue, contained workflow error, unauthorised action, data exposure, regulatory breach risk, customer harm, operational outage or suspected hostile manipulation.

The runbook also needs a maintenance rhythm. The UK AI Cyber Security Code of Practice includes principles on maintaining updates, monitoring behaviour, documenting data, models and prompts, and ensuring proper disposal. The NCSC and DSIT have worked with ETSI on baseline security requirements across the AI lifecycle, and the NCSC says the ETSI specification is the first global standard setting minimum security requirements across the whole AI lifecycle for AI supply chain stakeholders. In other words, a runbook that was correct at launch can become stale when a model version changes, a vendor adds a tool, a retrieval source grows, or an integration gains extra permissions.

For auditability, connect the runbook to evidence. Keep version history in Git, Confluence, SharePoint or your GRC tool. Record every tabletop exercise, every exception, every post-incident review and every change to permissions. Map each control to something observable: a Sentinel query, an Okta group, a GitHub environment rule, a ServiceNow workflow, a Datadog monitor, a SIEM alert, a vendor audit export or an API rate limit. The counterargument is that this slows AI delivery. The better answer is that it protects delivery. Teams can move faster when they know which workflows are allowed to act, how they fail, who can stop them and what evidence will exist when something goes wrong.

Frequently Asked Questions

What is an AI incident response runbook?

It is a practical response guide for incidents involving AI systems, especially workflows that can read from or write to live systems. It defines failure modes, severity levels, owners, evidence to preserve, containment steps, escalation routes and recovery actions.

How is this different from a normal cyber incident runbook?

It adds AI-specific questions about prompts, retrieval, memory, model behaviour, tool calls, approval quality and agent permissions. A normal runbook may miss whether a bad action came from prompt injection, poisoned context, excessive access or human over-reliance on an AI recommendation.

Do we need a runbook before using AI agents internally?

Yes, if the agent can access sensitive data, trigger actions, send messages, change records or influence decisions. For low-risk read-only assistants the runbook can be lightweight, but there should still be ownership, logging and a pause process.

Who should own the AI incident response runbook?

Ownership should sit with the business system owner and security or risk lead jointly. Legal, data protection, operations, IT, communications and the vendor owner should all be named in the escalation route.

What logs should we keep for autonomous workflows?

Keep prompts, system instructions, retrieval results, model outputs, tool call traces, approval records, credentials used, downstream system audit logs, errors, retries and memory changes. The runbook should say where each log lives and who can export it.

When does an AI incident become an ICO reportable breach?

If personal data has been accidentally or unlawfully destroyed, lost, altered, disclosed, accessed without authorisation or made unavailable in a way that is likely to risk individuals rights and freedoms, it may be notifiable. The ICO deadline is no later than 72 hours after awareness for notifiable breaches.

Should we disable all agents during an incident?

Not always. The better approach is proportional containment. You may disable write actions, revoke a specific token, pause one workflow, block a retrieval source or reduce autonomy while leaving read-only assistants available.

How often should the runbook be tested?

Test it before first live deployment, after major changes to models, tools, permissions or data sources, and at least annually for lower-risk workflows. High-impact workflows should be exercised more often, especially where customer, finance, HR or regulated data is involved.