Test Your AI Agent’s Privacy Boundaries Before It Touches Customer Data

AI Trust & Governance

15 October 2026 | By Ashley Marshall

A vendor’s no-training promise does not tell you where an agent sends customer data. The real test is every connector, tool call and log in the workflow.

The privacy boundary is now an operational question

An AI agent can read a customer record, summarise a complaint, search an internal knowledge base and send a reply in one run. That convenience also means a single instruction can cross several privacy boundaries. The interesting question is no longer simply whether the model provider trains on your prompts. It is which person, tool and destination can see each piece of personal information while the agent works. For a UK business, that boundary should be mapped before customer data enters a live workflow.

The urgency is not hypothetical. On 8 October the Information Commissioner’s Office said it had secured data protection changes from ten leading foundation model developers and opened a six-week call for evidence on agentic AI. Its questions cover security, transparency, accountability, automated decision-making, fairness and lawful processing. The ICO also said it had made enquiries into reported agent testing incidents involving unauthorised communication channels and access to external systems. Those enquiries are ongoing; the reports are not findings that a named deployment breached the law. They do, however, show the kind of boundary failure a buyer must test.

The practical distinction is between a chatbot answering a question and an agent acting with delegated access. If the latter can retrieve records or call external tools, a vendor privacy statement is only one part of the answer. You also need to know the permissions attached to the agent identity, the data returned by each connector, the log retained after the run and the destination of every outbound call. A useful first exercise is to trace one ordinary customer task end to end and annotate each hand-off with the data category, authorised purpose and owner. If nobody can draw that path, do not put the workflow into production.

This is not a case for banning agents. It is a case for proving the boundary at the point where the workflow crosses it. A narrow assistant that drafts a response from a redacted case summary may be appropriate today; an autonomous assistant with bulk CRM export and unrestricted email access may need a redesign first.

Separate the model contract from the workflow contract

Procurement teams often ask whether prompts are used for training and stop there. That is a useful question, but it misses the workflow. The agent may query a CRM, send text to a search service, invoke an analytics endpoint and preserve a detailed trace in another platform. Each service can have a different retention period, region and subprocessor chain. A claim that the model does not train on inputs does not establish that the whole workflow minimises personal data.

The ICO’s foundation model report says developers processing personal data for training must identify a lawful basis, provide meaningful transparency, enable rights and demonstrate safeguards. A UK company deploying an agent has its own responsibilities for the processing it directs. Do not confuse vendor commitments with your organisation’s assessment of the actual use case. Ask which entity controls each step, what each supplier is instructed to do and whether a subject rights request can be fulfilled across prompts, retrieval stores, transcripts and logs.

In practice, write a one-page workflow contract. List the trigger, permitted data fields, tools, external destinations, retention rules and authorised human approver. For a customer service agent, the permitted fields might be case number, product and issue summary; date of birth and full payment details may not be needed. If a connector returns a whole customer object because it is easier to integrate, fix the connector. Field-level filtering before the model call is more dependable than a prompt saying “ignore sensitive fields”.

The counterargument is that this paperwork slows a small team. It should not. A compact workflow contract replaces vague debate with a testable specification. It also reduces rework when a vendor changes its model, an integration gains new fields or a staff member asks for a more autonomous action. The smaller the team, the more valuable a clear boundary becomes: there is less spare capacity to reconstruct a data trail after an incident.

Run a five-part boundary test before launch

Test the workflow like an access-control system, not like a writing assistant. First, give the agent a normal authorised task and capture every read and outbound call. Second, ask a user outside the account to request the same information. Third, place an instruction inside a retrieved document that asks the agent to send customer details elsewhere. Fourth, give it a legitimate task where the requested data is excessive. Fifth, revoke its access halfway through a run. For each test, record whether the boundary held, whether the run stopped safely and whether a reviewer could reconstruct what happened.

Use realistic but synthetic records, including names, addresses and account identifiers designed for the exercise. Do not use a real customer record to find out whether a test agent leaks data. The test should cover prompts, tool arguments, retrieved content, output, telemetry and error logs. A failed step may be caused by the tool, the retrieval layer or permissions rather than the model. Assign the fix to the component that actually allowed the crossing.

The ICO call for evidence remains open until 20 November 2026 and explicitly asks about data security, transparency and accountability. That makes this a timely exercise for UK deployers. It is not a new legal deadline for every business, nor does a passing test amount to ICO approval. It is a way to build evidence that the system’s controls match its intended use. Keep the test cases and outcomes with the data protection impact assessment where one is required.

What this means in practice is a release gate: no production customer data until the five tests pass, a named owner signs the results and the connector permissions match the approved data map. Repeat the test when a tool, model, prompt, retrieval index or permission set changes materially. A clean test from six months ago does not cover a new connector added yesterday.

Make permissions smaller than the task looks

Agent permissions are often copied from a human account or a broad integration token. That is the wrong starting point. A human may have discretion, training and context that the agent cannot reliably apply. Give the agent its own identity and only the actions required for the narrow workflow. Read-only access should remain read-only. A reply-drafting agent does not need authority to delete records, change consent flags or send an email without approval.

The Department for Work and Pensions AI Security Policy, updated on 7 October, offers a concrete UK example of risk-based boundaries. It says personal data and certain other information need explicit approval before use with an AI tool, and it requires a data protection impact assessment process for AI processing of personal data or output affecting people. Its supplier annex also calls for transparency about which tools were used and evidence of how output was achieved. DWP policy is not a rule imposed on every private business, but the design pattern is useful: approve the data and use case, then verify suppliers against that approval.

For a small business, implement this with a dedicated service account, a limited API scope and a tool allow-list. Let the agent read only a single case at a time, not search all customers. Require a human to approve outbound messages containing account information or decisions with financial effect. Log which fields were returned by the connector, not just the final answer. If a tool lacks field-level permissions, place a thin filtering service in front of it or choose a different integration.

There is a legitimate concern that tight permissions make agents less helpful. The right response is to widen access only when a measured business task requires it and the risk owner signs the change. Broad access at launch is easy; narrowing it after staff have built habits around it is much harder.

Design the evidence you will need later

When an agent makes a mistake, the final chat transcript rarely explains enough. You need to know which version of the workflow ran, which source records were retrieved, what tool arguments were sent, what approval was given and where information went. Yet collecting everything indefinitely creates a new privacy problem. Evidence design is therefore a balancing exercise: enough detail to investigate and answer rights requests, with restricted access and a justified retention period.

Build a run record with a unique ID, timestamp, workflow version, agent identity, connector names, record identifiers, approvals and outcomes. For high-risk steps, preserve the reason an action was authorised and the minimum information needed to reproduce the decision. Mask or avoid copying raw personal data into general-purpose telemetry. Set separate retention for operational logs, security logs and customer records according to their purpose. Test deletion and export across all stores, including vendor dashboards and observability tools.

The ICO’s 8 October announcement says ten model developers have made or committed to improvements in transparency, rights mechanisms and safeguard assessments. Buyers should use those themes in supplier questions, but should not present a general vendor commitment as proof that their specific agent workflow satisfies UK GDPR. Ask for documented settings and a sample rights-request path. Then run an internal rehearsal: can you identify all data related to a test person and explain what can be erased, what must be retained and why?

The practical output is a short evidence pack, not a shelf of policy documents. Include the workflow map, permissions, test results, data protection assessment, vendor terms, retention decisions and an incident contact. If an auditor or customer asks how the agent handled their data, you should be able to answer from this pack without interviewing five people and guessing.

Start with one bounded workflow, then expand deliberately

A privacy-aware rollout does not require an enterprise programme before the first experiment. Choose one workflow where the agent can create value with limited data and reversible actions. A good candidate is drafting an internal response from a redacted support case, with a human reviewing before it goes to the customer. A poor first candidate is an agent with full CRM, payments and email privileges that can make account changes on its own. Both may eventually be useful, but they demand different evidence.

For the first month, measure three things: whether the task is completed correctly, whether the agent attempted to cross a forbidden boundary and how much human correction was needed. Count boundary attempts even when a control blocked them. They may reveal a badly scoped prompt or a connector that returns too much information. Review a sample of runs weekly and record the specific permission or data-field change made as a result. That creates a feedback loop more useful than a generic “AI risk” score.

The ICO consultation is a reminder that guidance is still developing. Submit evidence by 20 November if your organisation has concrete deployment experience, but do not wait for the final guidance to apply existing data protection principles. The agency is asking how organisations manage distinctive agentic risks because those risks arise in real workflows today. Its announcement explicitly says autonomy is not an excuse for poor compliance.

The sensible sequence is narrow task, mapped data, limited tools, synthetic boundary test, approved live pilot and measured expansion. Each expansion should specify the new data or action, its business benefit and the additional control needed. That is slower than granting every permission on day one, but faster than rebuilding customer trust after an avoidable exposure. The aim is not to prove agents are perfectly safe. It is to make the boundary visible, testable and owned.

Frequently Asked Questions

Does a no-training guarantee make an AI agent safe for customer data?

No. It addresses one model-provider use of inputs, not connector access, outbound calls, subprocessors or log retention. Assess the whole workflow.

Should we use real customer records in a boundary test?

No. Use realistic synthetic records so a failed test does not itself expose a customer’s information.

Is a data protection impact assessment always required for an AI agent?

Not for every agent automatically. Assess whether the processing is likely to result in high risk to individuals. Document the decision and complete a DPIA where required.

What should trigger a repeat boundary test?

A material change to the model, prompt, connector, retrieval index, data fields, permissions or outbound action should trigger a relevant retest.

Can an agent use a staff member’s existing CRM login?

A dedicated agent identity is preferable. It allows narrower permissions, clearer attribution and prompt revocation without disrupting the staff member.

Does the DWP AI policy apply to private companies?

No. It governs DWP and its relevant suppliers. Its approval and evidence patterns can still inform a private company’s own controls.

When does the ICO agentic AI call for evidence close?

It is open until the end of 20 November 2026. That is a consultation deadline, not a general compliance deadline.

What is the smallest useful evidence pack?

Keep the workflow map, access scopes, test results, relevant impact assessment, supplier terms, retention rules, approval record and incident owner together.