AI Personalisation Needs a Way to Correct Its Assumptions

Your customer asked for a cheaper option once. Has your AI quietly decided they will always choose on price?

A useful guess is not a confirmed customer preference

Personalisation promises less repetition. A customer explains what they need once, and the next interaction starts from there. The problem begins when an assistant turns a temporary request into a durable description of the person. Asking for a cheaper delivery option this week becomes prefers the cheapest option. Comparing accessible venues for a relative becomes an assumption about the customer's own needs. A hypothetical question becomes a fact in the customer record.

These are illustrative examples, not reported client incidents. They show why a profile can be plausible and still be wrong for its purpose. An assistant does not need to invent a completely false statement to cause trouble. It can faithfully preserve something said in one context and apply it too broadly in another. The commercial consequence is a service that repeatedly narrows choices instead of helping the customer choose.

This is receiving fresh regulatory attention. In its 8 October 2026 announcement, the ICO identified increasing personalisation of consumer-facing AI services as a priority. It also reported changes made, or committed to, by ten major foundation model developers, including stronger mechanisms for people to exercise their rights. That is a statement about supervisory work, not approval of every product those developers sell.

For a UK business owner, the practical question is smaller than the whole foundation model debate. Can somebody correct what your service thinks about them without abandoning their account? Before buying another personalisation feature, ask the supplier to demonstrate that journey. A settings screen is not enough if the wrong assumption continues to drive recommendations elsewhere.

A customer asking about a cheaper option has not authorised a permanent label saying price is all that matters.

The evidence supports caution, not a claim that every memory is wrong

Recent reporting provides a concrete reason to inspect how assumptions become persistent context. Mint's 22 October coverage revisited research into ChatGPT memory. The underlying paper, The Algorithmic Self-Portrait, analysed 2,050 memory entries donated by 80 users. In that dataset, 96% of entries were created without an explicit user request to remember the information.

Read that figure carefully. It does not mean 96% of memories were fabricated, unlawful or unwanted. The researchers also reported that 84% were directly grounded in user context. The paper was first submitted in February 2026, rather than being a new October experiment. Its volunteer dataset is not a representative survey of every ChatGPT user, and it certainly does not measure your CRM. Treat it as evidence of a mechanism worth checking, not a ready-made failure rate for a business case.

The important distinction is between source fidelity and continued suitability. A memory can reflect the original conversation accurately while becoming misleading when the system treats it as a universal preference. Asked about budget options for a one-off event is different from always wants budget options. An inference should retain that distinction rather than flattening it into a convenient tag.

In practice, inspect a small sample of profiles using test accounts or properly authorised records. For each preference, identify the source interaction, whether the person actually confirmed it, the context it applies to and when it should be reviewed. Avoid collecting extra personal information merely to make the audit look thorough. The objective is to understand which assumptions influence service, then remove unjustified certainty from those assumptions.

A study of 2,050 memories from 80 users found that 96% were created without an explicit request to remember.

Separate what the customer said from what the system inferred

A sensible profile distinguishes three things. The customer may explicitly confirm a preference, such as email rather than telephone. Your systems may observe behaviour, such as three purchases from one product range. An assistant may infer a preference, such as an interest in that range. Those statements should not become interchangeable simply because they fit into the same CRM field.

Start with the software you already have. Check whether its standard preference centre, custom fields, provenance notes and workflow controls can express the distinction. Do not commission a bespoke memory platform before establishing that the existing CRM cannot handle a small set of clearly defined preferences. A plain confirmed field with an owner can be more useful than an elaborate profile nobody understands.

For each inference that genuinely improves service, record its business purpose and permitted use. A suggested interest might help a member of staff ask a better question. It should not automatically become a reason to hide options, change eligibility or make a significant decision about someone. Keep higher-impact uses outside a casual marketing pilot and subject them to the appropriate legal and operational review.

Consider a fictional maintenance company. A customer asks whether an appointment can be moved because they are temporarily away. The assistant proposes a preference for remote appointments. Staff should see that as an unconfirmed suggestion linked to one conversation, not a permanent instruction. When the customer next books, the service can ask what suits them rather than silently excluding an onsite visit.

This design also makes later correction cheaper. If you cannot distinguish the original statement from the model's interpretation, staff have to investigate the entire conversation history to fix a single label. Clear separation gives them a smaller, understandable object to review and prevents a recommendation from quietly becoming an unquestioned business fact.

Confirmed preference, observed behaviour and inferred preference need different labels and different authority.

Build a correction journey that changes the next interaction

What this means in practice is a test that follows the customer's next interaction. Create a test profile with a deliberately incorrect, low-risk preference. Ask the assistant for a recommendation that would be affected by it. Then challenge the assumption through the same support route a real customer would use. Do not fix it directly in the database and call the customer journey successful.

The person handling the request should be able to identify the disputed preference, see why it exists and stop it being relied on while it is checked. Depending on the facts, the outcome might be a correction, a confirmation request or a note that the inference is disputed. A genuine difference of opinion is not always solved by overwriting a record. The service still needs a clear explanation of what will happen next.

Repeat the original recommendation in a fresh session after the change. Check any connected email sequence, account summary and staff-facing assistant that uses the same profile. A corrected CRM field is little help if yesterday's generated summary continues to supply the old assumption. Where a downstream system cannot be updated immediately, prevent it from using the disputed label until the situation is resolved.

Keep the evidence proportionate. A test record can show the original assumption, the correction request, the authorised change and the subsequent behaviour. It does not need a permanent archive of every sensitive conversation. Where an audit record must be retained, distinguish the historical fact that a wrong label existed from the live profile used for personalisation.

Assign one owner for the end-to-end outcome. Otherwise support can say it changed the field, marketing can say it only uses the segment supplied to it, and the assistant supplier can say it followed the context it received. The customer experiences one service, so your acceptance test must cover that service rather than each team's isolated task.

The correction is complete when the wrong assumption stops affecting service, not when a ticket closes.

A privacy policy cannot substitute for an operational rights process

The UK GDPR accuracy principle and right to rectification are relevant when personal data about an identifiable individual is incorrect or misleading. The ICO's rectification guidance explains that requests can be verbal or written and need not name Article 16. Staff therefore need to recognise an ordinary complaint such as that is not my preference, please correct it as something to assess, not merely negative chatbot feedback.

The same guidance distinguishes inaccurate facts from accurately recorded opinions and historical mistakes. Do not promise that every disagreement requires erasing the underlying history. Consider whether the record accurately shows an opinion, whether its source is clear, whether supplementary information is needed and how the disputed information is being used. While accuracy is being checked, the ICO recommends restricting the disputed processing as good practice. Its guidance also notes that it is under review following the Data (Use and Access) Act, so check the current position when designing a formal procedure.

The ICO's October report on generative AI development separately stresses meaningful transparency, lawful processing and effective rights mechanisms. That report addresses developers, not a universal checklist proving a deployer's compliance. Its relevance here is the direction of travel: practical control over personal data matters alongside the explanation of how a model works.

For your own service, map which organisation handles the customer's request and which suppliers must help implement the result. Keep a route for escalation when a profile concerns health, vulnerability or another sensitive matter. Do not invite a chatbot to infer such characteristics simply because a vendor offers that capability. The useful default is to collect and use less, explain what affects the interaction and make corrections achievable by the people operating the service.

People do not have to say Article 16 for a request to correct inaccurate personal data to matter.

Keep the benefit without making every interaction a fresh start

The strongest counterargument is reasonable: customers dislike repeating themselves, and requiring confirmation of every minor preference can make a service tedious. A business that disables all memory may produce worse support, more repeated questions and less relevant suggestions. Correction should not become an excuse to throw away the value of continuity.

The answer is to match certainty and persistence to consequence. A harmless display preference can be easy to change without a lengthy approval process. A tentative product interest can expire or be reconfirmed. An inference that would exclude an option, change treatment or influence an important decision deserves more scrutiny before it is used at all. There is no single retention period or confirmation schedule that fits every preference and every business.

Run a bounded pilot on one low-risk customer journey. Measure repeated wrong assumptions, the time taken to resolve a correction and whether the next interaction respects the outcome. Compare those with the genuine service benefit, such as fewer repeated questions or less time finding relevant information. These are proposed measures for your pilot, not published industry benchmarks. A faster reply is not a win if the person spends the next three exchanges correcting the same profile.

Ask your supplier to demonstrate how the behaviour differs between a confirmed preference and an unconfirmed inference. If the product cannot show what influenced a recommendation or cannot stop reintroducing a corrected assumption, narrow its use. You can still use AI to draft a response from the current enquiry without allowing it to maintain a lasting customer portrait.

Finally, train staff to use the profile as context for a conversation rather than a substitute for listening. Customers change jobs, circumstances, budgets and priorities. A useful assistant should support that change, not argue with it because an old summary sounds authoritative. The right question for the next procurement review is not simply what can this system remember? It is who can correct it, and how will we know that the correction worked?

Good personalisation remembers useful context while leaving room for a person to change their mind.