AI Customer Service Should Measure Resolved Problems, Not Deflected Tickets

ROI & Cost Optimisation

21 October 2026 | By Ashley Marshall

A quiet chatbot queue can hide an expensive customer problem. The question is not how many people avoided a human, but how many actually got help.

A quiet chat window is not a solved problem

A customer asks where a delayed order is. The assistant offers a delivery policy article, the customer closes the window, and no human ticket is opened. On a dashboard that can look like successful deflection. In the customer's life, the parcel is still missing. The distinction is not pedantic. If the customer phones tomorrow, requests a refund through another channel, or simply leaves, the apparent saving was borrowed from a later service failure.

Start by separating three events. Deflection means a contact did not enter a human queue. Containment means an AI conversation ended without a transfer. Resolution means the customer's original problem was actually dealt with. Fin's comparison of resolution and deflection explains why the first two do not prove the third. A page view, a silent conversation and a completed account change are not equivalent units of value.

The commercial temptation is obvious: a lower ticket count looks good in the first month. But a support operation exists to reduce customer effort and fix problems, not to keep a queue graph tidy. If a business rewards a bot for avoiding handoffs, the bot can improve the metric by making the human route harder to find. That may reduce visible workload while increasing repeat contacts and complaints. In practice, put a customer outcome next to every automation metric. Ask whether the issue was completed, whether the customer returned with the same issue, and whether the next human agent received enough context to help quickly. Keep deflection as a capacity measure, but never present it alone as proof of value.

Write a resolution rule before a vendor writes one for you

Vendors use different meanings of the word resolved. One may count a relevant answer followed by silence. Another may require a completed backend action. A third may use an AI grader to assess its own response. Those measures can all be useful, but they should not be compared as though they are interchangeable. Fin's buyer guide explicitly asks who grades a conversation, whether the grade triggers a bill, and what proportion of apparent resolutions are reopened within 24 or 48 hours. Those are sensible procurement questions even if you never buy Fin.

For your own business, define resolution separately for the ten most common contact reasons. For a delivery query, a correct tracking update might be enough. For a cancellation, it is not enough for the bot to describe the process: the request must be completed or clearly handed to someone who can complete it. For a billing dispute, record the promised next step and owner. Keep the denominator honest by including all eligible contacts, not merely the simple cases the AI chose to answer. Record exclusions explicitly, such as suspected fraud or vulnerable-customer cases that must move to a trained person.

Zendesk's explanation of AI resolution rate uses the stronger test of an issue handled end to end without human intervention. That is a useful starting definition, not a substitute for your own case rules. What this means in practice is a short written rubric, signed off by the service owner, before a pilot begins. Apply it manually to a sample of historical cases first. If two reviewers disagree about whether a case was resolved, fix the rule before asking software to score thousands of conversations. Then require the vendor to show how its billed outcomes map to your rubric.

Measure the gap between first contact and the customer's next move

The most revealing data usually lives outside the chatbot dashboard. A customer may start in web chat, email the next morning and telephone two days later. Unless those records can be linked to the same issue, each channel can claim a clean result while the customer has made three attempts. Define a repeat-contact window appropriate to the task, such as 48 hours for a simple answer and longer for a fulfilment or billing action. Match on case and reason, not just customer identity: the same person can have two unrelated questions.

A recent Lorikeet review of published vendor benchmarks reports Ada's roughly 72% containment against 52% automated resolution across more than 550 deployments. That is a 20 percentage point gap in vendor-reported measures, not a universal benchmark for every UK firm. The lesson is the size of the possible measurement error when a contained conversation is treated as a fixed problem. The same review notes that published platform averages use different definitions and customer mixes, so importing a headline percentage into a business case is unsafe.

Use a weekly scorecard with four linked measures: verified resolution by contact reason, repeat contact about the same issue, successful human handoff, and customer effort or satisfaction. Segment the result by channel and by customers who need adjustments. If the headline AI resolution number rises while repeat contacts rise too, investigate the classification or the handoff design. If apparent automation saves five minutes in chat but adds ten minutes to a later phone call, the cost saving has vanished. Where identification is uncertain, label the estimate and audit a sample manually rather than pretending the data is perfect. This makes the scorecard a learning instrument, not a league table.

Treat the handoff as part of the product

A common defence of aggressive automation is that customers can always ask for a person. Sometimes they can, but a transfer button is not the same as a useful handoff. The agent needs the customer's stated problem, what the AI has already tried, any action already taken, and a clear indication of where the AI was uncertain. The customer should not have to reconstruct the entire story. Nor should a bot promise a refund or a deadline that the human team cannot honour.

What this means in practice is designing escalation rules alongside the first AI answer. Trigger a human route for a failed action, repeated misunderstanding, a customer request for a person, a complaint, or a high-impact account decision. Give the receiving team a concise summary with a link to the underlying transcript, while restricting sensitive information to those who need it. Set an owner for unanswered transfers and measure time to human response. A bot that silently drops a case into an unstaffed queue has not solved the customer's issue, even if it completed its own workflow.

There is also a security reason to keep actions bounded. The UK's National Cyber Security Centre warns in its guidance for business leaders that AI systems can produce incorrect information and are susceptible to prompt injection. A customer-facing assistant that can alter addresses, issue credits or expose account information therefore needs permissions, verification and monitoring proportionate to those actions. Do not give a service bot the authority to improvise around identity checks because a conversation feels urgent. For sensitive or consequential cases, the best resolution may be a fast, informed human transfer. Count that as a successful service outcome even though it is not an autonomous resolution.

Build the business case on net value, not an automation headline

The objection is fair: if a bot genuinely handles routine questions, why burden a pilot with complex measurement? Because a sound business case should survive the second month, when novelty fades and difficult cases arrive. You do not need a laboratory-grade experiment. You need an agreed baseline and a way to recognise displaced work. Compare a matched set of contact reasons before and after rollout. Track the cost of licences, integration, knowledge maintenance, quality review, escalations and repeat contacts alongside the minutes of human work actually removed.

Consider a UK retailer that automates order-status requests. If a verified status update prevents a follow-up call, the gain is real. If the system answers correctly but cannot explain an exception, it should hand over with context. If it claims success because the customer left a chat and then emails, there is no saving. Run the pilot with a small but mixed set of historical cases: easy informational requests, requests requiring an action, ambiguous messages and unhappy customers. Review a sample of live outcomes each week. The decision to expand should be based on verified resolutions without worsening repeat contact or customer effort, not on the biggest number in a vendor slide.

Use a simple decision rule: continue if net service cost per genuinely resolved case falls and quality guardrails hold. Fix the knowledge base or workflow if resolution is low but handoffs are clean. Pause or narrow the bot's remit if inaccurate claims, missing handoffs or repeat contacts rise. This is not an argument against AI in customer service. It is an argument for buying the result customers actually care about. Good automation makes a problem disappear. Bad automation makes the customer disappear from one dashboard and reappear in another.

Frequently Asked Questions

Is deflection rate useless?

No. It helps forecast human queue capacity, but it cannot tell you whether the customer got the outcome they needed.

What should count as an AI resolution?

Use a case-specific rule: the answer must be correct or the required action completed, with no avoidable repeat contact within an appropriate window.

How long should we watch for repeat contact?

Start with 48 hours for simple enquiries and a longer window for fulfilment or billing cases. Review the period against your actual service cycle.

Can a human handoff still be a good outcome?

Yes. A prompt transfer with the issue, previous steps and a clear owner can be safer and better for the customer than an inaccurate automated answer.

How do we compare vendors with different resolution claims?

Ask each vendor for its written definition, denominator, exclusions, grading method, billing trigger and reopened-case rate. Apply your own rubric to the same case sample.

Should we trust AI to grade its own conversations?

Automated grading is useful for scale, but independently review a sample and compare the scores with repeat contacts and customer feedback.

What is a sensible first pilot?

Use one or two high-volume contact reasons and a mixed test set that includes exceptions, action requests and unhappy customers. Review live outcomes weekly.

Which financial measure belongs in the board report?

Use net cost per genuinely resolved case, including licence, integration, maintenance, human review, escalations and repeat contacts.