Measure AI Invoice Extraction by Exceptions, Not Accuracy

Tools & Technical Tutorials

15 October 2026 | By Ashley Marshall

A tool can read almost every field correctly and still leave your accounts team clearing a queue. The real test is what reaches the ledger without a human rescue.

The metric in the sales deck is not the metric in your accounts team

A supplier can demonstrate impressive field accuracy and still leave your finance team with nearly the same workload. Reading an invoice number correctly is one task. Getting an invoice from the shared inbox into the ledger, matched to a purchase order and ready for payment without intervention, is another. Procurement should insist on measuring the second outcome. An analysis published in September 2026 cites Ardent Partners' survey of 212 AP and finance leaders: average touchless processing was 32.6%, even while 55% of teams used automated data capture. Those are survey findings, not a prediction for your own business, but they expose the gap between recognition and resolution.

Start by defining four separate measures. Field accuracy is the proportion of extracted values that match a checked answer. Document accuracy is the proportion of invoices with every required value correct. Straight-through rate is the proportion that reaches your agreed end state without human correction or approval. Exception cost is the time and money spent resolving everything that does not. A vendor may report the first and imply the third. Ask for all four, calculated on your documents and under your approval policy.

The maths explains why this matters. If twenty required fields each have a 98% chance of being right, and errors were independent, the probability that every field is right would be about 67%. Real errors are not independent: poor scans and unusual suppliers create clusters. That makes the simple calculation an illustration, not a forecast. It does show why a headline accuracy percentage cannot replace an invoice-level acceptance test. What this means in practice is straightforward: write the end state into the pilot brief before inviting a demo. If a clerk still has to open every record to check totals, the process is assisted entry, not touchless automation.

Build a test set from the invoices that usually break the process

A polished vendor demonstration often uses clean, single-page PDFs from familiar suppliers. Your queue may also contain photographed receipts, credit notes, emailed duplicates, scanned construction invoices, multi-page line items, foreign currencies and purchase orders whose references are missing. If those cases are absent from the pilot, the reported accuracy describes the demonstration rather than your workload. Take a recent, representative sample from the actual AP inbox, remove or protect personal data appropriately, and record the source, format, supplier type and reason each invoice previously needed attention.

Do not let the supplier select the only test documents. Set aside a holdout sample that nobody uses to tune prompts, templates or confidence thresholds. Keep clean invoices and difficult invoices as separate cohorts so an overall average cannot hide failure on the documents that cost your team the most. A practical first pass could use 100 recent invoices, with an additional small set of known edge cases; the exact number is a pilot design choice, not an industry benchmark. Have two finance reviewers agree the expected values and handling route for disputed examples before scoring the tool.

Include negative tests. Feed in a statement that should not become an invoice, a duplicate of an existing bill, an altered bank account, a zero-rated VAT item and a credit note with a misleading positive-looking total. Success may mean refusing to post and sending a clear exception to a named person. This is not a failure of automation. It is a correct control decision. The recent AIMultiple OCR review notes that recognition alone returns text rather than the structured, validated record finance needs, and that vision-language systems can invent text absent from the page. Preserve the original image alongside the extracted value so a reviewer can see where a figure came from.

Score the whole workflow, including the human queue

Give every test invoice a final disposition: posted correctly, routed correctly to an exception, posted incorrectly or lost. The third and fourth outcomes deserve particular weight. A wrong supplier bank account or duplicate posting can be far more expensive than five minutes of clerical review. Record the time from receipt to disposition, the number of human touches, the reviewer minutes and the reason for each exception. Then group exception causes: unreadable source, extraction error, supplier master mismatch, missing purchase order, missing goods receipt, policy approval or integration failure.

That breakdown determines where the next pound should go. If the problem is supplier master data, changing OCR engines will not solve it. If invoices arrive without purchase order references, improve the ordering process and supplier instructions. If your ERP rejects otherwise correct records, fix mapping and retry handling. Zenthos's September analysis reports the Ardent survey found exceptions were the top AP challenge for 53% of respondents and that only 5% reported fully automated exception handling. Those figures are useful context, but your own tagged exception queue is the evidence for investment.

What this means in practice is to ask the vendor to show the complete exception screen, not just the extraction preview. Can a clerk compare the highlighted source field with the proposed value? Is the reason for a failed three-way match explicit? Can the item be reassigned, corrected and returned to the workflow without starting again? Does a retry create a duplicate ledger entry? Test an interrupted API call and verify idempotency. For every exception, capture who resolved it and how long it took. A tool that correctly identifies uncertainty but requires an awkward manual workaround may simply move labour from data entry to queue management.

Compare price per resolved invoice, not price per page

Cloud extraction prices look remarkably low when viewed alone. An October 2026 comparison of US list prices put basic OCR at $1.50 per 1,000 pages on Azure, AWS and Google. It listed prebuilt invoice extraction at $10 per 1,000 pages for Azure and AWS, while the Google invoice parser was priced per document, at $0.10 for up to ten pages. These are published US first-tier prices before tax, not UK quotes. They are also prices for different product capabilities. The comparison itself cautions that a layout service and a labelled forms service are not equivalent.

For a UK buyer, the purchasing model should include licences, implementation, integration, storage, human review, supplier onboarding, support and any model or feature surcharge. Convert currencies using an explicit rate on the date of the business case, then rerun the calculation at different monthly volumes. Divide total cost by invoices that reach the agreed end state correctly, not by pages parsed. If 10,000 pages are processed cheaply but 4,000 records need manual intervention, the page price is a small line item in a much larger operating cost.

Compare the pilot against today's baseline, not a fictional fully manual process. Measure current cost and cycle time by invoice class. Some simple invoices may already flow well through Xero, Sage, Dext or an existing AP tool. Replacing that route with bespoke AI could add risk without moving the outcome. Conversely, a high-volume queue of unusual invoices might justify specialised extraction or a custom validation layer. Price the exception workflow before signing a broad rollout. A lower model bill is only a saving if accuracy, controls and staff time hold up at production volume.

Put UK controls into the acceptance criteria

Invoices are business records, but many also contain personal data: a sole trader's name, a contact's email address or bank details. The ICO's guidance on AI and data protection separates questions of lawfulness, accuracy, fairness, transparency and accountability. It also says the guidance is under review following the Data (Use and Access) Act. Do not turn a vendor's security badge into a complete data-protection assessment. Ask where documents are processed, what is retained, who can access the images, whether inputs are used to train a model, and how deletion and subject-rights requests work.

Finance controls should be just as concrete. Verify totals by arithmetic, not model confidence alone. Check supplier identity against an approved master record. Treat a changed payment destination as a human-verification event through an independent channel. Make credit notes, duplicate invoice numbers and VAT treatment explicit test cases. Decide which exceptions must always receive human approval, and document who owns threshold changes. Confidence scores can prioritise review, but they do not prove a value is correct or that payment is authorised.

Finally, make the acceptance test repeatable. Keep a versioned sample set, scoring rules, configuration and results. Rerun it after a model update, prompt change or ERP mapping change. The output should be a short decision record: what improved, what regressed, how many invoices were posted without correction, how many were safely held, and what remained unresolved. That lets the finance director approve a controlled expansion rather than accepting a single impressive demo. The misconception to reject is that more accurate reading automatically means fewer AP staff hours. The real system is documents, policy, supplier data, approvals and people, with the model only one component.

Frequently Asked Questions

What is a good touchless invoice processing rate?

There is no universal target. Set a baseline from your own invoice mix and agree an improvement threshold for each document class before the pilot.

Is 99% field accuracy enough for automatic posting?

Not on its own. Check complete-document accuracy, payment controls and whether the record passes matching and approval rules.

How many invoices should a pilot test?

Start with a representative sample large enough to include your common failure types, then keep a separate holdout set. One hundred recent invoices can be a useful initial diagnostic, not a statistical guarantee.

Should a tool automatically reject uncertain invoices?

It should route them to a clear human exception workflow. An uncertain result must not silently become a ledger entry.

Does basic OCR include invoice validation?

No. OCR reads text. Validation, matching, duplicate checks and ERP posting are separate capabilities that must be tested.

What matters most in a vendor demonstration?

Watch an invoice move from receipt to the agreed end state, including failed matches, correction, audit trail and retry behaviour.

How should UK GDPR affect the pilot?

Map the personal data in invoices, document the processing purpose, inspect processor terms, retention, access and transfers, and use the ICO guidance appropriate to your case.

When is bespoke document AI worth considering?

When existing AP tools fail on a material volume of unusual documents or integration requirements, and a controlled pilot shows a better total cost per resolved invoice.