AI Document Comparison Needs a Missed-Deletion Test

The most expensive change in a revised supplier agreement might be the clause your AI comparison never mentions.

A tidy comparison table is not proof that nothing was missed

A supplier sends a revised agreement. You upload the old and new files, ask what changed, and receive a neat table of differences. The payment date moved, the contact name changed and the introduction was shortened. Everything looks manageable. But the new version also dropped the sentence promising to return your data after termination. That omission never appears in the answer. The comparison is useful, fluent and incomplete in precisely the place your business needed it to be reliable.

This is a different problem from an invented quotation. Every finding the assistant gives you might be correct. The failure is a missing finding. Checking the accuracy of the rows you received cannot establish the completeness of the rows you did not receive. For a UK business using AI to review procurement documents, insurance schedules or service agreements, that distinction should determine the workflow rather than remain a footnote in staff training.

An AArena review published on 24 September 2026 distinguishes revision comparison from comparing the substance of unrelated proposals. It names Adobe Acrobat, ChatGPT, Claude and Humata as options for different tasks, while explicitly saying its shortlist is not a hands-on accuracy ranking. That is a useful buying distinction, not evidence that any product catches every commercially important change.

Start by stating the job correctly: identify changes between these two specific versions, preserve evidence from both, and disclose anything not assessed. Do not ask merely for a summary of differences. A summary invites selection. An acceptance check needs a defined inventory of obligations and an explanation of what happened to each one. The practical question is not whether the answer reads well. It is whether the missing data-return clause has somewhere unavoidable to appear.

The clause the AI never mentions may matter more than every change it lists.

Build the change inventory before asking for the business impact

Before uploading anything, identify which file is the baseline and which is the proposed revision. Save both originals unchanged and record their filenames and revision dates. A revised filename is not proof of approval. Where several attachments arrived in the same email thread, confirm which pair the reviewer is actually meant to compare. Otherwise a technically accurate comparison can still answer the wrong question.

Create a short inventory of the commitments the decision depends on. For a supplier agreement, that might include service scope, fees, renewal, termination notice, data return, support hours, liability and exclusions. This is not a universal legal checklist. It is a task-specific coverage list, agreed by the person responsible for the purchase. Add relevant schedules and appendices explicitly rather than assuming that an upload automatically makes every attachment available.

Next, distinguish three outputs. The first is an observed textual change, supported by passages from the two versions. The second is a possible commercial implication, clearly labelled as interpretation. The third is a decision or question for the accountable reviewer. Removing the words “including onboarding” is an observed change. Saying that onboarding will definitely cost extra is unsupported unless the new agreement establishes that consequence. Asking the supplier to confirm onboarding coverage is a sensible next step.

In practice, put those distinctions into the output columns: obligation, baseline evidence, revised evidence, observed change, possible implication and reviewer action. Add a coverage state for every inventory item, including unchanged and unresolved. This avoids a common failure of changes-only tables: silence can mean no change, unreadable content, failed retrieval or a missed clause. Those are different states with different next actions. Your reviewer needs to see which state applies before they can accept the comparison or decide what to inspect themselves.

Separate finding a change from deciding whether that change is acceptable.

Use an ordinary comparison tool as an independent check

Many businesses already have a suitable comparison feature in their document software. Check that before buying a specialist AI service or building a custom pipeline. A dedicated comparison report and a generative explanation do different jobs. The report can help locate edits; the assistant can organise questions and explain possible implications. Neither should be described as infallible, and neither removes the need to choose the correct input files and settings.

Adobe documents Compare Files in Acrobat Pro as producing a report of differences between two PDF versions. Its settings distinguish flowing reports, presentation pages and scanned documents. It also states that annotations, formatting and background differences are turned off by default in the results view. That named example matters because “I ran comparison” says little unless the settings match what you needed to detect. This is an existing-product example, not a recommendation to buy a subscription.

For documents you already hold in an editable format, check the comparison capability in your existing word processor before converting everything to PDF. For an open-source route, diff-pdf is an option to investigate for visual PDF differences, subject to your team checking current maintenance and platform compatibility. A visual difference still needs interpretation: reflow, a changed font or a moved page can create noise without changing a commercial obligation. Conversely, a tiny word deletion can be material.

Run the independent report first, then reconcile its meaningful findings with the AI inventory. If the report shows a deleted paragraph that has no corresponding AI finding, investigate that gap rather than asking the assistant whether it is confident. If the AI claims a deletion but the passage merely moved into a schedule, correct the classification. This gives reviewers a concrete discrepancy to resolve. It does not prove complete coverage, but it avoids relying on the same generated summary to supply both the findings and the assurance.

Do not make one generative answer its own completeness test.

Test deletions, small words and numbers with known answers

Create two synthetic supplier documents that resemble your real workload without containing client information. Keep one as the baseline. In the revision, insert a small set of deliberate changes and keep a private answer key. Include a deleted obligation, a changed notice period, a different unit, a new exclusion, a moved paragraph and an unchanged clause. These are proposed test cases, not research findings or claims about a particular model’s failure rate.

For example, change an illustrative notice requirement from 30 calendar days to 30 business days, while leaving the numeral unchanged. Remove a promise to provide onboarding. Change a fee from £500 per year to £500 per month. Move the data-return clause into an appendix without changing its wording. The correct output must distinguish a changed obligation, a deletion, a unit change and a relocation. Finding the same number in both files is not enough, and calling every moved passage a deletion is not acceptable either.

Include at least one difficult layout from your actual documents, such as a table continuing across pages or a qualification printed underneath a pricing schedule. CustomGPT.ai’s October 2026 document-analysis guide separates source-cited question answering from structured extraction and advises evaluating scanned-document OCR separately. Its product perspective is commercially interested, so treat it as workflow guidance rather than independent proof of comparative accuracy.

Score the comparison against the answer key, not against its tone. Count material changes detected, material changes missed, unsupported change claims and cases correctly marked unresolved. A harmless wording edit should not cancel out a missed termination condition in an average score. Require human review for the consequential cases you cannot yet detect reliably. Repeat the same test after changing the document reader, model, prompt or comparison settings. Preserve the difficult examples so that an improvement to fee detection does not quietly break deletion detection.

A test must contain changes whose answers you know before the AI sees them.

Make the prompt demand coverage rather than confidence

Use a bounded prompt that defines the files, the baseline, the obligations and the evidence required. A workable starting point is: compare baseline.pdf with revision.pdf; treat baseline.pdf as the earlier version; account for every listed obligation; return short supporting passages and source locations for both documents; distinguish changed, unchanged, relocated and unresolved; separate observation from interpretation; report unreadable or unreviewed material. Ask for the coverage record before asking for a recommendation.

Add an explicit rule about absence. If a clause cannot be located in the revised document, the assistant should say “not found in the reviewed content” and name the material it inspected. That is narrower than declaring the clause deleted from the agreement. A schedule could be missing from the upload, a page could be unreadable, or retrieval could have returned only selected passages. The correct next step is to locate the missing material or ask a reviewer, not to turn incomplete access into a confident legal conclusion.

Two recent Digital Applied guides, on PDF reading order and table checks and document reading evidence, explain why text presence does not establish preserved meaning. They are related publications, not independent benchmark studies. Their relevant point for comparison is practical: a table value needs its heading and qualification, and a page reference needs a known relationship to the supplied document.

In practice, ask a colleague to locate one finding using only the comparison output. Can they open the correct file, find the exact passage and understand what changed without repeating the whole exercise? If not, improve the evidence fields before expanding usage. Also include an unchanged control clause. A system that confidently invents a revision where none exists can waste reviewer time and distort negotiations just as a missed deletion can hide a risk.

“Not found” is a reading result, not automatically a contractual deletion.

Keep acceptance with the person who owns the commitment

The strongest counterargument is straightforward: if someone must still check the result, where is the saving? The answer is that a useful comparison can organise the review, locate relevant passages and expose specific questions. It need not make the final acceptance decision to reduce effort. But the benefit disappears if the reviewer must reconstruct every citation, discover missing pages and untangle unsupported implications. Measure total reviewed work, not the seconds needed to produce a polished table.

Keep the review proportional. An internal draft of a low-stakes briefing can tolerate a different process from a supplier agreement committing money or handling personal information. Define who checks fees, who checks operational obligations and who decides whether specialist advice is needed. A purchasing manager might approve a documented commercial change but still route liability wording to an appropriate adviser. An AI tool does not acquire that authority because it can explain the clause in plain English.

The UK government’s Introduction to AI assurance, published in 2024, frames assurance around measuring, evaluating and communicating trustworthiness. It is background guidance, not a new statutory requirement to use this particular checklist. Applied here, the useful discipline is to communicate what was tested, which documents were covered and what remains unresolved. Where personal information is involved, use your organisation’s approved processing arrangements rather than assuming that permission to read a document includes permission to upload it anywhere.

For a first pilot, choose one recurring document family and a small approved set of known-answer comparisons. Record review time, missed material changes, false alarms and unresolved cases. Decide in advance which failures block operational use. Keep the original documents, comparison report, AI output and reviewer decision together under an appropriate retention policy. You will then have something more useful than a promise that the model is good at PDFs: evidence about whether this specific workflow helps your team identify and accept changes responsibly.

Let AI shorten the review, not silently redefine who accepts the risk.