Your AI Knowledge Base Needs a Superseded-Document Test

A perfectly cited answer can still quote last month's policy. The real test is whether your assistant knows which document no longer applies.

A real citation can still support the wrong answer

Your assistant can quote a genuine company document, show a working source link and still give a customer the wrong answer. The problem is not necessarily a hallucination. It may have retrieved a perfectly authentic policy that stopped applying last month. If your acceptance test only checks that answers have citations, you can approve a system that is confidently out of date.

Consider a hypothetical service business replacing a standard call-out charge. The old price list remains in a shared folder because accounts need it for historical disputes. A new list has been approved, but both documents use similar language. Search relevance can favour the older file because it contains more examples. A generated answer may then present the old charge as today's price. Nothing needs to be invented for the business to make a costly mistake.

Recent platform changes make source checking easier, but they do not remove this distinction. Google's 16 October 2026 Gemini Enterprise update makes configurable citation chips for custom MCP data stores generally available. That is useful traceability. It is not a guarantee that the cited policy is approved, applicable to this customer or effective on the date of the question.

Before adding more documents, decide what the assistant is allowed to treat as current authority. Write that rule down in business language: current approved UK service pricing for this customer type, effective on the requested date. Then make the implementation prove it follows the rule. A source link answers where the words came from. Your test must also answer whether those words should govern the decision being made.

A citation proves that a source exists, not that its instructions still apply.

Give every source a business status, not just a timestamp

The most recently edited document is not automatically the right document. An administrator might correct a spelling mistake in an archived policy today. A replacement policy might have been approved yesterday but only become effective next month. A customer-specific contract might override the standard terms even though it has an older modification date. Freshness needs meaning, not simply a newest-first sort.

Start with a small source register for the workflow you are actually deploying. For each policy or reference record, capture a stable identifier, accountable owner, approval state, effective date, expiry or review date where relevant, applicable customer or product scope, and the record it replaces. Preserve the distinction between a document's last edit, its approval and the point at which it becomes operational. These fields do not need an elaborate new platform; an existing controlled library may already provide them.

The underlying distinction also matters for personal data. The ICO's accuracy guidance explains that historical information is not necessarily inaccurate merely because circumstances changed. Its source and status must be clear, and personal data must be kept up to date where necessary for the purpose. The page is under review following the Data (Use and Access) Act, so check the current guidance before relying on a legal interpretation.

What this means in practice is that your assistant needs separate handling for current instructions and historical evidence. Keep an old employment policy where retention is justified, but do not let it answer a question about current entitlement as though nothing changed. Ask the responsible manager to approve the source rules before an engineer configures retrieval. Otherwise the technical team is being asked to invent a policy hierarchy that management has never agreed.

Last edited, approved and effective are three different facts.

Test the whole update path, not the connector's green tick

A successful connection tells you surprisingly little about whether the next answer will use the right information. A change may need to travel through source publication, connector polling, extraction, indexing and answer generation. Some systems also reuse earlier retrieval results or conversation context. A green synchronisation indicator can coexist with a stale answer if you only observe one part of that chain.

On 23 September 2026, AWS announced native Salesforce and Zendesk connectors for Amazon Bedrock Managed Knowledge Base. The named source types include Salesforce knowledge articles and Zendesk articles and community posts; the connectors handle crawling, metadata extraction and incremental synchronisation. That can reduce integration work. It does not give your business a measured end-to-end update time for its own documents and questions.

Use an authorised test area to change a harmless but recognisable fact, such as a sample service window. Record when the source owner approved it, when the connector observed it, when the searchable content changed and when a fresh answer reflected the replacement. Repeat the question in a new conversation as well as an existing one. Then mark the old version superseded and check that it no longer drives current answers.

Set the acceptable delay according to the consequence of being wrong. A background training guide may tolerate an overnight refresh; a changing stock promise may require a current system lookup or a human check before commitment. These are design choices, not universal industry limits. If the business cannot tolerate the measured delay, narrow the assistant's role. Let it explain the process while a live system supplies the decisive fact, rather than pretending a faster model can compensate for information that has not arrived.

Measure the delay from approved source change to corrected answer.

Run a replacement test with deliberate contradictions

Ordinary demonstrations tend to ask questions whose answers are easy to find. A useful freshness test deliberately makes retrieval uncomfortable. Prepare an approved current document, a superseded predecessor containing a different answer, and a future-dated replacement. Add a legitimate exception for one customer type. Ask the same underlying question with and without the context needed to select the exception.

For a hypothetical pilot, use twenty questions split across routine current answers, historical questions, future effective dates, exceptions and missing information. Twenty is a manageable starting exercise, not a statistically validated sample size. The point is to expose different failure modes before expanding the evidence set. Record the expected source and acceptable behaviour for each question before seeing the assistant's answer, so a fluent response does not redefine what counts as correct.

Your checks should go beyond whether the number in the answer matches. Does the assistant identify the effective date? Does its source actually support the claim? Does it keep historical and current conditions separate? If the customer scope is unknown, does it ask for clarification rather than merge two policies? If the correct source is unavailable, does it stop short of presenting an obsolete rule as a safe fallback?

Microsoft's August 2026 Azure AI Search updates describe preview citation URLs that allow authenticated lookup of backing documents. The same page warns that preview functionality is not recommended for production workloads and lacks a service-level agreement. Treat that as a useful example of the distinction between technical capability and deployment readiness. Whatever platform you choose, keep a small repeatable replacement test as part of acceptance, then rerun it after connector, indexing or retrieval changes. A polished first demonstration is not continuing evidence.

Test current, historical, future-dated and conflicting sources separately.

Do not fix stale knowledge by deleting business history

The obvious objection is reasonable: if the assistant can only see one current folder, why not remove everything else and stop worrying? For a narrow pilot, restricting the source set can be an excellent choice. You do not need a knowledge graph or a custom retrieval platform to answer a dozen controlled internal questions. A small, owner-approved library is often easier to understand and maintain.

The limit is that operational businesses need history, exceptions and evidence. Accounts may need the terms that applied when an invoice was raised. HR may need to distinguish a current policy from the version that governed an earlier event. Deleting old records solely to improve an assistant can damage a legitimate audit trail or conflict with an agreed retention policy. Keep retention decisions separate from whether a record may guide a current answer.

Another common shortcut is to add an instruction saying always use the latest information. That can help communicate intent, but it does not supply missing approval metadata or repair a stalled ingestion job. Nor does retrieving more text guarantee a better result; it can introduce additional obsolete versions. An assistant cannot reliably honour a distinction that the source library does not represent.

What this means in practice is a staged choice. Start by excluding archives from current operational retrieval. Provide a clearly separate historical route only if users need it. Make exceptional cases explicit rather than forcing a model to infer them from file names. If your existing SharePoint library, help centre or document system can manage approval and version status, use those capabilities first. Commission additional engineering only when you can demonstrate the gap with real failed tests. The objective is dependable answers, not the most sophisticated document architecture your supplier can describe.

Archive access and current operational authority are different permissions.

Make freshness someone’s weekly operational responsibility

A freshness test is not finished when the initial deployment passes. People change policies, suppliers change terms and source owners move jobs. The assistant will only remain aligned if those ordinary business events reach the knowledge workflow. Put responsibility beside the process that already approves the underlying information, rather than leaving it as an invisible maintenance task for the person who built the chatbot.

For each important source, name a business owner and a technical contact. The owner decides whether the document is approved and applicable; the technical contact checks whether the retrieval system reflects that state. Define what happens if review is overdue or synchronisation fails. Low-risk guidance might display an age warning. A customer commitment might need to be withheld until the source is verified. Agree that behaviour in advance instead of making it up during an incident.

Use a compact operational record: source identifier, current approved version, last successful update, last replacement-test result and unresolved exceptions. For a pilot, review it weekly and after significant policy changes. That frequency is a proposed operating habit, not a statutory deadline. Increase or reduce it based on how often the information changes and how consequential a stale answer would be. Include a route for staff to report an outdated answer with the question, source and time, without copying unnecessary personal data.

Finally, judge the workflow by corrected decisions, not document volume. Report how many sampled answers used the approved applicable source, how long replacements took to appear, and whether blocked or uncertain cases reached a responsible person. If an old rule escapes into a customer-facing answer, correct the source path and rerun the same case before restoring confidence. Better language generation is useful, but it cannot substitute for an organisation deciding which facts are allowed to speak for it today.

An AI knowledge base needs an owner for what is true today, not just an administrator for files.