Build AI Retrieval Test Sets Before Connecting Business Assistants to Live Knowledge Bases

Tools & Technical Tutorials

29 July 2026 | By Ashley Marshall

The fastest way to make a business AI assistant look clever in a demo is to connect it to everything. The fastest way to make it unsafe in production is exactly the same move.

Retrieval is the failure point most demos hide

Most business AI assistant pilots are judged too late in the chain. Someone asks a polished question, the assistant writes a fluent answer, and the team decides whether the answer sounds useful. That is not enough. In a retrieval-augmented generation workflow, the assistant can only answer from the evidence it has been given. If retrieval fails, the language model is left to fill the gap with general knowledge, old assumptions or a confident refusal. The visible answer may be the last step, but the first serious test is whether the system fetched the right knowledge in the first place.

This matters more when the assistant is connected to live business material: policies, product documentation, sales decks, CRM notes, HR handbooks, contracts, support articles, pricing pages, incident runbooks and internal procedures. A human can often spot when a document is old, irrelevant or incomplete. A retrieval system needs explicit tests for that judgement. Before go-live, the team should be able to show that a known question retrieves the current policy, not the retired PDF from last year; that a regional pricing question retrieves the UK source, not the US deck; and that a sensitive HR question triggers escalation rather than a shallow answer stitched from irrelevant chunks.

OpenAI's guidance describes evaluations as tests that check model outputs against style and content criteria, and says they are especially important when upgrading or trying new models. That idea should be moved upstream for business assistants. Build tests around the retrieval step before you argue about tone, model choice or interface polish. If the assistant cannot reliably find the right material in a fixed test set, connecting it to a live knowledge base simply gives it more ways to be wrong.

For UK businesses, the risk is practical rather than theoretical. A poor retrieval pipeline can leak old terms, misstate policy, miss a required escalation, cite the wrong source or make staff believe an answer is grounded when it is not. A retrieval test set gives leaders a release gate they can understand: these are the business questions, these are the approved sources, these are the expected citations, and these are the cases where the assistant must say it does not know.

A good test set is built from business questions and source truth

A retrieval test set is not a random pile of prompts. It is a compact representation of the knowledge work the assistant is expected to support. Each test case should include a user question, the expected source document or chunk, any unacceptable sources, the answer boundary, the citation requirement, and the pass or fail criteria. For a customer support assistant, that might include refund limits, cancellation language, delivery exceptions and vulnerable customer escalation. For an internal operations assistant, it might include onboarding processes, procurement approval thresholds, data handling rules, cyber incident steps and sales qualification criteria.

The source truth matters. If the knowledge base contains five versions of a policy, the test set should identify which one is authoritative and why. If a question should retrieve more than one source, record that explicitly. If a question is deliberately unanswerable from approved material, the expected behaviour should be refusal or handover. This is where a business assistant differs from a search box. Search can return a list and let the user decide. An assistant turns retrieved context into operational guidance, so the test set has to define the evidence standard.

OpenAI's SimpleQA benchmark gives a useful pattern, even though it is not a business retrieval benchmark. The paper describes 4,326 short, fact-seeking questions, each graded as correct, incorrect or not attempted, and says the benchmark was designed so questions have a single indisputable answer. It also reports that GPT-4o and Claude both scored less than 50 percent on the benchmark when it was created, which is a good reminder that fluent frontier models still need targeted factual tests. Business retrieval sets do not need thousands of cases to start, but they do need the same discipline: a question, a gold answer or source, and a clear grading rule.

In practice, start with 100 to 200 cases for a focused assistant and expand from there. Sample from search logs, support tickets, sales questions, staff Slack threads, call transcripts and policy owner interviews. Include common cases, edge cases, outdated-document traps, ambiguous questions, prompt-injection attempts embedded in documents, and queries where the right answer is that the knowledge base is silent. Label each case with owner, risk level and source freshness. The goal is not academic purity. It is enough coverage to stop obvious retrieval failures before they reach staff or customers.

Measure retrieval separately from the final answer

The common misconception is that if the final answer is acceptable, retrieval must be working. That is dangerous. A language model can produce a plausible answer from a weak context window, especially when the topic is common. It can also write an answer that looks grounded while citing the wrong document. Testing only the final response hides whether the system found the correct evidence, whether it ranked the evidence properly, and whether the model used that evidence rather than improvising around it.

Microsoft's Azure AI Foundry documentation separates process evaluation from system evaluation for RAG. Its RAG evaluator table names six evaluators: Document Retrieval, Retrieval, Groundedness, Groundedness Pro, Relevance and Response Completeness. The distinction is useful for business teams. Document Retrieval measures whether the right documents were retrieved against ground truth labels. Retrieval assesses whether the context chunks are relevant to the query. Groundedness asks whether the response aligns with context without fabrication. Relevance and Response Completeness judge the final answer. Those are different questions, and they should not be collapsed into one overall vibe score.

The same Microsoft page lists search quality metrics for document retrieval, including Fidelity, NDCG, XDCG, Max Relevance and Holes, and shows evaluator outputs on a 1 to 5 scale with a default pass threshold of 3 for several evaluators. You do not need to use Azure AI Foundry to adopt the principle. You can implement a simpler version in Braintrust, LangSmith, Humanloop, OpenAI Datasets, pytest, Ragas, TruLens, DeepEval or a custom notebook. What matters is that retrieval gets its own score before generation is judged.

A practical test record should capture top-k retrieved chunks, expected chunk IDs, rank position, source version, answer text, cited sources, judge score and reviewer notes. For high-risk assistants, require that the correct source appears in the top 3 and that prohibited sources do not appear. For lower-risk assistants, top 5 may be acceptable. Track false positives, where the assistant retrieves something irrelevant, and false negatives, where it misses required context. Over time, these metrics help tune chunking, metadata, permissions, embedding models, hybrid search, reranking and freshness rules without guessing.

UK governance makes pre-live testing a control, not a luxury

Retrieval testing is not only an engineering preference. It is part of responsible deployment. The UK government's AI Cyber Security Code of Practice sets out 13 principles for AI systems, including documenting data, models and prompts, conducting appropriate testing and evaluation, monitoring system behaviour, and ensuring proper data and model disposal. The same GOV.UK page says the proposed intervention was endorsed by 80 percent of respondents to DSIT's 2024 Call for Views, with support for each principle ranging from 83 percent to 90 percent. That is a useful policy signal: testing and evaluation are becoming expected evidence, not optional polish.

Principle 9 is especially relevant. GOV.UK says developers should conduct testing and evaluation throughout the AI lifecycle, and system operators shall conduct testing before the system is deployed with support from developers. That language maps directly onto retrieval assistants. If your business is the system operator connecting a vendor assistant to SharePoint, Google Drive, Confluence, Notion, Salesforce, Zendesk or a custom document store, you still own the deployment decision. Vendor reassurance is useful, but it does not replace your own evidence that your knowledge, permissions and workflows behave correctly.

NCSC's secure AI development guidance reinforces the lifecycle point. It frames AI security across secure design, secure development, secure deployment, and secure operation and maintenance, and says AI systems face novel security vulnerabilities alongside standard cyber threats. For retrieval systems, those vulnerabilities include data poisoning, indirect prompt injection in documents, oversharing through poor permissions, and unmonitored drift as the knowledge base changes. A retrieval test set is one of the few controls that can catch these issues before users do.

Data protection adds another layer. The ICO guidance on fairness in AI says fairness is a key principle when processing personal data and that organisations should consider data protection by design throughout the AI lifecycle. If a knowledge assistant can retrieve employee records, customer case notes or complaint histories, the retrieval layer is part of the privacy risk. Test sets should include permission-boundary cases: a manager asking for HR information they should not see, a support agent asking for another customer's details, or a user trying to reveal hidden system instructions. The expected answer should be refusal, masking or escalation, not creative helpfulness.

Build the first version without boiling the ocean

Some teams avoid retrieval testing because they think it has to be a large research programme. It does not. The first useful test set can be built in a week if the scope is narrow and the business owner is involved. Pick one assistant, one knowledge area and one release decision. For example: an internal sales assistant that answers questions about product eligibility, implementation timelines and pricing constraints. Then identify the 20 most common questions, the 20 most commercially risky questions, the 20 questions that staff currently answer inconsistently, the 20 questions that should trigger handover, and the 20 questions designed to break retrieval boundaries.

Each case needs a lightweight but structured schema. A useful starting point is query, expected_sources, forbidden_sources, minimum_answer_points, must_refuse, risk_level, owner, review_date and notes. If the assistant has permissioned retrieval, add user_role and expected_access_result. If the knowledge base has versioned documents, add source_version and valid_from. Keep it simple enough that policy owners can review it in a spreadsheet, but structured enough that engineers can run it automatically.

Implementation usually starts with exported questions and manually labelled source truth. Run the assistant against each query in a non-production environment. Store the retrieved chunks, generated answer and citations. Then review failures by type: missing document, poor chunking, wrong metadata, stale source, permission leak, ambiguous query, insufficient refusal, weak reranker or answer not grounded in retrieved context. Each failure should lead to one change in the retrieval system or one change in the knowledge base. Otherwise the test set becomes theatre.

The counterargument is that business knowledge changes too quickly for a fixed test set to stay useful. That is partly true, which is why the set should be versioned and reviewed. But live change is exactly why the test set matters. When a pricing page, HR policy or support process changes, rerun the tests. If a new source breaks old answers, you want to know before Monday morning. Static tests are not the whole answer, but they are the baseline that makes change visible.

Turn retrieval tests into a release gate and monitoring loop

The real value comes when retrieval tests move from a one-off pilot exercise into normal delivery. Every assistant release should run the retrieval set before connection to live knowledge, after major document updates, after chunking or embedding changes, after model or reranker upgrades, and after permission model changes. Treat it like regression testing for operational knowledge. If the assistant is going to answer staff or customers, the business should know which tests passed, which failed, who accepted the residual risk and what changed since the previous run.

A sensible release gate has three layers. First, retrieval quality: did the correct source appear at the required rank and were prohibited sources excluded? Second, grounded answer quality: did the answer use the retrieved context, cite the source and avoid unsupported claims? Third, operational control: did the assistant refuse or escalate where the knowledge base was silent, sensitive or permission-restricted? These layers give product, engineering, data protection and business owners a shared language. Instead of arguing whether the assistant feels ready, they can inspect evidence.

For implementation, wire tests into CI where possible. A nightly job can run a small smoke set against the latest index. A pre-release job can run the full set and publish a report to GitHub Actions, Azure DevOps, LangSmith, Braintrust or a simple dashboard. Store trend lines for recall, precision, groundedness, citation accuracy, refusal accuracy and permission-boundary pass rate. Add failed production conversations back into the test set after sanitisation. Over time, the test set becomes institutional memory: the cases your assistant must never get wrong again.

This also gives leaders a better buying question for vendors. Do not only ask whether the assistant supports SharePoint, Confluence, Google Drive or Salesforce. Ask how retrieval is evaluated, whether customers can supply ground truth labels, whether logs expose retrieved chunks, whether citation accuracy is measurable, whether permission failures are testable, and whether the vendor can rerun tests after model or index changes. A credible supplier should welcome those questions. If they cannot explain their retrieval evaluation story, be careful about connecting them to live knowledge.

Frequently Asked Questions

What is an AI retrieval test set?

It is a labelled collection of business questions, expected sources, forbidden sources, answer requirements and pass or fail criteria used to test whether an AI assistant retrieves the right knowledge before generating an answer.

How many questions should a first retrieval test set include?

For a focused assistant, 100 to 200 well-chosen cases is a useful starting point. Coverage matters more than volume, so include common questions, risky questions, stale-source traps, unanswerable questions and permission-boundary cases.

Should we test retrieval if the vendor already provides citations?

Yes. Citations prove only that a source was attached to an answer. They do not prove the source was the right one, that the answer used it correctly, or that a better source was not missed.

Which tools can run retrieval evaluations?

Options include Azure AI Foundry evaluators, OpenAI Datasets or eval workflows, LangSmith, Braintrust, Humanloop, Ragas, TruLens, DeepEval, custom notebooks and CI tests built with pytest or similar frameworks.

What should we measure besides groundedness?

Measure whether the correct source appears in the top results, whether prohibited sources are excluded, whether citations match the answer, whether the answer refuses when knowledge is missing, and whether user permissions are respected.

How do retrieval tests help with UK data protection?

They let you test whether assistants respect access boundaries, avoid exposing personal data from the wrong source, refuse sensitive requests and use approved knowledge only. That supports data protection by design and accountable deployment.

Can we build the test set from real staff and customer questions?

Yes, but sanitise the questions first, remove unnecessary personal data, and document the source and review process. For sensitive domains, use typed placeholders or expert-written equivalents.

How often should retrieval tests be rerun?

Run a small smoke set nightly or weekly, and run the full set before release, after major document changes, after chunking or embedding changes, after model upgrades and after permission model changes.