Monitor Retrieval Quality After Your Knowledge Assistant Launches

Monitor retrieval quality by checking whether current, authorized evidence reaches the answer model—not just whether users like its responses. Combine a fixed set of reviewed questions with ongoing production sampling, measure evidence coverage and ranking separately from answer quality, and give each failure category an owner and response rule. This lets you distinguish a search problem from missing documentation, context assembly errors or a model that ignores good evidence.

Measure the evidence path, not just the answer

Retrieval-augmented generation (RAG) supplies a language model with information found in a knowledge collection. Retrieval quality concerns what that search returns; answer quality concerns what the model does with it. A fluent answer can conceal weak retrieval, while a poor answer can follow a perfectly good search.

Microsoft’s RAG evaluation guidance recommends evaluating individual pipeline stages alongside the end-to-end experience. Apply that distinction after launch by recording three checkpoints:

• Search candidates: Which document passages were found, and in what order?
• Selected context: Which passages actually reached the model after filtering, reordering and length limits?
• Answer evidence: Which passages support the claims made in the response?

Record document and passage identifiers, source versions, search configuration, permission-filter outcomes and timing under a shared request identifier. Preserve enough information to reproduce a failure without logging entire sensitive documents by default. Restrict access to review records and set retention limits.

A relevant passage found during search but omitted from the model’s input is a context-selection failure, not necessarily a search failure. That distinction determines what to fix.

Maintain two complementary review samples

Use a stable regression set—a collection of reviewed cases rerun to detect deterioration—for comparisons across releases. Each case should identify the question, applicable user access, expected evidence and whether an answer is possible from the available collection. Include questions requiring multiple passages, ambiguous wording and deliberately unanswerable requests.

Alongside it, review a fresh production sample each week. Start with random requests, then add targeted samples of empty results, user corrections, new topics and high-consequence questions. Report targeted samples separately: deliberately collecting failures will otherwise make the overall failure rate misleading.

Break results down by language, department, source collection and question type. An aggregate score can stay stable while a small team’s searches deteriorate. Show sample counts, and avoid declaring trends from a handful of cases.

Ask reviewers to label the failure location, not merely mark an answer wrong. Was the evidence absent from the collection, missing from the index, ranked too low, removed before generation or ignored by the model? Our guide to building a representative evaluation set explains how to assemble reviewable cases from business work.

Use a small scorecard with explicit denominators

The following are recommended operating measures, not universal targets:

• Evidence recall at k: The fraction of reviewed, required evidence units present among the first k results. Define evidence units as distinct facts or passages, so duplicate chunks do not inflate the score.
• Complete-evidence rate: The percentage of answerable questions for which all required evidence reaches the model. This exposes searches that find the general policy but miss an exception.
• First relevant rank: The position of the first useful result. Track this alongside completeness; finding one useful passage early does not guarantee enough evidence.
• Unsupported-answer rate: The percentage of reviewed answers containing material claims not supported by their supplied evidence.
• Correct abstention rate: Among reviewed unanswerable cases, the percentage where the assistant declines to invent an answer or requests clarification.

Track retrieval latency and cost alongside quality. Do not treat similarity scores as calibrated probabilities of correctness, or assume their values remain comparable after changing search models or settings.

If an automated model scores relevance, compare its judgments with human-reviewed examples and audit disagreements. Automated scores should help prioritize review, not silently become unquestionable labels.

Worked example: a missing policy exception

Illustrative example: An internal purchasing assistant is evaluated on 100 answerable questions. For one question about equipment purchases, reviewers identify two required evidence units: the standard approval limit and a regional exception.

The first five search results contain the standard limit but not the exception. Evidence recall at five is therefore 1/2, or 50%, for that question. Its complete-evidence result is a failure even if the assistant confidently cites the standard policy.

Suppose complete evidence reaches the model for 86 of the 100 questions, compared with 94 on the same reviewed set before a search change. An illustrative release gate requires at least 92%, with no new failures on designated high-consequence cases. The change fails that gate.

Inspect the missing exceptions before increasing the number of retrieved passages. If they were never indexed, a larger result set cannot recover them. If they were found but discarded, examine context selection instead.

Turn failures into owned actions

Assign ingestion failures to the data-pipeline owner, ranking failures to the retrieval owner and missing or conflicting policies to the content owner. Route insufficient evidence to clarification or human review rather than an unsupported answer. Compare proposed fixes against the same reviewed cases before a staged release; roll back search configuration only when the prior version remains safe and current.

Handle suspicious retrieved instructions as a separate security exception. OWASP’s prompt-injection guidance identifies malicious retrieved content as an attack path and cautions that screening is not an enforcement boundary. Quarantine suspect content for review, preserve minimal incident metadata and keep authorization outside the model. A highly relevant search result can still be unsafe.

AIoverflow supports evaluation, deployment and ongoing improvement of applied AI workflows. To define a retrieval scorecard, review cadence and recovery responsibilities for your assistant, contact our team.