The easiest RAG demonstration starts with a question whose answer is clearly present in a document. The retriever finds the relevant passage, the model turns it into a friendly response, and the result looks convincing.
I find a different question more useful during development: what happens when the knowledge base does not contain the answer? That case exposes whether the application understands the difference between finding something similar and finding enough evidence.
A nearby passage can still be the wrong evidence
Suppose a store has a general returns policy, but no information about whether a particular made-to-order item can be returned. A retriever may find the general policy with a strong similarity score. The text is relevant to the topic without necessarily supporting the specific answer.
This is why I do not treat retrieval rank as permission to answer. The application needs to consider what the question asks, which facts the source actually establishes, and what remains unknown.
The original RAG paper by Lewis and colleagues describes combining generation with retrieved external memory. A customer-facing implementation still needs product-specific decisions about when that retrieved material is sufficient.
Build an evaluation set before optimizing the pipeline
I start with a small collection of representative questions and label the expected behavior. The label is more than a preferred sentence. It includes whether the system should answer, ask a clarifying question, or hand the issue to a person.
For an answerable case, I record the supporting source and the claims that are allowed. For an unanswerable case, I record what is missing. That distinction helps reviewers agree on why a response passes or fails.
- A question with a direct answer in one source.
- A question using a synonym rather than the source wording.
- A question about a product absent from the catalog.
- A question requiring a policy exception the documents do not establish.
- A question with an ambiguous reference such as “the larger one.”
- A question supported only by an outdated document.
I include difficult but plausible wording. Artificial nonsense is easy to reject and says little about realistic unsupported questions.
Separate retrieval errors from answer errors
If the correct source was never retrieved, changing the response prompt is unlikely to repair the underlying problem. If the source was retrieved but the answer invented a condition, the retrieval stage may be working correctly.
I therefore inspect candidate selection, ranking, evidence sufficiency, and final generation separately. A single overall accuracy number can hide a system that retrieves well but frequently overstates its evidence.
Thresholds should be calibrated against the actual corpus and query distribution. A similarity score is not a universal confidence percentage. The appropriate decision boundary may also differ between a general descriptive question and a request about a price or warranty.
Make uncertainty useful to the customer
A good abstention explains the missing information and offers a next step. If the product variant is unclear, ask for the variant. If the policy is absent, route the question to a person with the relevant context.
I avoid vague statements such as “I cannot help with that” when the system can say exactly what it needs. The goal is to preserve the customer journey while refusing to manufacture a fact.
During evaluation, I track unsupported answers separately from unnecessary abstentions. Reducing one by making every response unhelpful is not a useful improvement. The system needs to answer when evidence is adequate and recognize when it is not.
A negative test set gives that boundary a concrete shape. It turns “the assistant should be careful” into examples the engineering team can run after every meaningful retrieval or prompt change.
Updated 25 September 2026.