Teach a document assistant to recognize missing, stale or conflicting evidence and give a useful response without inventing facts.
Originally published on Stack & Craft.
What you will learn: Distinguish answerable questions from evidence gaps, design useful abstentions, and test both overconfidence and unnecessary refusals.
Prerequisites: Start with the Building and Evaluating a RAG System guide. Worked exercises use Python’s standard library and need no paid model.
What does abstaining mean?
Abstaining means declining to make an unsupported claim when the available evidence is insufficient. A useful assistant can explain the gap, show what is known, and suggest a next check. It should not imply that the information is absent from the entire world—only that the authorized sources supplied do not support an answer.
Missing fact
The source never states the requested information.
A refund policy gives eligibility but no bank settlement time.
Conflicting evidence
Applicable sources disagree and the system cannot establish which is authoritative.
Two policies with unclear dates give different deadlines.
Stale evidence
The only available source is too old for a current question.
A request asks for today’s price, but the source is an archived price list.
Distinguish a search miss from a knowledge gap
If retrieval returns nothing useful, the fact might still exist elsewhere in the approved collection. The system should explain that it did not find supporting evidence, rather than claiming no policy exists. Log the case for authorized investigation.
Say exactly what is missing
Better: “The supplied policy explains how to request a refund, but it does not state the bank settlement time.” Avoid inventing “three days” or making a sweeping claim that the company has no settlement policy.
A threshold is only one signal
Similarity scores and lexical overlap are not calibrated probabilities of answerability. A highly relevant passage can still omit the requested fact. Low scores can also occur for answerable paraphrases.
An evidence-aware response path
- Respect access — Search only the material the current user is authorized to use.
- Check applicability — Inspect source identity, version, audience and whether the question’s conditions match.
- Check coverage — Does the available passage support every important part of the requested answer?
- Answer or explain the gap — Provide supported facts with references, or state precisely what evidence is missing.
- Offer a safe next step — Suggest checking a named document owner or obtaining the current policy, without exposing restricted source details.
Write useful partial answers
A question can be partly answerable. Make the boundary visible: separate supported facts from missing facts instead of refusing everything or filling gaps with guesses.
| Situation | What it means | Next check |
|---|---|---|
| Eligibility known; settlement time absent | Answer eligibility with its source; explain the missing settlement detail. | Does the wording clearly separate the two? |
| Old and new policy conflict | Use documented authority/version rules if they resolve it; otherwise explain the conflict. | Are the versions and applicability explicit? |
| Restricted source would contain the fact | Do not reveal the restricted fact or private source details. | Was authorization enforced before context assembly? |
| Source does support the answer | An unnecessary refusal is also a quality failure. | Test known-answer cases alongside missing facts. |
See why relevant text can still fail
Run exercises/missing_evidence_demo.py. A toy source mentions refunds and request steps but omits settlement time. The retriever can still return it for a related question. The exercise prints the passage and verifies that no settlement deadline is present; it does not pretend that keyword checks solve general answerability.
Expected interpretation
A non-empty search result is not permission to invent the missing deadline. In a production application, use clear grounding instructions, source checks, evaluation and appropriate human review. No single prompt guarantees abstention.
Measure both kinds of mistake
Track unsupported answers on unanswerable cases and unnecessary abstentions on answerable cases. A system that always refuses may avoid some unsupported claims while being useless.
Make the product boundary clear
Tell users what collection the assistant covers and how to reach a human when needed. Keep sensitive logs protected and use genuine feedback to improve source coverage and retrieval.
A practical workflow
- Label whether each question is answerable from the authorized collection.
- Check source applicability and evidence coverage.
- Write a supported answer, a partial answer or a precise abstention.
- Evaluate unsupported claims and unnecessary refusals separately.
Practice
Write responses for a known refund deadline, an unknown bank settlement time and two conflicting undated policies. Include a useful next step for each gap, and name one case where refusing would be unnecessary.
Check your understanding
Does a non-empty retrieval result prove the question is answerable?
No. The passage may be related to the subject while omitting the requested fact.
Should an assistant say the fact does not exist anywhere?
No. State the limitation of the authorized sources available to this request.
Is always refusing a good strategy?
No. Track unnecessary refusals on known-answer cases alongside unsupported claims on missing-evidence cases.
Sources
- Original RAG research — Lewis et al.: Research background on combining retrieval and generation. Examples and exercises in this guide are original teaching material.
Continue learning
Read the original guide on Stack & Craft for the illustrated version and related lessons.
The Local RAG Document Assistant includes the companion exercises.
