Your RAG chatbot confidently answers questions about docs that don't exist in production. How to fix.

You built a RAG system. It scores 0.91 faithfulness on your eval set. You deploy it. A user asks about a return policy that isn't in your knowledge base. The chatbot answers — confidently, fluently, and completely wrong.

This isn't a prompt engineering problem. It's a retrieval architecture bug that almost every RAG tutorial skips.

What actually happens
When a user query has no good match in your vector store, cosine similarity doesn't return "nothing found." It returns the least-bad chunk — whatever is closest in embedding space, even if it's completely irrelevant. Your reranker then picks the best of these bad candidates. The LLM receives context that's wrong-but-plausible and does exactly what it's trained to do: generate a fluent, coherent answer. No hallucination flag. No uncertainty hedge. Just wrong.

Your eval set never caught this because you built it from documents in your knowledge base. Every golden query had a valid retrieval match. The failure mode only surfaces on out-of-distribution queries — which is most of production traffic.

The fix: similarity thresholding before generation

results = vectorstore.similarity_search_with_score(query, k=5) top_score = results[0][1] # cosine similarity of best match  if top_score < 0.72: # calibrate this on your domain  return "I couldn't find information about this in our knowledge base." context = [doc for doc, score in results if score >= 0.65] 

The threshold isn't universal — 0.72 is a starting point. Calibrate it by running your golden eval set and finding the score below which faithfulness drops sharply. For most general-purpose corpora it's between 0.65–0.78.

Why this matters for interviews too
This exact scenario — "your RAG system hallucinates on queries about content not in the knowledge base, how do you fix it?" — shows up verbatim in AI engineering interview loops at companies building production RAG systems. The answer they're looking for isn't "better prompting." It's similarity thresholding, out-of-distribution detection, and a graceful fallback path.

If you're prepping for AI engineering roles, the full RAG system design question set (including this one) is covered in depth on KnowEaseAI (https://knoweaseai.com/blog/rag-system-design-interview-questions)
— with the production failure modes that actually show up in interviews, not just the happy path.