A correct answer and a confident lie look identical. Here's how I made RAG tell them apart.

Your retrieval demo is beautiful. You dropped a thousand PDFs into a vector database, wired up an embedding model, and now it answers questions with uncanny confidence. You ship it. Everyone's impressed.

Then the corpus grows. Ten thousand documents. A hundred thousand. A million. Fifteen million.

Nothing crashes. No error in the logs. The latency is fine. The answers still arrive, still confident, still well-written.

They're just wrong more often. And you can't tell which ones.

That's the trap of RAG at scale: it doesn't fail loudly, it fails plausibly. The bigger your corpus, the more ways the system has to retrieve a chunk that looks right and build a fluent, cited-sounding answer on top of it. The failure mode isn't a stack trace. It's a sentence that sounds true.

Here's the part people don't want to hear: this isn't hypothetical, and it isn't solved by a bigger model. In 2025, Stanford's RegLab tested commercial, enterprise-grade legal RAG tools — the paid products built specifically to be trustworthy — and found they still hallucinated in roughly 17–33% of cases (Magesh et al., Journal of Empirical Legal Studies, 2025). One in three to one in six answers, wrong, in a domain where wrong gets people sued. (That number is specific to those legal products and the tested tasks — read it as how hard this is, not a universal failure rate every RAG system will hit.)

RAG reduces hallucination. It does not eliminate it. And at 15M+ documents, the margin gets thinner unless you change the architecture — not the model.

This is the pipeline I'd build for 15 million documents and near-zero confident hallucination. The trick isn't a magic retriever. It's four gates that each catch a different lie, and a system that would genuinely rather abstain than make something up.

Fluency is not evidence. The only thing that separates a true answer from a confident lie is whether each claim traces to a source that actually supports it.

What "near-zero" honestly means. It's a design objective, not a guarantee. Not zero errors — the goal is that zero confident, unsupported claims reach the user, because the system converts what would have been a hallucination into an honest "I couldn't verify this." Some will still slip past an imperfect verifier; the aim is to make that rare, bounded, and observable — not to promise it never happens. That reframe is the whole article.

First, why does RAG hallucinate at all?

If you only remember one thing, make it this: a RAG system hallucinates for exactly three reasons, and scaling the corpus makes all three worse.

The three ways an evidence-grounded answer fails — a similar chunk wins, available evidence goes unused, or missing evidence becomes a guess.

1. It retrieved the wrong evidence.
The answer exists in your corpus, but the retriever pulled a chunk that's similar to the question instead of one that answers it. At 1,000 docs there might be one chunk about "Q2 revenue." At 15 million there are forty thousand chunks that mention revenue, hundreds of near-identical boilerplate paragraphs, and a dozen that look perfect and say the wrong number. More documents means more plausible decoys.

2. It retrieved the right evidence and ignored it.
The correct chunk is sitting in the context window, and the model answers from its own parametric memory anyway — or the chunk got buried in the middle of a long context and the model's attention skated right past it. This is the famous lost in the middle effect (Liu et al., 2023): models reliably use information at the start and end of their context and quietly drop what's in the middle. Stuff 50 chunks in and the answer in position 26 may as well not be there.

3. It retrieved nothing useful and answered anyway.
This is the real hallucination. There is no supporting evidence in the top-k, and instead of saying so, the model does what it was trained to do: produce a fluent, helpful-sounding completion. It fills the gap. In a 15M corpus, plenty of legitimate questions simply have no good answer in the data — and a system with no "I don't know" has no choice but to invent one.

Every technique below exists to kill one of these three. If you can't name which failure a component targets, you're cargo-culting complexity.

The principle: you can't prompt your way out

Before the pipeline, the mindset. "Please only answer using the provided context and don't make things up" is a wish, not a control. It helps a little and fails exactly when you need it most — when the context is wrong, incomplete, or buried.

Near-zero hallucination is a pipeline property, not a prompt. Four gates, in order:

The whole pipeline: corpus prep → index → Retrieve · Constrain · Verify · Abstain, with the metrics you evaluate it on. The goal is to reduce confident, unsupported claims — not to promise zero errors.

1. Retrieve — get the right evidence into the context, and be able to prove it's there.
2. Constrain — force the answer to cite that evidence, claim by claim. No citation, no sentence.
3. Verify — independently check that every claim is actually supported by what it cites.
4. Abstain — when support is weak, say so, in calibrated language, instead of guessing.

Retrieve fixes failure #1. Constrain fixes #2. Verify and abstain, together, fix #3 — the dangerous one. Let's build each.

Gate 1 — Retrieve the right evidence (and prove it)

Retrieval is where most of your accuracy is won or lost, and it's the stage that breaks hardest at 15M. Four moves matter.

Deduplicate before you embed — or your index lies to you

At scale, your corpus is full of near-duplicates: the same disclaimer on 200,000 pages, boilerplate repeated across contract versions, articles syndicated to six sites. If you embed all of them, three bad things happen: your index bloats (and so does your bill), retrieval returns ten copies of the same chunk and crowds out diversity, and relevance scores get distorted by frequency.

The standard fix is MinHash + LSH (locality-sensitive hashing): hash each document into a signature, bucket similar signatures, and collapse near-duplicates before they ever reach the embedder. It's cheap, it's old, it's boringly effective, and at 15M docs it's not optional — it's the difference between a clean index and a hall of mirrors.

# Dedup before embedding — MinHash + LSH (datasketch). Streams; never holds 15M in RAM. from datasketch import MinHash, MinHashLSH def signature(text, num_perm=128): m = MinHash(num_perm=num_perm) for token in set(text.lower().split()): # or k-gram shingles for finer dedup  m.update(token.encode()) return m def dedup(docs, threshold=0.9): lsh, kept = MinHashLSH(threshold=threshold, num_perm=128), {} for doc_id, text in docs: sig = signature(text) if lsh.query(sig): # a near-duplicate is already kept  continue lsh.insert(doc_id, sig) kept[doc_id] = text return kept # near-dups collapsed pre-embedding 

Chunk with structure, then give every chunk its context back

Naive chunking cuts your documents into 500-token windows and destroys meaning at the seams. A chunk that reads "Revenue grew 3% quarter over quarter" is useless on its own — whose revenue? which quarter? The embedding has no idea, because the sentence was torn out of its document.

The fix that actually moves numbers is contextual retrieval: before embedding each chunk, prepend a short, machine-generated sentence that situates it in its parent document. "This chunk is from ACME Corp's Q2 2026 10-Q, in the section on cloud revenue: Revenue grew 3%..." Now the vector and the keyword index both carry the context.

# Prepend a one-line situating context, and KEEP the source id for later verification def contextualize(chunk, doc_meta, llm): prefix = llm( f"In one sentence, situate this passage in its document " f"(title: {doc_meta['title']}, section: {doc_meta['section']}).\n\n{chunk.text}" ) return { "id": chunk.id, # provenance → rides into the citation  "doc_id": doc_meta["doc_id"], "embed_text": f"{prefix}\n\n{chunk.text}", # what you embed AND BM25-index  "source_text": chunk.text, # what Gate 3 later checks the claim against  } # tip: prompt-cache the full document so the per-chunk context call stays cheap. 

Context makes a chunk retrievable; the preserved source ID and document provenance make it verifiable later.

This is the one architecture choice with clean, published proof. Anthropic measured the share of queries where the right chunk wasn't in the top 20 (Contextual Retrieval, Sept 2024):

A preprocessing step that turns a 5.7% miss rate into 1.9% is one of the best returns in the whole stack. Honest caveat: those are the best-performing embedding configuration in Anthropic's test, not an average across every model — treat it as "defaults are leaving a lot on the table," not "1.9% guaranteed on your data."

Hybrid search: meaning and the literal token

Dense vector search is brilliant at meaning — it knows "cancel my plan" ≈ "terminate subscription." It's terrible at exact strings: error code E-4021, a rare surname, a part number, a statute citation. Those are the queries where a semantic embedding smooths away the one token that matters.

So run both. Dense (vector) search for meaning, sparse (BM25) search for the literal token, and fuse the two rankings with Reciprocal Rank Fusion (RRF) — a dead-simple formula that scores each document by summing 1 / (k + rank) across the lists it appears in:

# RRF: no tuning, no training, just robust rank fusion def rrf(dense_ranking, sparse_ranking, k=60): scores = {} for ranking in (dense_ranking, sparse_ranking): for rank, doc_id in enumerate(ranking): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank) return sorted(scores, key=scores.get, reverse=True) 

Dense catches the paraphrase. BM25 catches the exact code. RRF lets a document that both rank reasonably well float to the top. This single change kills a huge share of "retrieved the wrong evidence" failures at scale.

def hybrid_retrieve(query, dense_index, bm25_index, n=150): dense = dense_index.search(embed(query), k=n) # semantic similarity  sparse = bm25_index.search(query, k=n) # exact tokens / codes / names  fused = rrf([d.id for d in dense], [s.id for s in sparse]) return [load_chunk(i) for i in fused[:n]] # ~150 candidates, over-fetched 

Over-fetch, then rerank down

Your first-pass retrieval is tuned for speed over a 15M index, so it's approximate — it casts a wide, slightly sloppy net. So cast it wide on purpose: pull ~150 candidates, then hand them to a cross-encoder reranker that reads the query and each chunk together and re-scores them, keeping the best ~20.

The reranker is slow and precise where retrieval is fast and fuzzy. Running it on 150 candidates (not 15 million) is affordable, and it's the step that earns the last chunk of that 5.7%→1.9% drop in the table above.

from sentence_transformers import CrossEncoder reranker = CrossEncoder("BAAI/bge-reranker-v2-m3") # reads (query, chunk) together  def rerank(query, candidates, keep=20): scores = reranker.predict([(query, c["source_text"]) for c in candidates]) ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True) return [c for c, _ in ranked[:keep]] # 150 -> 20 (tune on your eval set) 

Retrieve broadly, rerank narrowly. The candidate counts are illustrative — tune them on your own labeled set. Reranking surfaces promising evidence; it does not prove a claim is supported.

The 15M scale note nobody mentions. Fifteen million vectors at 1024 float32 dimensions is ~61 GB of raw vectors — before the index overhead. You will not hold that in RAM naively. Two levers: an on-disk ANN index (HNSW or IVF-PQ) so the index lives on SSD, and embedding quantization (float32 → int8, or even binary) which cuts memory 4–32× for a small, measurable recall hit you can budget for. Scale is a memory-layout problem as much as a modeling one.

And keep three kinds of evidence separate, because they get conflated constantly: an index benchmark (ANN latency/recall on 15M synthetic vectors) is not the same as end-to-end retrieval recall on your real, messy documents, which is not the same as extrapolated capacity. A synthetic index proves the search scales; it says nothing about whether the right chunk actually comes back on your corpus.

Which vector store survives 15 million vectors?

"Use a vector database" is where most tutorials wave their hands and name whatever they happened to use. At 15M the choice is a real engineering decision — on-disk vs in-memory, quantization support, and whether sparse (BM25) lives in the same system or a second one you have to keep in sync. My honest read of the landscape:

There's no single right answer — the point is that at 15M, "dense + sparse in one system, on disk, with quantization" is the requirement, and you pick the store that gives you those three without a second sync pipeline. I reach for Qdrant self-hosted; a Postgres shop should try pgvector before adding infrastructure.

Gate 2 — Constrain generation: cite, or don't speak

You've got 20 excellent chunks in context. Failure #2 says the model might still ignore them. Gate 2 makes ignoring them structurally hard.

Make citations mandatory, per claim

Don't ask the model to "use the context." Require it to attach a citation — a chunk ID — to every factual sentence it writes, and instruct it that any claim it cannot cite must be dropped or flagged as uncertain. Force the output into a shape like:

{ "answer": [ {"claim": "ACME's Q2 2026 cloud revenue grew 3% QoQ.", "cites": ["chunk_8842"]}, {"claim": "Growth was driven by enterprise seat expansion.", "cites": ["chunk_8847","chunk_8851"]} ], "unsupported": [] } 

This does two things. It drags the model's attention onto the evidence (it has to point at a chunk for each sentence), and it gives Gate 3 something concrete to check. A claim with no citation is a claim the model made up — and now you can see it, instead of it hiding inside a fluent paragraph.

CITED_SYSTEM = """Answer ONLY from the numbered context below. Return JSON: {"answer": [{"claim": "...", "cites": ["<chunk_id>"]}], "unsupported": []} Every claim must cite a chunk id that FULLY states it. If you cannot support a sentence from the context, do not write it — name the topic in "unsupported" instead. Never cite a chunk that does not state the claim.""" def generate_cited(question, chunks, llm): context = "\n\n".join(f"[{c['id']}] {c['source_text']}" for c in chunks) raw = llm(system=CITED_SYSTEM, user=f"{context}\n\nQuestion: {question}") return json.loads(raw) # {"answer": [{claim, cites}], "unsupported": [...]} 

Beat "lost in the middle" with order and brevity

Remember failure #2: the right chunk gets buried mid-context. Two defenses:

• Keep k small. This is why you reranked 150 down to 20 (and often down to 5–8 for the final prompt). Fewer, better chunks means less middle to get lost in.
• Order for attention. Place the highest-scoring chunks at the start and end of the context, not the middle, because that's where models actually read (Liu et al., 2023).

A qualitative pattern, not a law — the effect varies by model, task, and context length. Treat evidence ordering as something to test on your target model, not a fixed rule.

Honest nuance: a 2025 replication found the lost-in-the-middle effect isn't universal — it weakens as inputs approach the model's full context window, and long-context studies don't always reproduce it (arXiv:2508.07479). So treat ordering as cheap insurance, not a law. The free win is still "fewer, better chunks."

Route and decompose before you retrieve

One retrieval pass can't serve every question. "What's ACME's refund policy?" is a lookup. "How did ACME's margins change across the last three quarters and why?" is a multi-hop question that no single chunk answers.

So classify the question first, then:

• Simple lookup → one hybrid retrieval, answer.
• Multi-hop / comparative → decompose into sub-questions, retrieve evidence for each separately, then synthesize. Each sub-answer carries its own citations.

Decomposition directly prevents a nasty failure: answering a three-part question from one chunk that happened to rank first. Give each part its own evidence or don't claim it.

Gate 3 — Verify: grade the claims, not the vibes

Here's the uncomfortable truth about Gate 2: a model told to cite will sometimes cite a chunk that doesn't actually support the claim. It'll attach chunk_8842 to a sentence that chunk never said. Citations are necessary, not sufficient. You have to check them.

Citations are necessary, not sufficient — a claim can cite a real passage that doesn't actually entail it. Check each claim against what it points to; keep the supported ones, drop or abstain on the rest.

The move is claim-level faithfulness verification:

1. Decompose the generated answer into atomic claims (one assertion each).
2. For each claim, pull the chunks it cited.
3. Ask a verifier: is this claim entailed by these chunks? (an NLI model, or an LLM-as-judge prompted narrowly for entailment, not helpfulness).
4. A claim that fails entailment is unsupported — strip it, or send the whole answer to the abstain gate.

# Check each claim against ONLY the chunks it cited — entailment, not helpfulness NLI = ("Does the EVIDENCE fully entail the CLAIM? " "Reply SUPPORTED, NOT_ENTAILED, or CONTRADICTED.\nCLAIM: {c}\nEVIDENCE: {e}") def verify(answer, chunks_by_id, judge): kept, rejected = [], [] for item in answer["answer"]: evidence = "\n".join(chunks_by_id[cid]["source_text"] for cid in item["cites"]) verdict = judge(NLI.format(c=item["claim"], e=evidence)).strip() (kept if verdict == "SUPPORTED" else rejected).append(item) return kept, rejected # rejected → stripped, or trigger abstention (Gate 4) 

Why atomic and claim-level? Because answer-level scores hide per-claim lies. An answer with nine supported claims and one fabricated one scores 0.9 on a naive groundedness metric and ships the fabrication. Grade each claim and the fabrication has nowhere to hide. This is exactly the direction the measurement research has moved — the RAGAS framework formalized faithfulness as a first-class metric (Es et al., 2023), and 2025 work like Vectara's FaithJudge builds leaderboards specifically for detecting unsupported content (arXiv:2505.04847).

Keep yourself honest: the verifier is itself a model, and FaithJudge's own finding is that detecting unsupported content is still hard — judges miss things. Verification reduces hallucination that reaches the user; it doesn't zero it. Anyone selling you "100% faithful" is selling you the same overconfidence you're trying to remove.

This is also why the Self-RAG and Corrective-RAG lines of work exist — they bake the check into the loop. Self-RAG (Asai et al., 2023) trains the model to emit reflection tokens that judge relevance and support as it generates. Corrective RAG / CRAG (Yan et al., 2024) puts a lightweight evaluator outside the model that scores retrieved docs and, on a bad score, triggers a fallback (like a web search) instead of plowing ahead. Both are the same instinct as Gate 3: don't trust the first draft of the evidence.

Gate 4 — Abstain: the feature everyone treats as a bug

This is the gate that actually gets you to near-zero, and it's the one most teams refuse to build because it feels like admitting defeat.

When retrieval confidence is low, or Gate 3 strips too many claims, or the remaining support is thin — the system says so. "I don't have enough reliable information to answer that." Not a guess dressed as an answer. An honest, calibrated abstention.

The right way to think about this is a 2×2 that keeps two different questions apart: is there sufficient evidence? (answerable vs unanswerable) and did the system answer? (answer vs abstain). Those are not the same axis, and conflating them is how people fool themselves.

Separate "is there evidence?" from "did it answer?" The dangerous cell is unanswerable × answered — a confident unsupported answer. Abstaining when evidence exists is only a missed opportunity; abstaining when it doesn't is a safe refusal. Measure coverage and risk together.

The naive version is a similarity threshold ("if top chunk score < 0.7, abstain"). It works a little and is impossible to tune across question types. The rigorous version is calibrated, risk-controlled abstention:

• Conformal abstention (Abbasi-Yadkori et al., NeurIPS 2024) samples several answers, measures how much the model agrees with itself, and uses conformal prediction to control the error rate. But read the guarantee precisely: it holds under its sampling assumptions and a defined calibration set, and it bounds the rate of the specific quantity you calibrated on — it is not a switch that magically pins "any real-world hallucination rate" to a number you type in. It's the rigorous target to aim at, with conditions, not a free guarantee.
• The generalization everyone is reaching for is to fuse several uncertainty signals — sequence likelihood, self-consistency across samples, retrieval score, and verifier feedback — into one calibrated probability of correctness, and refuse when it can't clear the budget. This is an active, fast-moving research area in 2025–26, so treat specific framework names as preprints-in-flux and lean on the conformal result above as the load-bearing proof.

The engineering version you can ship today: combine (a) retrieval score, (b) fraction of claims that survived Gate 3, and (c) answer self-consistency across 2–3 samples, into a single confidence, and abstain below a threshold you calibrate on a labeled held-out set so the threshold means something. Be honest about what this is: calibrated thresholding is what you actually ship; the conformal machinery above is the rigorous upgrade once you have the calibration data and are willing to defend its assumptions. Don't call a tuned threshold a "guarantee."

def decide(kept, rejected, retrieval_score, samples, tau): # tau is CALIBRATED on a labeled held-out set — not guessed  support_ratio = len(kept) / max(1, len(kept) + len(rejected)) consistency = agreement(samples) # do 2-3 resampled answers agree?  confidence = 0.4*retrieval_score +...