Staging “looks fine” is how RAG regressions ship.

Someone updates the corpus, bumps an embedding model, or tweaks chunk size. Answers still sound confident. A week later support hears “it cites the wrong policy page.”

Put metrics on the pull request

You need a frozen question set with expected doc ids or answer keys. Run it in CI when retrieval code or corpus fingerprints change.

Useful offline signals:

  • Recall@k — did the right chunk appear in the top k?
  • MRR — how high was the first relevant hit?
  • Faithfulness / citation hit — does the answer point at retrieved ids?
  • Empty retrieval rate — sudden spikes mean a broken index build

You do not need a perfect LLM judge on day one. Start with retrieval metrics. Add answer judges once retrieval is stable.

# evals/rag_ci.py from statistics import mean def recall_at_k(hits, relevant, k=5): top = {h.doc_id for h in hits[:k]} return len(top & set(relevant)) / max(1, len(relevant)) def evaluate(cases, retrieve): scores = [] for case in cases: hits = retrieve(case["question"], k=5) scores.append(recall_at_k(hits, case["relevant_ids"], k=5)) return mean(scores) def main(): score = evaluate(load_cases("evals/rag_cases.json"), build_retriever()) threshold = 0.75 print(f"recall@5={score:.3f} threshold={threshold}") if score < threshold: raise SystemExit(1) if __name__ == "__main__": main()

When CI should run

Trigger on:

  • changes under rag/, embeddings/, chunking/
  • corpus fingerprint file changes
  • scheduled nightly for full rebuild drift

Skip on unrelated frontend PRs so people do not learn to hate the job.

Honest trade-off

A fixed case set goes stale. Budget an hour every two weeks to add failures from production. If you never refresh cases, CI protects an old product.

LLM-as-judge is convenient and noisy. Keep it advisory until you measure agreement with humans on a sample.

Wire it like any other test 

- run: python -m evals.rag_ci - uses: actions/upload-artifact@v4 if: failure() with: name: rag-eval path: evals/last-run.json

Make the check required when retrieval is on the critical path for customers. If RAG is still experimental, keep the job visible but non-blocking, and write a date to flip it to required.

Monday plan

  1. Export 30 real questions from support or search logs.
  2. Label relevant doc ids by hand (painful once, valuable forever).
  3. Fail the PR under a recall threshold you can defend.

Do you know last week’s recall@5, or only that the chatbot “felt better”?