Short answer: faithfulness in RAG means every claim the answer makes is supported by the retrieved context — not by what the model memorized. Three metrics hold up: a RAGAS faithfulness score for eval suites, a deterministic grounding check for CI, and a narrow LLM-as-judge for the edge cases neither catches.
The confusion I see most often: people conflate correctness and faithfulness. A correct answer can be unfaithful — the model happened to state the right fact from memory, not from the context you gave it. That's a different failure mode than getting it wrong, and it matters a lot when the retrieved context changes.
What does faithfulness actually measure — and what it doesn't?
Faithfulness answers: did the model only use what I gave it?
It says nothing about whether the retrieved context was right in the first place (that's context precision and recall), and nothing about whether the answer addressed the question well (that's answer relevancy). Three separate dimensions, easy to conflate.
The concrete definition: a faithful answer contains zero claims that require knowledge beyond the retrieved chunks to verify. If your context says "the return window is 30 days" and the model outputs "returns are accepted within 30 days" — faithful. If it outputs "returns are accepted within 30 days, free of charge" and your context says nothing about fees — the second half is unfaithful. It might even be correct. Doesn't matter.
How does RAGAS measure faithfulness?
RAGAS faithfulness works in two steps: decompose, then check.
Step 1 — decompose the answer into atomic statements. An LLM splits the answer into the smallest verifiable claims. "Returns are free within 30 days of purchase" becomes two statements: "returns are free" and "the return window is 30 days."
Step 2 — check each statement against the context. Another LLM call checks whether each statement is entailed by the retrieved chunks. The score is supported / total.
from ragas import evaluate from ragas.metrics import faithfulness from datasets import Dataset # Your data: question, answer your RAG produced, and the contexts it retrieved data = { "question": ["What is the return policy?"], "answer": ["You can return items within 30 days for a full refund, free of charge."], "contexts": [["Purchases may be returned within 30 days. No restocking fee applies."]], } dataset = Dataset.from_dict(data) result = evaluate(dataset, metrics=[faithfulness]) print(result["faithfulness"]) # e.g., 0.5 — "free of charge" wasn't in the context A score of 0.5 here means one of the two decomposed claims wasn't supported. The model said "free of charge" — that's consistent with "no restocking fee," but RAGAS decomposed them into separate verifiable statements and the LLM-as-judge for that specific claim didn't find the exact phrase. That's a known limitation of the decomposition step.
The practical threshold: score below 0.6 on a golden dataset question → manual review. Below 0.4 → hard failure. Above 0.8 → trust it.
How do you implement your own faithfulness scorer?
You don't always want RAGAS's decomposition step. Sometimes you want a scorer that maps exactly to your rubric. Here's the core:
FAITHFULNESS_PROMPT = """\ You are a strict factual auditor. Answer ONLY with a JSON object. Retrieved context: {context} Generated answer: {answer} For each factual claim in the answer, decide: "supported": the claim is directly entailed by the context "unsupported": the claim cannot be verified from the context alone "contradicted": the claim conflicts with the context Respond with: {{"claims": [{{"text": "...", "verdict": "supported|unsupported|contradicted"}}], "faithfulness_score": 0.0-1.0}} Score = supported_claims / total_claims. Contradicted claims count as unsupported.""" def faithfulness_score(answer: str, context_chunks: list[str]) -> dict: context = "\n---\n".join(context_chunks) resp = client.messages.create( model="claude-haiku-4-5-20251001", max_tokens=400, temperature=0, messages=[{"role": "user", "content": FAITHFULNESS_PROMPT.format( context=context, answer=answer )}], ) import json return json.loads(resp.content[0].text.strip()) # Usage in an eval loop result = faithfulness_score( answer="Returns are free within 30 days. Standard shipping applies.", context_chunks=["Returns accepted within 30 days of purchase."], ) # result["faithfulness_score"] = 0.5 # result["claims"][1]["verdict"] = "unsupported" # "standard shipping applies" wasn't in context Pin temperature=0 and the model version. When you upgrade the judge model, re-baseline your score distributions before trusting comparisons.
The grounding check: fast faithfulness proxy for CI
The LLM-based scorer costs tokens and takes ~500ms per call. You need something that runs on every PR without a bill. A grounding check gives you ~70% of the signal for free:
import re def extract_facts(text: str) -> set[str]: """Extract numbers, percentages, timeframes — the claims most likely to hallucinate.""" return set(re.findall( r'\b\d+(?:[.,]\d+)?(?:\s?%|\s?days?|\s?hours?|\s?months?|\s?\$)?\b', text, re.IGNORECASE )) def grounding_check(answer: str, context_chunks: list[str]) -> list[str]: """Return specific facts in the answer that appear in no context chunk.""" context_text = " ".join(context_chunks) context_facts = extract_facts(context_text) answer_facts = extract_facts(answer) return list(answer_facts - context_facts) def test_refund_answer_grounded(): context = ["The return window is 30 days from delivery date."] answer = "You have 30 days to return items. A $5 handling fee may apply." ungrounded = grounding_check(answer, context) assert not ungrounded, f"Answer claims facts not in context: {ungrounded}" # fails: {'$5'} — the model invented a fee that isn't in the source This catches number hallucinations instantly. What it misses: semantic inversions ("returns are not accepted" when the source says they are), invented qualitative claims without numbers, and omissions. The LLM scorer handles those.
Combining the metrics in an eval suite
Don't pick one metric. Run the cheap check first, escalate to scored eval only when needed:
def full_faithfulness_eval(answer: str, context_chunks: list[str]) -> dict: # Layer 1: instant, free ungrounded = grounding_check(answer, context_chunks) if ungrounded: return {"verdict": "FAIL", "reason": f"invented facts: {ungrounded}", "score": 0.0} # Layer 2: scored, ~500ms scored = faithfulness_score(answer, context_chunks) score = scored["faithfulness_score"] if score < 0.6: return {"verdict": "FAIL", "reason": "faithfulness score below threshold", "score": score} if score < 0.8: return {"verdict": "REVIEW", "reason": "borderline faithfulness", "score": score} return {"verdict": "PASS", "reason": "", "score": score} In CI: run only Layer 1. In your nightly eval suite: run both. Any REVIEW verdict feeds a Slack thread for a human — expect 2–5% of your eval set to land there, not 100%.
What faithfulness doesn't tell you
A score of 1.0 means the model said nothing beyond what the context contained. That's the ceiling. The floor — whether the context itself was right — is a retrieval problem: context precision (did the retriever pull relevant chunks?) and context recall (did it pull all the relevant chunks?). Faithfulness doesn't touch either.
It also doesn't catch omissions. If the source says "this product is for adults only" and the model writes a glowing summary with no age restriction, every claim in the summary may be supported by some chunk — faithfulness passes. The omission of a safety constraint is invisible to faithfulness scoring. You need a separate "completeness" rubric for that, targeted at your high-stakes fields.
The faithfulness + grounding check combo gets you ~90% of attribution failures. The other 10% — omissions, subtle re-framings, retrieval failures — need purpose-built checks for your domain.
If you want a working eval harness with grounding check, RAGAS integration, and a custom judge prompt already wired up, the free test lab has it. The full chapter on RAG eval design — including how faithfulness interacts with the other three RAGAS dimensions and when to weight them differently — is in AI Software Testing in Practice.
FAQ
What is answer faithfulness in RAG?
Faithfulness measures whether every factual claim in the generated answer is supported by the retrieved context — not by the model's training data. A faithful answer only asserts things the retriever actually returned. RAGAS defines it as: supported_claims / total_claims, where each claim is checked for entailment against the context.
How does the RAGAS faithfulness score work?
RAGAS faithfulness decomposes the answer into atomic statements, then uses an LLM to check whether each statement is entailed (supported) by the retrieved context. The final score is the fraction of statements that pass: 1.0 means every claim is grounded, 0.0 means none are. Scores below 0.6 almost always indicate real hallucinations worth reviewing.
What faithfulness metric should I run in CI versus a nightly eval?
Run a deterministic grounding check (keyword/number extraction against source chunks) in every CI build — it's free and instant. Reserve the RAGAS faithfulness score or an LLM-as-judge for your nightly eval suite; they cost tokens and take seconds per call. Never gate a PR solely on a probabilistic score.