When building production Retrieval-Augmented Generation (RAG) pipelines, the primary failure mode isn't retrieval latency or vector index size—it's silent hallucination.
LLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making.
To solve this in cyclic multi-agent graphs, I built and published rag-critic on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for LangGraph.
🔍 The Architectural Bottleneck
Standard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws:
- Coarse-Grained Feedback: Telling an agent "this answer feels inaccurate" doesn't specify which sentence was hallucinated.
- High Latency: Re-prompting large frontier models for full evaluations adds 3–5 seconds per step.
- Lack of Determinism: Without an entailment matrix, confidence scores drift across model runs.
📦 Project Links & Open-Source Repositories
PyPI Package: pypi.org/project/rag-critic
GitHub Repository: github.com/samjanjua6/rag-critic
Author Engineering Portfolio: https://shaheerulhassan.com
💻 Quickstart & Python Usage
Install directly from PyPI:
bash pip install rag-critic from rag_critic import RAGCritic critic = RAGCritic() # Evaluate a draft answer against retrieved source chunks result = critic.evaluate( draft_response="Python was released in 1991 by Guido van Rossum. It was originally named Monty.", retrieved_chunks=[ "Python is an interpreted programming language released in 1991 by Guido van Rossum." ] ) print(f"Status: {result.status}") # REVISE / APPROVED print(f"Grounding Score: {result.grounding_score:.2f}") # e.g. 0.50 print(f"Hallucination Index: {result.hallucination_index:.2f}") # Inspect claim-level verification for claim in result.claims: print(f"[{claim.status.value}] {claim.text}") # [SUPPORTED] Python was released in 1991 by Guido van Rossum. # [UNGROUNDED] It was originally named Monty. from langgraph.graph import StateGraph, END from rag_critic import create_critic_node, critic_router builder = StateGraph(MyRAGState) # Add standard generator and retriever nodes builder.add_node("retrieve", retrieve_docs) builder.add_node("generate", generate_draft) # Add rag-critic as an adversarial validation checkpoint builder.add_node("critic", create_critic_node()) builder.add_edge("retrieve", "generate") builder.add_edge("generate", "critic") # Dynamic cyclic routing based on hallucination thresholds builder.add_conditional_edges( "critic", critic_router, { "approved": END, # Output is grounded; return to user "rewrite": "generate", # Minor gaps; instruct re-writer "re_retrieve": "retrieve", # Heavy hallucination; expand search } ) 
