fast-rag-auditor: zero-config RAG retrieval & chunking audit (free, open-source)

A zero-config CLI that audits the two things that silently break RAG: chunking quality and retrieval speed/recall.

The problem

Most RAG failures aren't in the LLM — they're in how you split and retrieve text. Chunks that are too small lose context; chunks that are too large dilute the embedding; hidden near-duplicates bloat your index; and slow top-k queries kill UX. By the time you notice in production, it's expensive.

The tool

fast-rag-auditor is a tiny, dependency-free Python CLI from Autonomous Studio. Point it at a folder of files or a live URL:

pip install git+https://github.com/3km-collab/fast-rag-auditor.git fast-rag-auditor ./docs fast-rag-auditor https://en.wikipedia.org/wiki/Retrieval-augmented_generation --chunk-size 300 

It prints an instant report:

  • Chunk size distribution (tiny / small / good / large)
  • Self-retrieval recall@k — does each chunk return itself as a top-k match?
  • Near-duplicate ratio — redundant content wasting tokens
  • Retrieval latency (ms/query)
  • Health score (0–100) + actionable warnings, with --fail-under for CI gates

Everything runs locally with deterministic hashed embeddings — no API keys, no vector DB, no config.

Free vs. production

The auditor is the free entry point. When you're ready for production, the RAG Production Automation Kit adds real embeddings, hybrid search, ingestion connectors, evaluation, and Docker deployment.

Capability fast-rag-auditor (free) RAG Production Automation Kit (pro)
Chunk-size audit ✅ ✅
Recall@k proxy ✅ ✅ real embeddings
Duplicate detection ✅ ✅ auto-dedupe
Latency benchmark ✅ local ✅ real vector DB
Ingestion / hybrid / eval / deploy – ✅

Get started

MIT licensed. Feedback and stars are welcome.