A broken queue job throws. A bad migration stops the deploy. A RAG feature that gets worse does neither: it keeps answering confidently and keeps citing sources. They are just the wrong sources, and you find out when a user does.

I built laravel-rag, an open-source Composer package, so that kind of regression fails a build instead. Its rag:eval command runs golden questions against your corpus, scores recall and MRR, and with --ci exits non-zero when a score drops below a floor you set. Version 0.3.0 rebuilt the package on the official Laravel AI SDK, and this tutorial is written for it. The laravel-rag case study covers why it exists; this post goes from install to a red GitHub Actions check.

Why retrieval regressions never throw an error

Three ordinary changes cause most of them: a different chunk size, a cheaper embedding model, or a few hundred new documents crowding the old ones out of the top results. None raise an exception, and unit tests keep passing because they never check what the retriever returns for a real question.

So test retrieval directly. Scoring final answers needs an LLM in the loop, which is slow, costs money and varies between runs, so it belongs in a nightly job. Scoring retrieval costs one embedding call per question, and it is where most regressions start: if the right chunk never reaches the prompt, no prompt change recovers it. The wider retrieval fixes are in RAG that doesn't hallucinate.

Install laravel-rag on the Laravel AI SDK

The package is ikramulmustafa/laravel-rag on Packagist. Version 0.3.0 needs PHP 8.3+ with Laravel 12.62+ or 13.15+, plus the official Laravel AI SDK (laravel/ai ^1.4), which makes the embedding calls.

composer require laravel/ai ikramulmustafa/laravel-rag php artisan vendor:publish --tag=ai-config php artisan vendor:publish --tag=rag-config # then, in .env, before migrating: # DB_CONNECTION=pgsql # OPENAI_API_KEY=sk-... # RAG_EMBEDDING_PROVIDER=openai # RAG_EMBEDDING_MODEL=text-embedding-3-small # RAG_EMBEDDING_DIMENSIONS=1536 php artisan migrate 

Credentials and endpoints live in config/ai.php, which is the SDK's file. The package only says which provider and model to use. RAG_EMBEDDING_PROVIDER names a provider from config/ai.php; left empty, it falls back to the SDK's ai.default_for_embeddings. It has to be a single name, not a failover list, because a fallback provider would store another model's vectors under this model's id. Any SDK provider that supports embeddings works. A local model through Ollama needs no key:

RAG_EMBEDDING_PROVIDER=ollama RAG_EMBEDDING_MODEL=nomic-embed-text RAG_EMBEDDING_DIMENSIONS=768 RAG_EMBEDDING_SEND_DIMENSIONS=false 

The last line is "native" mode: no dimensions parameter is sent, for models that reject one (it works on the ollama and openai-compatible providers, and only while that provider has no models.embeddings.dimensions in config/ai.php, because the SDK then sends that value; the SDK's openai provider always sends a width). The width is still checked on every vector that comes back.

On PostgreSQL the migration runs CREATE EXTENSION IF NOT EXISTS vector (the database role needs permission), then creates rag_documents, which in 0.3 also keeps each document's text, rag_chunks with a vector column and an HNSW cosine index (for widths up to 2,000), and rag_eval_runs for eval history. Set RAG_EMBEDDING_DIMENSIONS first: the migration writes it into the column type. Changing it later on Postgres means php artisan rag:reindex --resize, which swaps the column for one of the new width without rewriting the table and then re-embeds everything.

SQLite and MySQL work too, computing cosine similarity in PHP. That loads every candidate row into memory, so it suits tests and small corpora, which is exactly what an eval job holds. Coming from 0.2? Read UPGRADING.md first, and save a baseline under 0.2 before you upgrade, as described further down.

Configure the embedder, chunking and floors

Settings live in config/rag.php. The embedding driver defaults to sdk. The fake driver is a deterministic, network-free embedder for tests; it builds vectors as wide as RAG_EMBEDDING_DIMENSIONS (default 1536), so the package's own tests and CI set 64 explicitly. The old openai driver name still works as a deprecated alias for sdk, and Rag::extend() registers a driver of your own. chunking.size and chunking.overlap are in characters (1000 and 200), and retrieval.min_score starts at 0.0, which drops nothing. The eval section, trimmed:

'eval' => [ 'suite_path' => env('RAG_EVAL_SUITE', base_path('tests/rag/suite.json')), 'k' => (int) env('RAG_EVAL_K', 5), 'thresholds' => [ 'pass_rate' => 0.90, // RAG_EVAL_MIN_PASS_RATE 'recall_at_k' => 0.80, // RAG_EVAL_MIN_RECALL_AT_K 'mrr' => 0.60, // RAG_EVAL_MIN_MRR ], 'tolerances' => [ 'pass_rate' => 0.0, 'recall_at_k' => 0.0, 'mrr' => 0.0, 'mean_top_score' => 0.02, ], ], 

Thresholds are absolute floors. Each one can be overridden per environment with the variable in its comment, and an empty value is ignored rather than read as 0.0. That is how a real embedding model gets higher floors than the hashed fake one. Tolerances apply only against a saved baseline and set how far a metric may drop before it counts as a regression.

Ingest a corpus with rag:ingest

Commit a small, representative set of real documents next to the suite for CI to rebuild on every run, say under tests/rag/corpus. Cover the topics users ask about most, plus documents that are easy to confuse.

# every *.md file under the directory (the default pattern) php artisan rag:ingest tests/rag/corpus # other file types, and no confirmation prompt (needed in CI) php artisan rag:ingest tests/rag/corpus --pattern='*.mdx' --force 
  • Sources are exact strings. Each file is stored under the path the command found it at, with forward slashes on every platform, so this records tests/rag/corpus/refunds.md, and the suite must name exactly that. Ingesting from the database instead? Whatever you pass as source to Rag::ingest() is what the suite matches.
  • Unchanged content is skipped. Content is hashed, so re-running does not pay to embed it again.
  • Big documents ask first. If one document is estimated above 100,000 tokens (ingestion.token_warning_threshold), the command asks before embedding it. --force skips the prompt, which CI needs.

Write the eval suite as JSON

Say the corpus holds refund, shipping and annual-leave policies. A suite is a JSON object keyed by case ID. Save it as tests/rag/suite.json, the default suite_path, and rag:eval finds it without a flag:

{ "refund-window": { "query": "How long do customers have to request a refund?", "expected_sources": ["tests/rag/corpus/refunds.md"], "must_contain": ["30 days"] }, "non-refundable-items": { "query": "Which items cannot be refunded?", "expected_sources": ["tests/rag/corpus/refunds.md"], "must_contain": ["gift cards"] }, "shipping-time": { "query": "How long does standard shipping take?", "expected_sources": ["tests/rag/corpus/shipping.md"] }, "leave-not-shipping": { "query": "What is the annual leave accrual rate?", "expected_sources": ["tests/rag/corpus/leave.md"], "must_not_contain": ["PO boxes"] }, "not-in-corpus": { "query": "What is the CEO's home address?", "expect_no_answer": true } } 
  • expected_sources: must appear in the top k. Recall and MRR come from these.
  • must_contain and must_not_contain: case-insensitive text checks on the retrieved chunks. The second catches a neighbouring document leaking in, like shipping rules turning up for a leave question.
  • expect_no_answer: passes only if retrieval returns nothing. Combining it with expected_sources is rejected when the suite loads.

No-answer cases are the only test of whether your system can say "I don't know", so include several. At the default min_score of 0.0 they fail against any non-empty corpus, and in a five-case suite that one failure drops pass_rate to 0.800, under the floor. The failure message prints the top score each one got; compare it with what real questions score (Rag::retrieve($question)->topScore() in Tinker) and set RAG_RETRIEVAL_MIN_SCORE between the two. Production retrieval uses the same floor, which is the point.

Run rag:eval and read the report

php artisan rag:eval prints a report. --k sets how many chunks each case retrieves (default 5), --suite picks another file, and --json prints only JSON on stdout. Here is a real ingest and eval of the package's own fixture suite on v0.3.0: three short policies and five cases much like the ones above, with the embedder, chunking and k its CI eval job uses (the fake embedder at 64 dimensions, 200-character chunks with an overlap of 40, k=2), run locally on a SQLite file database instead of Postgres. Inside a package, vendor/bin/testbench stands in for php artisan.

$ export RAG_EMBEDDING_DRIVER=fake RAG_EMBEDDING_DIMENSIONS=64 \ RAG_CHUNK_SIZE=200 RAG_CHUNK_OVERLAP=40 \ DB_CONNECTION=sqlite DB_DATABASE=/tmp/rag.sqlite $ touch /tmp/rag.sqlite && vendor/bin/testbench migrate --force $ vendor/bin/testbench rag:ingest tests/Fixtures/corpus --force 0/3 [>---------------------------] 0% 3/3 [============================] 100% INFO 3 ingested (7 chunks), 0 unchanged and skipped. Next: `php artisan rag:eval` to check retrieval still holds up. $ vendor/bin/testbench rag:eval --suite=tests/Fixtures/suite.json --k=2 --save=.rag-baseline.json INFO Suite 'suite' — 5 cases at k=2. +----------------+-------+-------+------+ | metric | score | floor | | +----------------+-------+-------+------+ | pass_rate | 1.000 | 0.900 | pass | | recall_at_k | 1.000 | 0.800 | pass | | mrr | 1.000 | 0.600 | pass | | mean_top_score | 0.550 | — | | +----------------+-------+-------+------+ INFO All cases passed. 
  • recall_at_k: the share of each case's expected sources found in the top k, averaged.
  • mrr: mean reciprocal rank, 1 divided by the rank of the first chunk from an expected source (0 if none). It rewards putting the right material first, since a prompt only fits a few chunks.
  • pass_rate: the share of cases that passed every check.
  • mean_top_score: average similarity of each case's best match. No floor; a steady slide usually means corpus dilution.

No-answer cases count toward pass_rate only. Keep k well below the corpus size. The fixture corpus is seven chunks, hence --k=2. At --k=4 the same corpus drops pass_rate to 0.800 while recall and MRR stay at 1.000: the window now holds the shipping chunk that mentions PO boxes, so the must_not_contain case fails whatever the ranking is. A window holding most of the corpus measures nothing. The eval searches every collection, so keep the eval database to your suite's corpus.

Without --ci, a completed run exits 0 whatever the scores. With --ci, it exits 1 when any metric under thresholds is below its floor. A missing suite file exits 1 either way.

Gate pull requests in GitHub Actions

The package gates its own pull requests with an eval job on Postgres and pgvector. Adapted for an application that embeds with OpenAI through the SDK:

name: Retrieval eval on: pull_request: jobs: eval: runs-on: ubuntu-latest services: postgres: image: pgvector/pgvector:pg16 env: POSTGRES_PASSWORD: postgres POSTGRES_DB: rag_test ports: ['5432:5432'] options: >- --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5 env: DB_CONNECTION: pgsql DB_HOST: localhost DB_PORT: 5432 DB_DATABASE: rag_test DB_USERNAME: postgres DB_PASSWORD: postgres OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} RAG_EMBEDDING_PROVIDER: openai RAG_EMBEDDING_MODEL: text-embedding-3-small RAG_EMBEDDING_DIMENSIONS: 1536 RAG_RETRIEVAL_MIN_SCORE: ${{ vars.RAG_RETRIEVAL_MIN_SCORE }} steps: - uses: actions/checkout@v5 - uses: shivammathur/setup-php@v2 with: php-version: '8.3' extensions: mbstring, pdo, pdo_pgsql coverage: none - run: composer install --prefer-dist --no-interaction --no-progress - name: Ingest the eval corpus run: | cp .env.example .env php artisan key:generate php artisan migrate --force php artisan rag:ingest tests/rag/corpus --force - name: Fail the build if retrieval regressed run: php artisan rag:eval --ci 

Job-level env values take precedence over the copied .env, and config/ai.php reads OPENAI_API_KEY from the repository secret. RAG_RETRIEVAL_MIN_SCORE comes from a repository variable: set it to the score floor you chose from your no-answer cases above. Until it is set, retrieval keeps the default of 0.0, the not-in-corpus case fails, pass_rate lands at 0.800 and the job stays red. The container starts empty, so every run embeds the whole CI corpus plus one call per question; keep it small.

For a job that checks the wiring without any key, replace the four embedding lines with RAG_EMBEDDING_DRIVER: fake and RAG_EMBEDDING_DIMENSIONS: 64. That is what the package's own eval job does, adding RAG_CHUNK_SIZE: 200 and RAG_CHUNK_OVERLAP: 40 so its three fixture documents become seven chunks, and running rag:eval with --k=2 --ci. The fake embedder hashes words into buckets. It is free, offline and deterministic, but not semantic, so its scores say nothing about how your real questions rank. Gate on the model you ship.

A real model does not have to mean a paid key. The package also runs a weekly workflow, eval-real.yml, that installs Ollama in the runner, pulls nomic-embed-text (768 dimensions, native mode) and runs the same fixture suite against a SQLite file database. Its first run, on 2026-11-19, passed all 5 cases at k=2 with 200/40 chunking: recall@k 1.000, MRR 1.000 and a mean top score of 0.769. Its floors are 0.95, set through the three RAG_EVAL_MIN_* variables, and it runs rag:eval with --ci --json. One detail worth copying: a final step checks that the saved report names ollama:nomic-embed-text:768 as its embedding model. The fixture suite also passes on the fake embedder, so a job that quietly fell back to it would look green.

Each run is also recorded in rag_eval_runs with its scores, failure reasons, embedding model, the settings it ran with (k, chunking, min_score, reranking) and the commit SHA when it can be found (on GitHub Actions, or from .git/HEAD). The history pays off where the database persists, such as staging, not in a throwaway CI container. A failed write only warns on stderr, so --json output stays parseable, and --no-record skips it.

Catch regressions the floors miss with a baseline

Floors answer "is this bad?". As a suite grows, "did this get worse?" matters more. Run php artisan rag:eval --save=.rag-baseline.json on main, commit the file, and gate pull requests with php artisan rag:eval --baseline=.rag-baseline.json --ci. Here is the fixture suite re-chunked from 200 characters to 80 (overlap 40 to 10, via RAG_CHUNK_SIZE and RAG_CHUNK_OVERLAP), reindexed, and compared with a baseline saved from the run above:

$ export RAG_CHUNK_SIZE=80 RAG_CHUNK_OVERLAP=10 $ vendor/bin/testbench rag:reindex --force WARN Re-embedding 3 document(s) with fake:deterministic-hash:64. This calls the embedding API for every chunk. 0/3 [>---------------------------] 0% 3/3 [============================] 100% INFO Reindex complete. $ vendor/bin/testbench rag:eval --suite=tests/Fixtures/suite.json --k=2 \ --baseline=.rag-baseline.json --ci INFO Suite 'suite' — 5 cases at k=2. +----------------+-------+-------+------+ | metric | score | floor | | +----------------+-------+-------+------+ | pass_rate | 1.000 | 0.900 | pass | | recall_at_k | 1.000 | 0.800 | pass | | mrr | 0.900 | 0.600 | pass | | mean_top_score | 0.538 | — | | +----------------+-------+-------+------+ INFO All cases passed. INFO Against baseline: pass_rate 1.000 → 1.000 +0.000 recall_at_k 1.000 → 1.000 +0.000 mrr 1.000 → 0.900 -0.100 mean_top_score 0.550 → 0.537 -0.012 WARN Baseline was run with different settings: chunk_size, chunk_overlap. WARN 1 metric(s) regressed: ↓ mrr dropped 0.1000 (1.0000 -> 0.9000) 

On floors alone, this change ships. Every case passes, every metric is above its floor, and the report says "All cases passed." The baseline shows what the floors cannot: MRR fell from 1.000 to 0.900, which across five cases means one question now gets its right document second instead of first. With a tight context budget, second can mean cut. Nothing threw, and the command exits 1. mean_top_score also slipped, by 0.012 (the comparison rounds to four places before printing, hence 0.537 against the table's 0.538), but that is inside its 0.02 tolerance, so it is reported and not counted.

A smaller chunk size can also hide a regression behind better-looking scores. In this post's 0.2 version, cutting the same corpus from 1,000-character chunks to 80 at k=1 raised mean_top_score from 0.311 to 0.436 and kept recall and MRR at 1.000, while the chunk now ranked first for "Which items cannot be refunded?" no longer said "gift cards". Only the must_contain check caught it.

The settings warning is new in 0.3. Each run records its k, chunking, min_score, reranking and embedding model, and a comparison across different settings says so, because a metric that moved after a chunking change is not a like-for-like comparison. It is a warning only; changing k or chunk size on purpose is a normal reason to re-save a baseline, not a build failure in itself.

With both flags, a build fails on a floor breach or any drop beyond a metric's tolerance. A missing or unreadable baseline file warns and skips the comparison, so a first run is not punished for having nothing to compare with. A baseline from a different embedding model gets its own warning that the scores are not comparable and should be re-saved.

What to do when the eval fails

Read the failure reasons first: each names the case, the query and the problem, such as an expected source missing from the top k, along with what came back instead. Then match the metric to a cause:

  • recall_at_k dropped: the right document left the window. Check chunking, the embedding model, or new documents outranking old ones.
  • mrr dropped, recall held: the right document ranks lower, where a tight context budget may cut it.
  • Only pass_rate dropped: usually a text check or no-answer case. A fact was split out of the top chunk, or something irrelevant is getting in.
  • mean_top_score sliding: the corpus is diluting. Check what was added.

After a chunking change, run php artisan rag:reindex, not rag:ingest. A chunk-size edit changes no content hash, so ingest skips every document; on the fixture corpus it reports 0 ingested (0 chunks), 3 unchanged and skipped, and the eval goes on scoring the old chunks. rag:reindex --stale does not help here either: it selects documents embedded with another model, and the model has not changed, so it reports "No stale documents to reindex." A plain rag:reindex re-embeds every stored document from the text saved at ingest time, so the working directory no longer matters, except for documents ingested before 0.3, which have no stored text and are re-read from their source path.

After a model swap, the retriever throws MixedEmbeddingModels on the first search, naming both model ids, until you reindex or restore the old model. Run php artisan rag:reindex --stale: it re-embeds only documents that still have chunks from another model or chunks without a vector. On Postgres, if the new model has a different width, run rag:reindex --resize instead: it changes the column, then reindexes as --stale. A document that fails is listed as collection/source, the command exits non-zero, and running --stale again resumes. Then re-save the baseline, because scores from different models are not comparable.

If a drop is a trade-off you accept, re-save the baseline in the same pull request so the reviewer sees the new numbers. Lowering a floor to get a green build is deleting a failing test with extra steps.

Where the package stops

laravel-rag still keeps to retrieval. Since 0.3 it has an optional second stage: SDK reranking, off by default, which fetches more candidates by vector similarity, asks a reranking model to pick the best, keeps both scores on each citation, and falls back to vector order with a logged warning if the reranker fails. Rag::tool() gives an SDK agent a search tool that returns scored citations as JSON, with an optional fixed collection, an allow-list of filters the model may use, and an output budget. What it still does not do: no BM25 blending, so exact identifiers like product codes remain the weak spot of pure vector search, and no answer-level or LLM-judge evals. Those still belong in a nightly job you build on top. If you are still choosing sources and chunking, start with my guide to implementing RAG.

To get retrieval gated like this in your own product, see my RAG development and Laravel development services.