You change one line of a prompt. The demo still works. You merge. Two days later a user screenshots a nonsense answer and asks what happened.

What happened: nothing caught the regression, because nothing ran the prompt against real cases before merge.

Unit tests do not cover this. A prompt is not a function with one right answer - it has a distribution of acceptable answers, and the only cheap way to pin that distribution down is a small golden dataset plus a runner that scores every change against it.

The 20-minute setup

  1. Write 10-30 cases from real traffic. Each case: the input, plus what a good answer must contain (or must not contain). JSONL is fine: 
{"input": "refund request, order late by 3 weeks", "must_include": ["refund", "apolog"], "must_not_include": ["as an AI"]} {"input": "ask for discount on enterprise plan", "must_include": ["annual"], "max_words": 120}

  1. Run your app against the cases on every PR. Score with simple checks first (must_include, must_not_include, length, JSON validity). Add model-graded checks later if you need them - most teams never do.

  2. Fail the build when the pass rate drops below the last known-good run.

That is the whole pattern. The tooling should stay out of the way: a YAML file for config, a JSONL file for cases, one command in CI.

What this looks like with Litmus

I maintain Litmus, a zero-dependency harness for exactly this (single Python file, Python 3.8+, standard library only, MIT license):

{ "name": "support-bot-suite", "provider": "openai", "cases": [ { "name": "refund-request", "prompt": "Reply to this customer: {{message}}", "vars": {"message": "my order is 3 weeks late, I want a refund"}, "checks": { "contains": ["refund"], "not_contains": ["as an AI"], "max_latency_ms": 8000 } } ] } 
$ python3 litmus.py run tests.json --baseline .litmus-baseline.json Litmus 1.0.0 - suite 'support-bot-suite' (provider: openai) [FAIL] refund-request failed check: contains 'refund' [PASS] discount-question 1/2 cases passed REGRESSIONS vs baseline: - refund-request: PASS -> FAIL (contains 'refund') # exit code 1 - the merge is blocked

In CI the non-zero exit blocks the merge, and JUnit XML output feeds your CI dashboard. There is a deterministic offline mock provider, so you can test your wiring without spending tokens. The full setup guide is in the free starter pack (link on my profile).

The mistakes everyone makes first

  • Testing with vibes. "I tried it three times" is not a signal. Thirty saved cases beat three fresh tries because they are the same thirty every time.
  • Golden datasets that never grow. Add a case every time a user reports a bad answer. The dataset is the memory of every failure you have already paid for.
  • Model-graded everything on day one. String and length checks catch the majority of regressions and cost nothing. Start there.

Prompts change weekly now. The teams that treat them like code - versioned, reviewed, tested - ship faster precisely because they break less.

If you want the runner already built: Litmus Starter is a free sample pack (config, sample dataset, full CLI) and the complete version is $12 with the CI templates included. Links on my profile.