Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days.

The fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted pytest energy: one file, run it, exit code.

So I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI.

What a case looks like 

{"id": "refund-q", "input": "How do I get a refund?", "expect": {"contains": ["30 days"], "regex": "refund"}}

That's it. Input, plus what a good answer must contain. Run it:

python3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini 
 LITMUS REPORT ============================================ pass rate 97.4% (76/78) failed 1 errored 1 latency mean 0.84s p50 0.71s p95 1.9s ============================================================ [FAIL] refund-question (0.91s) contains: missing: 30 days

The part that actually saves you: compare

Run Litmus against your production prompt, save the JSON report as the baseline, and check every change against it:

python3 litmus.py compare baseline.json current.json --threshold 0.02

It lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide.

Design rules I stuck to

  1. One file, no dependencies. pip install is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it.
  2. No lock-in. Your dataset is JSONL in your repo. If you stop using Litmus tomorrow, your evals still mean something.
  3. Any app. Pipe mode tests anything that reads a case on stdin and prints an answer - Python, Node, a shell script. Endpoint mode talks to any OpenAI-compatible server, including local Ollama models, which makes the whole loop free.
  4. Cheap scorers first, judge last. contains and regex are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed.

What I learned building it

  • Mine real failures. Every user-reported bad answer becomes a case the same day. Ten real failures beat a hundred synthetic cases.
  • Small cases. One expectation per case, so the failure list reads like a bug report.
  • Latency is a regression too. A prompt change that doubled p95 response time is a production incident you haven't had yet.

Litmus is $12, one time: https://litmustools.gumroad.com/l/litmus. You get the tool, the field guide (eval-set patterns, LLM-judge rules, the CI workflow), and a runnable example. If it saves one bad deploy, it paid for itself.