For a few weeks I improved my prompts the way most people do: I changed a sentence, ran three examples I liked, and decided it was better. Then one "harmless" wording tweak quietly broke JSON extraction in a nightly automation. It ran for three days against a slightly renamed key before I noticed anything was wrong — not through an alert, but by scrolling a log for an unrelated reason.
That is the whole argument for this post. A prompt edit is a code change, and you cannot review a code change by feeling. You need a number that goes up or down, and you need the build to stop when the number goes the wrong way.
The harness below does that in under 600 lines of standard-library Python: no pytest plugin, no LangChain, no API bill. It ships with replay fixtures, so the same suite runs on a laptop with no GPU, no key, and no network.
Why not just ask a model to grade the answers?
That is the first thing everyone reaches for, and it is the wrong first tool. An LLM judge costs tokens on every run, gives a different verdict on Tuesday, and cannot tell you which field broke — only that "the answer seems somewhat worse".
Most of what actually breaks in production is not subtle. A key got renamed. A number is off by a rounding step. The model answered a policy question it should have refused. The summary grew from two sentences to six and no longer fits the card. All of those are checkable with json.loads, a set comparison, a float tolerance, and len().
So the design rule is: deterministic checks first, judge later, and only for the open-ended tail. In the dataset I run daily, every case is graded by strings and numbers. It costs zero, runs in milliseconds, and when it fails it tells me exactly which check and which output.
The four parts
- Cases — a JSONL file of prompts, checks, and thresholds.
- Metrics — small functions that turn
(output, spec)into a score in[0, 1]. - Runner — runs cases against any callable, never lets one crash kill the run.
- Report + exit code — Markdown for a PR comment, JSON for trends,
0/1for CI.
. ├── evals/golden.jsonl # the dataset ├── fixtures/model_good.jsonl # replays known-good outputs (free, hermetic) ├── fixtures/model_drift.jsonl # replays a regressed model ├── harness/ │ ├── metrics.py # scoring primitives │ ├── runner.py # cases, checks, report │ ├── models.py # replay / Ollama / OpenAI-compatible adapters │ ├── report.py # Markdown + JSON │ └── cli.py # the gate └── tests/test_harness.py # the harness evals itself 1. Cases are data, not code
A case is a prompt and one or more checks. Keeping it in JSONL means the dataset has a diff you can read in review, and adding a regression case takes fifteen seconds — which is the only reason people actually add them.
{"id": "extract-01", "prompt": "Extract the customer name, plan and issue category from this message, as JSON with keys name, plan, category. Message: \"Hi, this is Ada Lovelace on the Pro plan - my CSV exports keep timing out since Tuesday.\"", "checks": [{"metric": "json_subset", "spec": {"name": "Ada Lovelace", "plan": "Pro"}}, {"metric": "json_keys", "spec": ["name", "plan", "category"]}], "meta": {"area": "extraction"}} {"id": "nohalluc-01", "prompt": "Context: \"Starter plan includes 1,000 API calls per month. Upgrades require a support ticket.\" Question: Can I get a free upgrade to more calls? Answer in one sentence using only the context.", "checks": [{"metric": "contains_none", "spec": ["free upgrade", "unlimited"]}, {"metric": "contains_all", "spec": ["support", "ticket"]}], "meta": {"area": "grounding"}} Two details worth copying:
- Every check can carry its own
min, and the case carries amin_score. A case passes only if all checks clear their floor. Graded scores still get reported, so a slow slide from 1.00 to 0.75 is visible before it becomes a failure. - A check can carry metric-specific arguments (
"tol": 0.01fornumeric_close). Unknown keys are passed through, so adding a metric option never needs a loader change.
2. Metrics return a number, not a boolean
A boolean throws away the most useful signal you have. json_subset returns the fraction of expected fields that matched, so two of three wrong is 0.33, not just "failed" — and a case that used to be 1.00 and is now 0.67 tells you the model got worse at a specific field, not that everything is on fire.
def normalize(text: str) -> str: text = unicodedata.normalize("NFKC", text).strip().lower() text = _PUNCT.sub("", text) text = _WS.sub(" ", text) # Trailing sentence punctuation is noise for matching, not content. return text.strip(" .,") def _loads_object(text: str): """Return a parsed JSON object from possibly chatty output (fences, prose).""" start, end = text.find("{"), text.rfind("}") if start == -1 or end <= start: return None try: return json.loads(text[start:end + 1]) except json.JSONDecodeError: return None def json_subset(pred: str, spec, **_) -> float: got = _loads_object(pred) if not isinstance(got, dict) or not isinstance(spec, dict) or not spec: return 0.0 hits = 0 for key, want in spec.items(): if key not in got: continue have = got[key] if isinstance(have, str) and isinstance(want, str): if normalize(have) == normalize(want): hits += 1 elif have == want: hits += 1 return hits / len(spec) def contains_none(pred: str, spec, **_) -> float: """spec: forbidden substrings (hallucinated names, promises you cannot keep).""" hay = normalize(pred) bad = [normalize(s) for s in (spec if isinstance(spec, list) else [spec])] return 1.0 if not any(b and b in hay for b in bad) else 0.0 The registry has eight entries and I have not needed a ninth: exact_match, contains_all, contains_none, token_f1, json_subset, json_keys, numeric_close, length_between. Three do 90% of the work, and each fails in a way worth knowing:
-
contains_noneis the one that pays for the whole project. It is how you encode "never say free upgrade", "never invent a refund you cannot authorise", "never mention a competitor price". You cannot test a promise like that with a similarity score; you can test it with a banned-phrase check on every case that touches policy. -
token_f1is for prose only, and it is blunt. Word overlap rewards hedging and punishes a correct answer phrased differently. I use it with a low floor (0.7) as a "did the answer stay on topic" smoke test, never as a quality measure. -
length_betweencatches format rot that semantic checks miss — the summary that became six sentences, the Slack line that no longer fits, the product blurb that turns into an essay after a model upgrade.
normalize earns its keep more than any single metric: NFKC plus lowercase plus collapsed whitespace makes "Pro Plan", "pro plan" and "Pro Plan" one string, which is what you meant when you wrote "Pro".
3. The runner gives partial credit and absorbs crashes
The runner takes any Callable[[str], str]. That is the entire coupling to a model, and it means the same cases run against a hosted API, a local Ollama instance, and a JSON fixture with no changes.
def run_cases(cases, model, clock=time.perf_counter): results = [] for case in cases: start = clock() output, error = "", None try: output = model(case.prompt) or "" except Exception as exc: # a crashed model fails the case, not the run error = f"{type(exc).__name__}: {exc}" latency = clock() - start scores = [] if error is None: for check in case.checks: value = score_metric(output, check.metric, check.spec, **check.kwargs) scores.append((check.metric, value)) passed = error is None and all( value >= (check.min if check.min is not None else case.min_score) for (_, value), check in zip(scores, case.checks) ) results.append(CaseResult(case.id, output, scores, passed, latency, error)) return results The except Exception line is deliberate and it is the difference between a usable gate and an abandoned one. A rate limit, a timeout, a context-length error — those are findings, not reasons to lose the other nine results. Each failing case keeps its raw output, so the report shows you the text that broke, not just the id.
The report is the part a colleague actually reads. On the regressed fixture:
## LLM eval: FAIL — 6/10 cases - model: `replay:model_drift.jsonl` - mean case score: **0.715** | case | score | checks | time | |---|---|---|---| | `extract-01` ❌ | 0.75 | json_subset=0.50, json_keys=1.00 | 0 ms | | `classify-01` ❌ | 0.00 | contains_all=0.00, contains_none=0.00 | 0 ms | | `nohalluc-01` ❌ | 0.00 | contains_none=0.00, contains_all=0.00 | 0 ms | | `gloss-01` ❌ | 0.40 | token_f1=0.40 | 0 ms | ... extract-01 at 0.75 is the case I care about: json_subset=0.50 means the model answered with "plan": "pro plan" instead of "Pro". Structurally valid JSON, semantically wrong, invisible to any schema-only validator, and caught here in a diff of one line. That is the difference between "the fine-tuned model looks fine" and "the fine-tuned model renamed a value three downstream jobs depend on".
4. The gate is an exit code
CI does not read Markdown. It reads status.
$ python -m harness.cli --model replay:fixtures/model_good.jsonl --min-mean 0.95 eval: PASS 10/10 cases, mean 0.995, gate 0.95 # exit 0 $ python -m harness.cli --model replay:fixtures/model_drift.jsonl --min-mean 0.95 eval: FAIL 6/10 cases, mean 0.715, gate 0.95 # exit 1 Two independent conditions, because they catch different failures: a single critical case failing (a banned phrase, a broken schema) must block even when the mean is high, and a broad, uniform slide must block even when every individual case technically still clears its floor. The workflow file is nine lines:
name: eval on: pull_request jobs: gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: python -m harness.cli --cases evals/golden.jsonl --model replay:fixtures/model_good.jsonl --min-mean 0.95 --md eval.md Free by default: replay fixtures
Here is the constraint that shaped the design. Most of my automations run on free tiers, and "run the eval suite on every commit" must not mean "spend money on every commit" — least of all in a repo whose whole point is zero-cost automation.
So every case has a recorded output in fixtures/model_good.jsonl, and the model adapter for it is a dictionary lookup:
class ReplayModel: def __call__(self, prompt: str) -> str: if prompt not in self.table: raise KeyError(f"no recorded output for prompt: {prompt[:60]!r}") return self.table[prompt] This looks circular, and it is — on purpose. Replay mode does not test the model. It tests the harness: that the loader rejects duplicate ids and bad JSON with a useful line number, that a metric scores partial credit correctly, that a crashed model fails its case and not the run, that the CLI exits 1 when it should. Those are exactly the bugs that make people stop trusting an eval suite. Nineteen unittest cases cover the harness itself, and the suite runs on the default python:3.12 image because everything is stdlib.
Then, when you want a real measurement, swap the adapter:
# local, free, offline model = OllamaModel("llama3.2:1b") # or any OpenAI-compatible endpoint; the key comes from the environment model = ChatCompletionsModel("https://your-host/v1", "your-model", api_key_env="EVAL_API_KEY") The key is read from an environment variable and never written into a case file, a report, or a log — the JSON report contains scores and outputs, not credentials.
model_drift.jsonl exists for the opposite reason: it is the same ten cases answered by a deliberately worse model (a wrong enum value, a misclassification, one hallucinated upgrade, one thin definition). It is how the suite proves that the gate can still go red. A test suite that has never failed is a test suite nobody has verified.
Measured on my machine, both fixtures, ten cases: model_good → 10/10, mean 0.995; model_drift → 6/10, mean 0.715, with extract-01, classify-01, nohalluc-01 and gloss-01 failing for four different reasons. Two runs of the real dataset, one day apart, cost nothing and are directly comparable because the case file did not change.
What I would do differently
- Write cases from bugs, not from imagination. Every case in my file started as something that actually broke; invented cases test scenarios you already handle.
- Twenty to forty cases is the sweet spot. Below ten you are testing vibes again; above fifty, every run surfaces a flake and you start ignoring red.
- Never compare scores across dataset versions. Editing a spec moves the mean for reasons unrelated to the model. Version the case file next to the prompt.
- Pin temperature to 0 for eval runs, then run twice before believing a one-case difference. Two identical runs that disagree mean the case is flaky, and a flaky case in a gate is worse than none.
- Keep a
model_driftfixture committed. It is the cheapest proof that your red light still works. - Judge only what metrics cannot see. If you later add an LLM judge for tone or helpfulness, keep it out of the blocking path: run it nightly, store its score, and let a human look at the delta.
Starting in ten minutes
- Copy
harness/(five files, no dependencies). - Write five cases from the last five things that broke.
- Record their current outputs into
fixtures/model_good.jsonl— that is your baseline, not a target. - Add the gate to CI with
--min-mean 0.95. - Run it against a deliberately worse model before you trust the red.
That is the whole discipline: a prompt change that cannot move a number cannot be reviewed, and a change nobody can review is a change you are shipping on faith.
Further Resources
The five harness files, the ten-case starter dataset, both replay fixtures, and the CI workflow I actually use are collected in my workflow pack: The Complete 2026 AI Side-Hustle Blueprint: Zero-Cost Prompts & Workflows. Everything in this post is complete and runnable without it.
Note: this post contains one link to my own workflow pack. The code above is free, standalone, and does not require it.



