Fixtures Rot: Put a Staleness Gate in Front of Your Model-Backed CI

A recorded model response is a cache entry with no expiry date. That is the entire bug. You record it once, replay it forever, and six weeks later your pipeline is confidently green against a prompt nobody remembers editing.

The merge queue is usually not the weak point. Teams that adopt recorded fixtures tend to get the gate right and the lifecycle wrong. The fixture becomes a binary blob that everyone is afraid to touch, so it never gets refreshed, and its assertions slowly stop describing anything real.

Below is the lifecycle I use: a decision table for where each check belongs, a local hook that never touches the network, a manifest with a hash and a deadline, and a nightly job that opens a reviewable diff instead of quietly rewriting history.

1. Put each check where it can actually be trusted

Not every check deserves the same home. Local hooks and server-side CI have different failure modes, and mixing them is how you get developers bypassing --no-verify as a reflex.

Here is the placement table I start from. The latency budgets are example defaults, not measured numbers — set your own after one week of real data.

Two rules fall out of that table. Anything requiring a network call lives off the pull-request path, because network-dependent checks are the ones that produce random red builds. And any check that can fail for a reason unrelated to the diff must not block the merge — it reports instead.

2. Make the pre-push hook refuse to touch the network

The hook has one job: catch the deterministic failures in under two minutes, then get out of the way. It should be boring, and it should be safe to run on a plane.

1. Create the hook file inside the repo so it is versioned, not per-machine.
2. Force the offline project selection through an environment variable.
3. Add a hard timeout, and fail open when the timeout fires.
4. Register the hooks directory once per clone.

mkdir -p .githooks cat > .githooks/pre-push <<'HOOK' #!/usr/bin/env bash set -euo pipefail # Only the offline project. Live model calls never run from a hook. export MODEL_TESTS=offline if ! command -v node >/dev/null 2>&1; then echo "pre-push: node not found, skipping offline suite" >&2 exit 0 fi timeout 120 npx --no-install vitest run --project offline || { status=$? if [ "$status" -eq 124 ]; then echo "pre-push: offline suite exceeded 120s, deferring to CI" >&2 exit 0 fi exit "$status" } HOOK chmod +x .githooks/pre-push git config core.hooksPath .githooks 

Failing open on a timeout is deliberate. A slow laptop should not be able to block a push, and the same suite runs again in CI where timeouts are real failures. What the hook buys you is a fast local signal, not authority.

The part to enforce in review: no hook may read MODEL_ENDPOINT or any API key. If a model call appears in a hook, it will eventually hang on someone's flaky hotel wifi, and they will start pushing with --no-verify and never come back.

3. Give every fixture an owner, a hash, and a deadline

A fixture without a deadline is documentation. The manifest is what turns it into a check.

{ "schema": 1, "fixtures": [ { "id": "triage-missing-index", "path": "fixtures/triage/missing-index.json", "prompt_sha256": "REPLACE_WITH_REAL_SHA256", "model": "REPLACE_WITH_MODEL_ID", "recorded_at": "2026-11-01T02:14:00Z", "max_age_days": 14, "owner": "@your-team" } ] } 

Three fields do the heavy lifting. prompt_sha256 ties the recording to the exact prompt text that produced it. max_age_days is the point where an engineer has to look at the thing on purpose. owner means the staleness alert has a destination other than a shared channel nobody reads.

The verifier is short enough to read in full, and it has no dependencies beyond the standard library:

#!/usr/bin/env python3 """Fail if a fixture is stale, its prompt changed, or it is not in the manifest.""" import hashlib import json import pathlib import sys from datetime import datetime, timezone ROOT = pathlib.Path(__file__).resolve().parent.parent MANIFEST = ROOT / "fixtures" / "manifest.json" PROMPTS = ROOT / "prompts" NOW = datetime.now(timezone.utc) def sha256(path: pathlib.Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() def main() -> int: manifest = json.loads(MANIFEST.read_text()) errors: list[str] = [] declared: set[str] = set() for fx in manifest["fixtures"]: declared.add(fx["path"]) body = ROOT / fx["path"] prompt = PROMPTS / f'{fx["id"]}.md' if not body.exists(): errors.append(f'{fx["id"]}: recorded body missing') continue if not prompt.exists(): errors.append(f'{fx["id"]}: prompt file missing') continue if sha256(prompt) != fx["prompt_sha256"]: errors.append(f'{fx["id"]}: prompt edited since recording') recorded = datetime.fromisoformat(fx["recorded_at"].replace("Z", "+00:00")) age = (NOW - recorded).days if age > fx["max_age_days"]: errors.append(f'{fx["id"]}: {age}d old, budget {fx["max_age_days"]}d') for orphan in sorted((ROOT / "fixtures").rglob("*.json")): rel = orphan.relative_to(ROOT).as_posix() if rel != "fixtures/manifest.json" and rel not in declared: errors.append(f"{rel}: on disk but not in the manifest") print(f"checked {len(declared)} fixtures, {len(errors)} problem(s)") for err in errors: print(f" - {err}") return 1 if errors else 0 if __name__ == "__main__": sys.exit(main()) 

The orphan check is the one people skip and later regret. Fixtures deleted from the manifest but left on disk keep passing locally long after they stop being maintained, which is worse than deleting them.

Wire it as the first step of the PR job:

name: model-checks on: pull_request jobs: offline: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: python3 scripts/verify-fixtures.py - run: npm ci - run: npx vitest run --project offline 

Two seconds of hashing before two minutes of tests saves the team from debugging an assertion that was never about their change.

4. Flake control means separating drift from randomness

Most "flaky model test" reports are one of three things: the fixture is stale, the prompt changed, or the assertion is genuinely non-deterministic. They need different responses, and conflating them trains the team to retry blindly.

• Prompt hash mismatch — not a flake. The prompt moved; re-record deliberately.
• Age over budget — not a flake. Refresh or raise max_age_days with a written reason.
• Same fixture, different result on replay — a real bug in your harness or a genuinely unstable assertion.
• Live smoke test failing intermittently — the only case that earns a retry.

Keep a quarantine file with a mandatory expiry, and make expiry itself a CI failure:

{ "quarantine": [ { "id": "live-smoke-summarizer", "owner": "@your-team", "reason": "endpoint timeouts under load", "expires_on": "2026-11-21" } ] } 

If a quarantined entry reaches its expiry date, the gate blocks and someone has to either fix it or extend it in a visible diff. Silent permanent quarantine is how a suite ends up with twelve disabled tests and a green badge.

The retry policy I recommend: zero automatic retries on the offline suite, up to two on the nightly live smoke, and both attempts recorded in the job log. If you cannot see both attempts, you cannot tell drift from noise.

5. Re-record on a schedule, and make the job produce a diff

Refreshing fixtures by hand does not survive a busy quarter. Put it on a schedule, and make the output a pull request rather than a commit to main.

The loop below is complete and runnable; the model call is a single stub you point at whatever endpoint you use. Labeling it that way is honest — endpoint shapes differ, and faking a client would only hide the one line you need to change.

#!/usr/bin/env python3 """Nightly re-record. Signals drift by exiting non-zero so CI opens a PR.""" import hashlib import json import pathlib import sys from datetime import datetime, timezone ROOT = pathlib.Path(__file__).resolve().parent.parent MANIFEST = ROOT / "fixtures" / "manifest.json" def call_model(prompt: str) -> dict: """Your only integration point: wire your model client here.""" raise NotImplementedError("connect the endpoint you use") def main() -> int: manifest = json.loads(MANIFEST.read_text()) changed: list[str] = [] for fx in manifest["fixtures"]: prompt_path = ROOT / "prompts" / f'{fx["id"]}.md' body_path = ROOT / fx["path"] old = json.loads(body_path.read_text()) new = call_model(prompt_path.read_text()) if new == old: continue body_path.write_text(json.dumps(new, indent=2, sort_keys=True) + "\n") fx["prompt_sha256"] = hashlib.sha256(prompt_path.read_bytes()).hexdigest() fx["recorded_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") changed.append(fx["id"]) if changed: MANIFEST.write_text(json.dumps(manifest, indent=2) + "\n") print("drift:", *changed, sep="\n ") return 1 print("no fixture drift") return 0 if __name__ == "__main__": sys.exit(main()) 

Returning non-zero on drift is intentional. In most CI systems that turns the workflow into a failed run you can convert into a PR, which means a human reads the new outputs before they become the baseline. A job that silently commits is a job that launders regressions into expectations.

This worker is where a hosted environment actually helps, because the job needs two things a laptop cannot provide: a stable schedule and credentials that never sit in a developer shell. MonkeyCode is an open-source project that offers free model access and a free server option, so you can run the loop without provisioning anything first. The operator's current offer states 10 million tokens and a free server; treat that as the offer at the time of writing and check the live terms before you plan a schedule around it.

Do the arithmetic before you schedule anything. Tokens per day is roughly fixtures × average prompt tokens × runs per day, and a nightly run over a few dozen fixtures is a very different number from an hourly run over hundreds. My advice is to start nightly, measure one real week, then decide whether you need a second run per day.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One operational note: run this on a schedule or on prompt changes only. Never run it as part of a pull-request workflow, and never in a context where fork PRs can read the credentials.

6. Where this approach breaks, and who should skip it

This is a lifecycle for recorded fixtures, not a quality argument. A green replay proves your harness is deterministic and your fixtures are current — it says nothing about whether the model is good.

• High-variance outputs. Free-form generation with byte-level comparison will churn on every run and teach the team to ignore the job. Use schema or rubric assertions instead of equality.
• Sensitive prompts. If prompts or outputs contain customer data, do not record them at all. Write synthetic prompts and accept the coverage loss.
• Fast-moving models. If model versions change faster than your max_age_days, the staleness gate becomes background noise. Shorten the budget or move the whole check to nightly reporting.
• Very small suites. Under roughly ten fixtures, the manifest and verifier are more machinery than you need. A single test file with inline expectations is fine until it hurts.
• Teams that want to block merges on model quality. Don't. Block on staleness and determinism, report on quality, and let a human read the drift diff.

If you already have recorded fixtures and no manifest, the cheapest first step is the verifier alone. Add hashes and ages for your existing files, run it once, and count the failures. That number tells you how much of your current green is earned.

If you want to try the nightly loop without a provisioning conversation, the free server option and free model access are enough to run it at small scale — start with ten fixtures and a seven-day budget. Keep one thing in mind as the suite grows: green should mean "not stale", and nothing more.