Skip Is Not a Freeze: Revocable Seeded Freezes for Agent Patches

A skipped test is not a frozen test. It is a deleted test that still compiles. The difference matters the moment an agent starts shipping patches into the same suite, because a skip has no expiry, no signature, and no obligation to ever come back.

This post describes a freeze that revokes itself: every frozen test carries the seed that reproduces its failure, and every agent patch must re-run that seed and classify the result. If the seeded run now fails deterministically, the freeze is void — that is a regression, not flake.

All code below is a reference implementation. I have not run it against a production repository, so treat the thresholds and the signature function as starting points you calibrate on your own suite.

The timeline that motivates this

1. An agent patch touches a module next to test_export_roundtrip.
2. The test fails once in roughly twenty runs, so a reviewer adds @pytest.mark.skip(reason="flaky").
3. Four patches later, a real regression lands in that path. The test would have caught it, except it has been skipped for eleven days and nobody remembers the reason.

The flake was real. The freeze was not a freeze; it was amnesty. A test that only fails one time in twenty is still the cheapest detector you have for that code path — but only while something forces the question back open.

What a freeze has to contain

A freeze entry is a lease with four obligations:

• A reproducer: the interpreter environment plus the seed that turns an intermittent failure into a deterministic one.
• A signature: a coarse hash of the failure shape, so you can tell "same flake" from "different break" without reading logs.
• Collateral: at least one property check and one pinned fixture that keep partial coverage alive while the test is skipped.
• A revocation rule: a condition under which the freeze must be removed, evaluated by a job rather than by memory.

The ledger is human-owned. The gate that reads it runs read-only, and qa/freeze.json should sit under CODEOWNERS. If the agent can edit its own freeze ledger, you have automated the process of hiding failures from yourself.

Step 1: Capture a deterministic reproducer

You do not need to control every source of nondeterminism. You need to control enough that one seed reproduces one failure.

# Minimum environment pins that cover the common cases. PYTHONHASHSEED=0 TZ=UTC SOURCE_DATE_EPOCH=0 \ pytest tests/test_export.py::test_export_roundtrip -q -p no:cacheprovider -p no:randomly 

If repeated runs at a fixed seed still disagree, stop. That is not a freezable flake — it is scheduling, clock, network, or shared-state nondeterminism, and a seed will not describe it. Say so in the ledger and escalate instead of skipping.

Step 2: Write the ledger entry

{ "nodeid": "tests/test_export.py::test_export_roundtrip", "signature": "9f2c1ab4d7e0", "seeds": [0, 7, 13], "env": { "PYTHONHASHSEED": "0", "TZ": "UTC", "SOURCE_DATE_EPOCH": "0" }, "witnesses": [ "props/test_export_props.py::test_roundtrip_is_idempotent", "props/test_export_props.py::test_export_preserves_row_count" ], "fixtures": { "tests/fixtures/export_small.csv": "sha256:4e1f..." }, "opened": "2026-11-07", "owner": "finley" } 

Three seeds, not one. One seed proves the failure can be reproduced; three seeds give the gate something to distinguish "fails always" from "fails sometimes at fixed seeds." If all three reproduce the same signature, the flake behaves like a deterministic bug under those seeds, which is exactly what you want for a revocation rule.

Step 3: Classify on every agent patch

# gate.py — reference implementation, not executed here. import hashlib, json, os, subprocess, sys from pathlib import Path LEDGER = Path("qa/freeze.json") def signature(output: str) -> str: lines = [l for l in output.splitlines() if l.strip()] tail = lines[-1] if lines else "no-output" frame = next((l.strip() for l in reversed(lines) if l.strip().startswith("File ")), "no-frame") shape = "".join(c if not c.isdigit() else "N" for c in tail)[:120] return hashlib.sha1(f"{frame}|{shape}".encode()).hexdigest()[:12] def run_seeded(entry: dict, seed: int) -> tuple[int, str]: env = {**os.environ, **entry["env"], "PYTHONHASHSEED": str(seed)} p = subprocess.run( [sys.executable, "-m", "pytest", entry["nodeid"], "-q", "-p", "no:cacheprovider"], capture_output=True, text=True, env=env, timeout=300, ) return p.returncode, signature(p.stdout + p.stderr) def classify(entry: dict) -> str: results = [run_seeded(entry, s) for s in entry["seeds"]] codes = {rc for rc, _ in results} sigs = {sig for _, sig in results} if codes == {0}: return "SEED_STABLE" if codes == {1} and sigs == {entry["signature"]}: return "STILL_FROZEN" if codes == {1}: return "DRIFTED" return "UNSTABLE_FREEZE" 

The classification table is where the policy lives, and it should be written down before the first patch arrives:

The DRIFTED row is the one that earns its keep. Without it, a patch can convert a flaky assertion into a hard crash and the ledger keeps reporting a benign freeze, because the test is still red in the same place by name.

Keeping coverage alive while the test is dark

A freeze removes one assertion. It should not remove the whole code path from observation.

Each entry needs at least one property check that runs outside the skipped test. From the frozen test's assertions, extract the invariant that must hold for all inputs, not just the fixture row: idempotent roundtrip, row count preserved, ordering stable under sort. Those checks are weaker than the original assertion, and that is the point — weak coverage that runs beats strong coverage that is skipped.

Pin the fixtures by content hash and verify them before running the witnesses. An agent that cannot make the witness pass has an obvious next move: adjust export_small.csv. A hash check on the fixture turns that move into a failed gate instead of a green build.

Limitations and who should not use this

• Infra flake is out of scope. If failures correlate with container restarts, disk pressure, or shared database locks, seeds describe nothing. Fix the harness first.
• The cost multiplies. Total replay work is patches × frozen tests × seeds. A suite with eighty freezes and three seeds pays 240 extra test invocations per patch. Keep a hard cap on open freezes per repo, and treat hitting the cap as a signal to fix tests rather than to raise the cap.
• Time-dependent logic resists pinning. TZ and SOURCE_DATE_EPOCH cover a lot, but code that reads wall-clock deltas will still drift. Freeze those only if you can inject the clock.
• Do not use this on a suite with no property layer. If you have no invariants to hold open, a freeze is just a skip with extra JSON, and the ledger becomes the artifact you maintain while coverage decays.
• This is not a substitute for fixing the flake. It is a way to stop a fix from being forgotten. If nobody owns the entry, the owner field is decoration.

Where the free tier fits

The gate has two repetitive parts: drafting property invariants from a patch diff, and replaying a seeded matrix on dedicated compute. Neither is intellectually hard, and both are exactly the kind of work that stalls behind a budget conversation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I draft the witnesses with MonkeyCode's free model access — the prompt is the diff plus the frozen test's assertions, and the required output is invariants that do not touch the skipped code path. The useful failure mode to watch for is a model returning the original assertion restated as a "property"; a restated assertion will pass trivially and keep the freeze alive with fake coverage. Then the free server option runs the seeded replay loop, which is the part you cannot run once per patch by hand.

Everything above survives deleting the product name. The ledger, the signature, and the DRIFTED rule are the article; the tooling is convenience.

If you adopt one thing from this post, make it this: track how many freezes revoked themselves in the last month. A ledger where the answer is always zero is not clean — it is unread.