When a Flake Freeze Expires: Three Ledger Checks Before the Test Comes Back

A quarantine list only grows. Someone adds a test on Monday, someone else adds two more on Thursday, and by the end of the quarter nobody remembers which entries are still justified.

The decision that actually matters is not "is this test flaky." You already settled that argument when you froze it. The hard decision is the exit interview: what evidence do you need before that test runs in the merge gate again?

My answer is deliberately boring. Three small YAML ledgers and one replay log, validated by a script that fails the build when they disagree with reality. No dashboard. No weekly triage meeting. If a freeze is stale or a fixture has drifted, CI says so.

One caveat before the code: the rows below are schema illustrations, not measurements from a real repository. Treat them as fixtures for the checker, not as results.

1. Split the quarantine file into three ledgers

A single flaky.txt is unqueryable. It cannot tell you who owns a freeze, what input the test depended on, or which behavioral property you were hoping to preserve. Split it.

ledgers/freezes.yaml — one row per quarantined test. It carries an owner, a frozen date, a hard expiry, and the property ID that the test was supposed to protect.

# ledgers/freezes.yaml (illustrative rows) - test: tests/test_orders.py::test_partial_refund owner: @finley frozen_on: 2026-10-20 expires: 2026-11-19 reason: "assertion depends on wall-clock ordering" property_id: PROP-014 replay_env: pinned-py311 

ledgers/fixtures.yaml — content hash for every fixture file a frozen test or a property touches. This is the part teams skip, and it is the part that catches silent drift.

# ledgers/fixtures.yaml - id: FX-refunds path: tests/fixtures/refund_events.json sha256: "0b1c...e9" - id: FX-clock path: tests/fixtures/clock_sequence.json sha256: "7d4a...12" 

ledgers/properties.yaml — the properties you actually care about, bound to both tests and fixtures. A property with no fixture binding is a wish.

# ledgers/properties.yaml - id: PROP-014 statement: "sum(refunds) never exceeds original charge" fixtures: [FX-refunds, FX-clock] tests: [tests/test_orders.py::test_partial_refund] 

ledgers/replays.jsonl — append-only. One line per replay run, with the seed list and the pinned environment label.

{"test":"tests/test_orders.py::test_partial_refund","run_at":"2026-11-04T02:10:00Z","env":"pinned-py311","seeds":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20],"passed":20,"total":20} 

2. Write the checker before you write the policy

Policies in prose get ignored. A checker that exits non-zero gets fixed. This script is the whole enforcement layer, and it runs in under a second on a ledger of a few hundred rows.

#!/usr/bin/env python3 # tools/check_ledger.py """Fail the build when a flake freeze is stale, unowned, or unevidenced.""" import hashlib, json, sys from datetime import date from pathlib import Path import yaml ROOT = Path(__file__).resolve().parent.parent L = ROOT / "ledgers" MAX_WINDOW_DAYS = 30 MAX_REPLAY_AGE_DAYS = 14 MIN_SEEDS = 20 def yaml_rows(name): return yaml.safe_load((L / name).read_text()) or [] def sha256(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() def check_freezes(freezes, today): errs = [] for f in freezes: t = f["test"] if not f.get("owner"): errs.append(f"{t}: freeze has no owner") frozen = date.fromisoformat(str(f["frozen_on"])) expires = date.fromisoformat(str(f["expires"])) if expires < today: errs.append(f"{t}: freeze expired on {expires}, unfreeze or re-justify") if (expires - frozen).days > MAX_WINDOW_DAYS: errs.append(f"{t}: freeze window exceeds {MAX_WINDOW_DAYS} days") if not f.get("property_id"): errs.append(f"{t}: no property_id, nothing protects the behavior") return errs def check_fixtures(fixtures): errs = [] for fx in fixtures: p = ROOT / fx["path"] if not p.exists(): errs.append(f"{fx['id']}: fixture file missing at {p}") continue if sha256(p) != fx["sha256"]: errs.append(f"{fx['id']}: content hash drifted, lock is stale") return errs def check_properties(props, freezes, fixtures): errs = [] frozen = {f["test"] for f in freezes} known_fx = {fx["id"] for fx in fixtures} for pr in props: pid = pr["id"] if not pr.get("tests"): errs.append(f"{pid}: property has no test mapping") for t in pr.get("tests") or []: if t in frozen: errs.append(f"{pid}: only protected by frozen test {t}") for fxid in pr.get("fixtures") or []: if fxid not in known_fx: errs.append(f"{pid}: references unknown fixture {fxid}") if not pr.get("fixtures"): errs.append(f"{pid}: property has no fixture binding") return errs def check_replays(freezes, replays, today): errs = [] for f in freezes: t = f["test"] rows = [r for r in replays if r.get("test") == t] if not rows: errs.append(f"{t}: no replay record, freeze is unevidenced") continue last = max(rows, key=lambda r: r["run_at"]) age = (today - date.fromisoformat(last["run_at"][:10])).days if age > MAX_REPLAY_AGE_DAYS: errs.append(f"{t}: last replay is {age} days old") if len(last.get("seeds") or []) < MIN_SEEDS: errs.append(f"{t}: replay used fewer than {MIN_SEEDS} seeds") if last.get("env") != f.get("replay_env"): errs.append(f"{t}: replay env mismatch ({last.get('env')} != {f.get('replay_env')})") return errs def main() -> int: today = date.today() freezes = yaml_rows("freezes.yaml") fixtures = yaml_rows("fixtures.yaml") props = yaml_rows("properties.yaml") replays = [json.loads(l) for l in (L / "replays.jsonl").read_text().splitlines() if l.strip()] errs = ( check_freezes(freezes, today) + check_fixtures(fixtures) + check_properties(props, freezes, fixtures) + check_replays(freezes, replays, today) ) for e in errs: print(f"LEDGER: {e}") print(f"{len(freezes)} freezes, {len(fixtures)} fixtures, {len(props)} properties, {len(errs)} violations") return 1 if errs else 0 if __name__ == "__main__": sys.exit(main()) 

Wire it into two places: a pre-commit hook so authors see violations before pushing, and the merge gate so nobody can route around it.

python tools/check_ledger.py || exit 1 

3. Replay on a pinned host, not on a laptop

A replay record is worthless if the environment floats between runs. Fix the interpreter, the timezone, and the random seed set, then append the result.

pytest "tests/test_orders.py::test_partial_refund" \ -p no:randomly \ --tb=short \ $(python -c 'print(" ".join(f"--seed={i}" for i in range(1,21)))') 

Most projects will not accept that invocation as-is; the seed flag comes from a plugin. Adapt the flag, keep the twenty-seed floor. The floor is the point: one green run proves nothing about a test that failed one time in forty.

This is where I use the free server option in MonkeyCode: a small always-on box with a fixed image, so replay environment labels are stable across weeks and I am not burning a laptop for a nightly job. The free model access in the same workspace is useful for one narrow task — drafting candidate properties from a frozen test's assertions. I hand-edit every draft, because a model that proposes the property will also happily propose one the code cannot violate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

4. Turn the ledger state into a decision

The checker reports violations. It does not decide what to do. That mapping stays human and explicit, which is why it fits in a table.

Row two is the unpopular one. Deleting a test feels like losing coverage, but a test that fails one run in forty has already stopped providing a signal — it provides noise plus a habit of clicking retry. The property survives the deletion. That is the whole reason the property ledger exists.

5. What this scheme cannot see

• Semantic drift. A hash lock catches bytes, not meaning. If someone rewrites the fixture generator to produce equivalent-but-different data, the hash changes and you re-lock it, and you have learned nothing about whether the new data still exercises the boundary.
• Ordering and shared state. A twenty-seed replay on one host under-samples cross-test pollution. If your failures cluster when the full suite runs, replay the suite, not the test.
• Network and clock dependencies. The checker will happily unfreeze a test that is green on the pinned box and red on the CI runner. Vendoring the dependency is a prerequisite, not an optimization.
• True model behavior. Property drafting with a hosted model is a suggestion engine. It cannot know which invariants your domain treats as sacred.

6. Who should not adopt this

Skip the ledger if you quarantine fewer than a handful of tests a quarter — the bookkeeping will cost more than the flakes. Skip it if you cannot pin fixture data, for example because fixtures are generated from live PII-bearing records; hashing those files creates a compliance question you do not want. And skip it if your flakes come from a shared integration environment you do not control, because a replay record from your own pinned host tells you nothing about the environment that actually fails.

Everyone else: the script is about 90 lines, the ledgers are three files, and the first run will probably surface a freeze from eight months ago that nobody owns. Delete that one first.