Quarantine Withdraws Coverage: Find the Only Witness Before You Freeze a Test

A flaky test freeze is a coverage withdrawal, and most teams approve it without checking the balance. If the test you are about to quarantine is the only check that observes some invariant, every agent patch merged during the freeze is silently approved against one fewer witness. The fix is boring: make witnesses explicit, then refuse freezes that would drop a property to zero coverage.

This is the workflow I use now. It takes about an hour to adopt on a single service and it has one hard rule — no freeze without a second witness.

The rule

A test may be frozen only if every property it witnesses is also witnessed by at least one non-frozen check.

That is the whole policy. The rest of this article is the machinery that makes the policy cheap to enforce, because a policy you have to remember is a policy you will skip on a Friday afternoon.

Why agent patches make this urgent

A human patch usually arrives with intent. An agent patch arrives with a diff and a green CI run. Neither of those tells you which invariants the author had in mind, so CI status becomes the primary evidence — and CI status is exactly what a freeze quietly weakens.

Worse, flake freezes are the cheapest knob an agent can reach for. If an agent can edit the freeze list, it can turn a failing assertion into a skipped assertion without touching the assertion. That means the freeze list is a security boundary, not a housekeeping file, and it deserves the same review as production config.

Step 1 — Turn invariants into seeded property checks

Example tests say "this input works." Properties say "no input breaks this." For freeze decisions you need the second form, because a property keeps witnessing after the example test is gone.

# tests/properties/test_ledger.py from hypothesis import given, settings, strategies as st from app.ledger import Ledger, InsufficientFunds @given(st.lists(st.integers(min_value=-10_000, max_value=10_000), max_size=64)) @settings(max_examples=300, derandomize=True) def test_balance_never_negative(amounts): ledger = Ledger() for amount in amounts: try: ledger.apply(amount) except InsufficientFunds: pass assert ledger.balance >= 0 @given(st.integers(min_value=1, max_value=10_000)) @settings(max_examples=50, derandomize=True) def test_rejected_withdrawal_is_a_noop(amount): ledger = Ledger(initial=0) before = ledger.snapshot() try: ledger.apply(-amount) except InsufficientFunds: pass assert ledger.snapshot() == before 

Two details matter. derandomize=True makes the sweep deterministic, so a failure is reproducible without re-running the generator loop. And the property is named after the invariant, not the input — that name is the key you will join on later.

# reproduce a failure locally with a pinned seed pytest tests/properties -q --hypothesis-seed=12345 

If you already have a property library, skip the code and go to the join. The join is the part nobody does.

Step 2 — Record who witnesses each property

Keep a small JSON file next to the freeze list. No database, no dashboard. It has to be reviewable in a diff.

{ "properties": [ { "id": "prop.ledger.balance_non_negative", "check": "tests/properties/test_ledger.py::test_balance_never_negative", "witnessed_by": [ "tests/test_ledger.py::test_overdraft_rejected", "tests/test_ledger.py::test_withdraw_more_than_balance" ] }, { "id": "prop.ledger.rejected_is_noop", "check": "tests/properties/test_ledger.py::test_rejected_withdrawal_is_a_noop", "witnessed_by": [ "tests/test_ledger.py::test_overdraft_rejected" ] } ], "frozen": [] } 

witnessed_by lists the example tests that would catch a violation of the same invariant with concrete, readable inputs. Note the asymmetry in the sample above: prop.ledger.rejected_is_noop has exactly one witness. That property cannot be frozen today, and the gate below will say so.

Step 3 — Gate the freeze in CI

The gate is short enough to read in a review, which is the point.

# scripts/witness_gate.py — stdlib only import json import sys def main(path="properties.json") -> int: with open(path) as f: model = json.load(f) frozen = set(model.get("frozen", [])) orphans = [] for prop in model["properties"]: witnesses = set(prop.get("witnessed_by", [])) if not witnesses: orphans.append((prop["id"], "no witnesses recorded")) continue survivors = witnesses - frozen if witnesses & frozen and not survivors: orphans.append((prop["id"], f"last witness frozen: {witnesses}")) if orphans: for prop_id, reason in orphans: print(f"BLOCK {prop_id}: {reason}") return 1 print("ok: every property keeps at least one live witness") return 0 if __name__ == "__main__": sys.exit(main(*sys.argv[1:])) 

Wire it to the diff so it only bites when the freeze list actually changes:

if git diff --name-only "$BASE"...HEAD | grep -qx properties.json; then python scripts/witness_gate.py || exit 1 fi 

A blocked freeze is not a dead end. It produces exactly one of three follow-ups: write a property, promote an untracked test into witnessed_by, or delete the invariant on purpose and say why in the PR body. All three are better outcomes than a silent withdrawal.

Step 4 — Lock the generator, not just the fixture

Fixture locks catch edited fixtures. They do not catch a narrowed generator, and narrowing is the cheaper move: max_size=64 becomes max_size=4, everything still passes, and the property now witnesses almost nothing.

# scripts/prop_lock.py — fingerprint what the property actually exercises import hashlib import pathlib PATHS = [ "tests/properties/conftest.py", "tests/properties/strategies.py", "fixtures/ledger_golden.json", ] def fingerprint() -> str: h = hashlib.sha256() for p in PATHS: h.update(p.encode()) h.update(pathlib.Path(p).read_bytes()) return h.hexdigest()[:16] if __name__ == "__main__": print(fingerprint()) 

Commit the output to prop.lock and fail CI when the fingerprint moves without a reviewer ack. This is the same idea as a manifest diff, applied one layer down: the generator config is part of the test contract, so it belongs in the diff you read, not in the diff you skim.

Step 5 — Run the randomized sweep off the critical path

Here is where hosted capacity genuinely helps. Property sweeps are CPU-shaped and bursty: a 300-example sweep across a handful of properties is cheap, but running it on every push competes with the same runners your patch CI needs.

I keep the deterministic witness gate (Step 3) and the fingerprint check (Step 4) on the main runners, because they must be fast and authoritative. The long randomized sweep goes to a free server option so a bigger max_examples does not slow the merge path. In this setup that means MonkeyCode's operator-provided free server, with free model access used for one narrow job: proposing property candidates from a diff, always into a draft file that a human merges or discards.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The division of labor is deliberate. The model proposes invariant names and generator shapes; it never approves a freeze, never edits witnessed_by, and never decides that a property is satisfied. Those are review actions. Treat any free tier as convenience, not as an availability guarantee, and keep the authoritative gate where you control the runner.

Decision table

Row three is the one people get wrong. A property that fails at a pinned seed is a reproduced defect, so freezing it converts a known bug into an unowned one.

Limitations, and who should skip this

This workflow assumes your properties are honest. A generator that only emits small positive integers will happily certify a ledger that breaks on zero, and no amount of gate machinery fixes a distribution that never explores the boundary. Review the strategies like you review the assertions.

It also assumes a freeze actually matters to you. Teams with no rerun history, no flake inventory, and no agent patches in the loop should spend the hour elsewhere — a quarantine list with no attribution data is just a skip file with extra steps. And if your suite is mostly end-to-end tests sharing a database, the property-plus-witness split does not map cleanly; fix the shared state first.

Finally, this is a policy layer, not a proof. It tells you which invariants lose their last observer. It cannot tell you whether the invariant was ever the right one to hold.

Wrap

The gate is thirty lines and one JSON file, and it converts a silent coverage withdrawal into a reviewable decision. Start with a single property that maps to a test you are already tempted to freeze. If the gate blocks you on the first try, it is working.

If you want the sweep off your merge path, the free server option is a reasonable place to put it — just keep the witness gate on a runner you own.