Every Frozen Test Casts a Shadow: Auditing What Your Patch Gate Stops Seeing

A quarantine list is a subtraction. Every test you freeze removes a region of your codebase from the set of things your gate can still say "no" about — and nothing in a normal CI log tells you how large that region has grown.

This post is about the failure mode that follows: an agent patch lands green because the only tests covering its changed lines are the ones you froze last month. Below is a reproducible audit that computes that region explicitly, joins it to the diff, and fails the gate when a hunk is covered only by frozen tests.

It is a complement to test-time budgeting and property checks, not a replacement. It answers one question precisely: after quarantines, which lines can no longer be defended?

The failure mode, stated precisely

Take three sets for a given commit:

• C — lines changed by the patch (added or modified), from git diff -U0.
• K(l) — the set of tests whose execution covered line l, from per-test coverage contexts.
• F — the frozen/quarantined tests.

A line is shadowed when K(l) is non-empty and K(l) ⊆ F. Something does cover it. Nothing that can block a merge covers it. Plain "uncovered lines" reports miss this entirely, because the line is covered — by a test with a mute button.

The distinction matters because the two conditions want different fixes. Uncovered lines need a new test. Shadowed lines need either a re-entry decision on the frozen test, or an explicit acceptance that this hunk ships unguarded.

Step 1: Get per-test coverage, not aggregate coverage

Aggregate coverage cannot answer K(l). You need a context label per executed line. With coverage.py that is a config change, not a plugin:

# .coveragerc [run] branch = True dynamic_context = test_function 

coverage run -m pytest -q coverage json --show-contexts -o coverage.json 

dynamic_context = test_function labels each executed line with the test running at the time. --show-contexts emits those labels in the JSON report under a contexts map keyed by line number.

Two things to verify before trusting the output, because the exact JSON key layout has moved between coverage.py releases:

1. coverage.json contains a files -> <path> -> contexts object, not just executed_lines.
2. The context strings match your test node IDs (e.g. tests/test_billing.py::test_proration), not a generic bucket.

If either check fails, stop. An audit built on the wrong key silently reports zero shadow, which is the worst possible result.

Step 2: Keep frozen tests running — just not gating

This is the step most teams get backwards. If you skip or deselect frozen tests, their contexts vanish from the coverage report, and shadowed lines collapse into the ordinary uncovered bucket. You lose the ability to tell "nobody tests this" from "only the muted test tests this."

Run them. Mark the result non-blocking. Two workable shapes:

• Mark at runtime: pytest -m "not gating" for the blocking run, and a second unmarked run purely to feed coverage.
• Or run the full suite once, and let the report be non-blocking for frozen IDs while the exit code ignores them.

The first is cheaper to reason about in CI; the second keeps a single execution and one coverage artifact.

Step 3: The audit script

The script below joins the diff with the context map. It is deliberately small enough to read in one sitting.

#!/usr/bin/env python3 # freeze_shadow.py — flag diff hunks covered only by quarantined tests import fnmatch, json, re, subprocess, sys QUARANTINE = "quarantine.json" # {"frozen": ["tests/test_x.py::test_y", ...]} COVERAGE = "coverage.json" BASE = sys.argv[1] if len(sys.argv) > 1 else "origin/main" def added_lines(base): diff = subprocess.run( ["git", "diff", "-U0", f"{base}...HEAD", "--", "."], capture_output=True, text=True, check=True).stdout out, cur = {}, None for line in diff.splitlines(): if line.startswith("+++ b/"): cur = line[6:] out.setdefault(cur, set()) elif line.startswith("@@") and cur: m = re.search(r"\+(\d+)(?:,(\d+))?", line) start, count = int(m.group(1)), int(m.group(2) or 1) out[cur].update(range(start, start + count)) # ignore test files themselves  return {f: ls for f, ls in out.items() if ls and not f.startswith("tests/")} def is_frozen(ctx, frozen): return any(fnmatch.fnmatch(ctx, p) or ctx.endswith(p) for p in frozen) def main(): frozen = set(json.load(open(QUARANTINE))["frozen"]) cov = json.load(open(COVERAGE))["files"] shadowed, uncovered, ok = [], [], 0 for path, lines in added_lines(BASE).items(): fdata = cov.get(path) if fdata is None: uncovered.append((path, sorted(lines))) continue contexts = fdata.get("contexts", {}) for ln in sorted(lines): covering = contexts.get(str(ln), []) if not covering: uncovered.append((path, [ln])) continue if all(is_frozen(c, frozen) for c in covering): shadowed.append((path, ln, sorted(set(covering)))) else: ok += 1 for path, ln, ctxs in shadowed: print(f"SHADOW {path}:{ln} only covered by frozen: {', '.join(ctxs)}") for path, lns in uncovered: print(f"UNCOVERED {path}:{','.join(map(str, lns))}") print(f"\nchanged lines defended: {ok}; shadowed: {len(shadowed)}") sys.exit(1 if shadowed else 0) if __name__ == "__main__": main() 

Run it against a PR branch:

python freeze_shadow.py origin/main 

Sample output shape (illustrative, not a measured result):

SHADOW src/billing/proration.py:41 only covered by frozen: tests/test_billing.py::test_proration_rounds_down UNCOVERED src/api/routes.py:12,13 changed lines defended: 37; shadowed: 1 

Step 4: Turn the report into a policy

A hard fail on any shadowed line is too brittle for a large repo, and a warning is too easy to ignore. Classify, then decide:

The fourth row is the one that keeps this honest. If your shadow ratio never moves, you are not freezing tests — you are deleting them with extra paperwork.

Step 5: Where candidate patches come from

To calibrate the gate you need patches that touch shadowed regions. If you only ever test the audit against clean diffs, you learn nothing about whether it fires.

This is the narrow point where MonkeyCode's free model access is useful in my workflow: generating a batch of candidate patches aimed at previously shadowed files, so the audit has something adversarial to classify. The free server option covers the nightly run of the audit plus the frozen-suite coverage export, which is the slow part. Both are operator-supplied availability claims — verify the current terms before you plan around them, and keep the gate itself on infrastructure you control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The product does not enter the pass/fail decision. It changes patches; it does not judge them.

Limitations worth knowing before you adopt this

• Import-time lines get the first test's context. Module-level statements executed once during collection are attributed to whichever test triggered import. Treat shadow classification for those lines as advisory.
• Runtime cost is real. Per-test contexts increase report size and run time. On very large suites, scope the audit to changed packages rather than the whole tree.
• Subprocess coverage needs explicit wiring. Anything the suite spawns is invisible unless you propagate context.
• A tolerant matcher hides bugs. The endswith fallback in is_frozen() will match a differently-named test. Prefer exact node IDs and fail on unresolved entries.
• This measures coverage, not correctness. A line covered by a live, tautological test still isn't defended. Shadow auditing catches one specific class of blindness.

Who should skip this

Do not build this if your suite has no per-test context mechanism, if aggregate coverage is already below the point where line-level joins are meaningful, or if no one owns the quarantine manifest. In those cases a shadow report is a second artifact nobody reads — fix the manifest discipline first.

Start with the two assertions in Step 1. If contexts is empty, everything downstream is fiction, and green CI will keep telling you so.