Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test

Most agent-patch review stops at the diff of src/ and tests/. That is one layer too high. The patch that makes a red suite green often edits the setup layer — conftest.py, factories, seed data, golden files — and those edits are invisible to any check that only counts collected test IDs or compares test bodies.

This post adds the missing layer: a fingerprint over fixture files, a recorded map of which test uses which fixture, property assertions on the fixtures themselves, and a flake freeze ledger that expires. Everything below is a small script you can run in CI, not a framework.

The failure mode: green suite, different world

Three shapes recur in agent patches. A factory starts returning objects with the same email, so a uniqueness check becomes unreachable. A conftest.py fixture stops rolling back the DB session, so test N+1 sees rows from test N. A golden JSON file is regenerated to match new output, so the assertion now compares the patch to itself.

Each of those passes the suite honestly. None of them is a code change you would have approved if you had seen it. So make the setup layer an explicit, reviewable input.

Step 1 — Fingerprint the setup layer

Start with a content digest over every file that can change test state rather than test assertions.

# audit/setup_fingerprint.py import hashlib, json, pathlib, sys ROOT = pathlib.Path(__file__).resolve().parent.parent GLOBS = ( "conftest.py", "**/conftest.py", "tests/fixtures/**/*.py", "tests/factories/**/*.py", "tests/data/**/*", ) def digest() -> dict[str, str]: out = {} for pattern in GLOBS: for path in ROOT.glob(pattern): if not path.is_file() or "__pycache__" in path.parts: continue rel = path.relative_to(ROOT).as_posix() out[rel] = hashlib.sha256(path.read_bytes()).hexdigest() return dict(sorted(out.items())) if __name__ == "__main__": current = digest() if len(sys.argv) > 1 and sys.argv[1] == "--write": (ROOT / "audit/setup_baseline.json").write_text( json.dumps(current, indent=2, sort_keys=True) + "\n" ) print(f"wrote {len(current)} setup files") else: baseline = json.loads((ROOT / "audit/setup_baseline.json").read_text()) changed = sorted( p for p in set(baseline) | set(current) if baseline.get(p) != current.get(p) ) print("\n".join(changed) or "setup layer unchanged") sys.exit(1 if changed else 0) 

Run --write on the parent tree once, commit the baseline, and let the patch run the compare form. A non-zero exit means the review now has to look at setup files it would otherwise skim.

Step 2 — Record which test needs which fixture

A changed factory matters only if some test consumed it. Collect the usage map during a normal run.

# audit/fixture_usage_plugin.py import json, os, pathlib _usage: dict[str, list[str]] = {} def pytest_collection_modifyitems(session, config, items): for item in items: names = getattr(item, "fixturenames", None) if names is None: # older/newer internals shift; degrade instead of crash  names = [] _usage[item.nodeid] = sorted(set(names)) def pytest_sessionfinish(session, exitstatus): out = pathlib.Path(os.environ.get("FIXTURE_USAGE_OUT", "audit/fixture_usage.json")) out.parent.mkdir(parents=True, exist_ok=True) out.write_text(json.dumps(_usage, indent=2, sort_keys=True) + "\n") 

Enable it from the root conftest.py so CI and local runs agree:

pytest_plugins = ["audit.fixture_usage_plugin"] 

python -m pytest -q -p no:cacheprovider python -c "import json;u=json.load(open('audit/fixture_usage.json'));print(len(u),'tests mapped')" 

The map is a heuristic: it records fixture names, not definitions. Combine it with a grep for def <name>( under the setup globs to attribute a fixture to a file. Good enough to answer "does anyone still use this?" — and honest about not being a call graph.

Step 3 — Assert properties of the fixtures, not just outputs

Fixture bugs are invariants bugs. Write a handful of properties that fail loudly when the setup layer drifts. Adapt fixture names to your project; these are runnable as written against a conventional factory + session fixture.

# tests/test_setup_properties.py from tests.factories import make_user def test_factory_ids_are_unique(): ids = {make_user().id for _ in range(500)} assert len(ids) == 500, f"factory collided: {500 - len(ids)} duplicates in 500 draws" def test_factory_does_not_leak_defaults(): a, b = make_user(), make_user() assert a.email != b.email, "email default is a constant" assert a.created_at <= b.created_at, "created_at is not monotonic" def test_db_fixture_isolates_writes(db_session): db_session.add(make_user(email="isolation@example.com")) db_session.flush() assert db_session.query(type(make_user())).count() == 1 

Three properties, one minute of runtime, and they catch the exact class of patch that turns an assertion into a tautology. Keep them in a file the patch author is unlikely to touch, and require review when it changes.

Step 4 — Freeze flakes with an expiry date

Quarantining a flaky test is legitimate. Quarantining it forever, or quarantining a test the patch itself introduced, is not. Make the freeze data, not a comment.

[ { "test": "tests/test_billing.py::test_retry_window", "frozen_at": "2026-10-10", "expires": "2026-11-09", "reason": "upstream clock skew in CI; see #4120", "owner": "payments" } ] 

Three rules make the ledger useful:

1. expires is mandatory and at most 30 days out; an expired entry fails CI.
2. A frozen test may not be edited by the same patch that relies on the freeze — thaw first, then change.
3. A newly added test may never appear in the ledger. You cannot freeze evidence you just created.

Step 5 — One gate that enforces all four

# audit/gate.py import datetime, json, pathlib, subprocess, sys ROOT = pathlib.Path(__file__).resolve().parent.parent SETUP_MARKERS = ("conftest.py", "tests/fixtures/", "tests/factories/", "tests/data/") def git(*args): return subprocess.run(["git", *args], cwd=ROOT, capture_output=True, text=True, check=True).stdout.split() def main() -> int: changed = git("diff", "--name-only", "origin/main") added = git("diff", "--name-only", "--diff-filter=A", "origin/main") failures, today = [], datetime.date.today() touched_setup = [p for p in changed if any(m in p for m in SETUP_MARKERS)] if touched_setup and not (ROOT / "audit/fixture_changes.md").exists(): failures.append( f"setup layer changed {touched_setup} without audit/fixture_changes.md" ) ledger = json.loads((ROOT / "audit/flake_ledger.json").read_text()) for entry in ledger: path = entry["test"].split("::")[0] if datetime.date.fromisoformat(entry["expires"]) < today: failures.append(f"freeze expired: {entry['test']} ({entry['expires']})") if path in changed: failures.append(f"patch edits a frozen test: {path} — thaw it first") if path in added: failures.append(f"new test submitted already frozen: {path}") for f in failures: print(f"FAIL {f}") return 1 if failures else 0 if __name__ == "__main__": sys.exit(main()) 

Wire it as the first CI step, before tests. It costs under a second and it fails on the two things review reliably misses: silent setup edits and self-granted quarantine.

Where MonkeyCode fits, and where it does not

Property candidates are the tedious part: reading a factory and inventing invariants. I used MonkeyCode's free model access for that proposal loop — dump a fixture file, ask for five candidate properties, keep the ones that are falsifiable, throw the rest away. The gate above stays deterministic; no model runs at merge time. Its free server option is the other half: sampling a suspected flake over a week of repeated runs needs a machine that is not your laptop and not a metered CI runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Decision table for a red run on an agent patch

Limitations and who should skip this

This gate is about state, not correctness. It will not tell you whether the patch solves the bug — only whether the test world stayed comparable. The fixture usage map is name-based and will over-attribute fixtures defined in shared base classes. The property file needs a human to keep it meaningful; three noisy properties are worse than none.

Skip it if your suite is small enough that a reviewer reads every fixture diff, if you have no CI gate to hang the script on, or if your flakiness comes from a third-party network call you cannot replay — there, the honest fix is a recorded transport, not a longer freeze. And if your team treats audit/flake_ledger.json as a place to park problems indefinitely, delete the ledger: you have built a monument, not a gate.

Start with Step 1 today. One baseline file, one second of CI, and the next agent patch that quietly rewrites conftest.py shows up as a red line instead of a green suite.