A golden master test records exactly one run. That run happened once, in one locale, on one machine.
If you pin it immediately, you freeze a coincidence. The suite goes green today and red at 03:00 in CI.
This is the cheapest refactor failure to prevent. Prove the capture is deterministic, then pin it.
1. The Failure Mode: One Sample, Many Ambients
Legacy code reads ambient state everywhere. Clocks, locales, timezones, hash seeds, filesystem order, environment variables.
Your characterization test captures whatever those returned during recording. Every later run compares against that snapshot.
Nothing in the test says which inputs were ambient. So a failure six weeks later looks like a refactor bug.
It is usually a changed container timezone or a different LANG.
2. Build a Perturbation Matrix, Not a Bigger Fixture
Do not guess which ambient inputs matter. Measure them.
Step 1: list the perturbations you can reproduce cheaply. Timezone, locale, hash seed, working directory.
Step 2: run the target function in a child process for each one. Twice per cell.
Step 3: hash the serialized output. Compare hashes, never prose.
Step 4: report which cells differ from baseline. That set is your volatility set.
#!/usr/bin/env python3 """probe_stability.py — find ambient state a characterization test silently captured.""" import hashlib, json, os, subprocess, sys CHILD = r""" import importlib, json, sys module_path, fn_name, raw_arg = sys.argv[1], sys.argv[2], sys.argv[3] fn = getattr(importlib.import_module(module_path), fn_name) print(json.dumps(fn(json.loads(raw_arg)), sort_keys=True, default=str)) """ PERTURBATIONS = [ ("baseline", {}), ("tz_tokyo", {"TZ": "Asia/Tokyo"}), ("tz_utc", {"TZ": "UTC"}), ("locale_c", {"LANG": "C", "LC_ALL": "C"}), ("hashseed_1", {"PYTHONHASHSEED": "1"}), ("hashseed_2", {"PYTHONHASHSEED": "2"}), ] STRIPPED = {"PYTHONHASHSEED", "TZ", "LANG", "LC_ALL"} def run_once(module, fn, arg, extra_env): env = {k: v for k, v in os.environ.items() if k not in STRIPPED} env.update(extra_env) proc = subprocess.run( [sys.executable, "-c", CHILD, module, fn, json.dumps(arg)], capture_output=True, text=True, env=env, timeout=120, ) if proc.returncode != 0: last = proc.stderr.strip().splitlines()[-1][:120] return "ERROR:" + last return hashlib.sha256(proc.stdout.strip().encode()).hexdigest()[:12] def main(): module, fn = sys.argv[1], sys.argv[2] arg = json.loads(sys.argv[3]) if len(sys.argv) > 3 else {} rows = [] for name, extra in PERTURBATIONS: digests = [run_once(module, fn, arg, extra) for _ in range(2)] rows.append({ "perturbation": name, "runs": digests, "stable_in_env": digests[0] == digests[1], }) baseline = rows[0]["runs"][0] for r in rows: r["differs_from_baseline"] = r["runs"][0] != baseline print(json.dumps(rows, indent=2)) volatile = [r["perturbation"] for r in rows if r["differs_from_baseline"] or not r["stable_in_env"]] print("\nVOLATILE:", ", ".join(volatile) or "none", file=sys.stderr) if __name__ == "__main__": main() Run it against the exact function your test plans to capture:
python probe_stability.py orders.pricing quote '{"sku": "A1", "qty": 2}' The stderr line is the only output you need to gate on. VOLATILE: none means you can pin.
3. Read the Report as Three Different Signals
A cell that differs between two runs in the same environment is in-process nondeterminism. Reach for random, uuid, or iteration order.
A cell that matches its twin but differs from baseline is ambient dependence. Reach for a pin.
A cell that returns ERROR: is an environment trap, not a bug. Missing locale, unwritable path, absent config.
Do not collapse these into one "flaky" bucket. Each has a different fix.
4. Decision Table for the Volatility Set
| Ambient input | Cheapest honest fix | Never do this |
|---|---|---|
| Wall clock | Inject a clock seam at the function boundary | Call a freeze library inside business logic |
| Timezone | Set TZ in the test runner | Format dates as strings earlier |
| Locale, number format | Pin LANG/LC_ALL in the runner | Hardcode one separator in source |
| Hash seed, set order | Sort before serializing | Sort inside production code |
uuid, secrets, tokens | Stub at the call site the test owns | Regex the value out of the snapshot |
| Filesystem order | Use tmp_path, seed the directory | Assume os.listdir is stable |
| Network, DB state | Quarantine the test, fix the seam first | Retry until green |
The rule is constant. Push the ambient control to the edge of the system, never into the logic you are about to refactor.
5. Pin Only Inside the Characterization Suite
Scope the pins to the recording suite. Do not leak them into unit tests that should notice a real clock change.
# tests/characterization/conftest.py import random import pytest @pytest.fixture(autouse=True) def pin_ambient(monkeypatch, tmp_path): monkeypatch.setenv("TZ", "UTC") monkeypatch.setenv("LANG", "C") monkeypatch.setenv("LC_ALL", "C") monkeypatch.chdir(tmp_path) random.seed(0) yield One caveat matters here. PYTHONHASHSEED is read at interpreter start, so a fixture cannot change it.
Set it in the runner instead, and verify it was actually applied:
PYTHONHASHSEED=0 pytest tests/characterization -p no:randomly -q Do not trust the variable blindly. Print os.environ.get("PYTHONHASHSEED") in one test and assert it.
6. Make the Smallest Safe Change, Then Re-Probe
Now you can refactor. Extract one function. Change one call site. Keep the public signature untouched.
After the edit, re-run the probe on the same inputs. Two numbers must hold.
- The
VOLATILEset did not grow. - The golden output is byte-identical, or every difference is explained in the commit message.
A growth in the volatile set is a regression you caused. Fix it before the diff grows further.
Keep the patch small enough to review in one sitting. Aggregate line counts before you request review:
git diff --numstat -- src/legacy | awk '{a+=$1; d+=$2} END {print a" added, "d" deleted"}' If a file outside src/legacy changed, stop. That is a scope question, not a refactor.
7. Where Hosted Model Access Helped, and Where It Did Not
This workflow is child-process heavy. Six perturbations times two runs, times every function you plan to capture.
I ran those matrices on MonkeyCode's free server option so the loop did not occupy my laptop. I also used its free model access to draft the pinning fixture and the decision-table rows. Both availability claims are operator-supplied, so confirm the current limits yourself before relying on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The models were useful and wrong in equal measure. One proposed patching the clock deep inside the pricing function, which would have hidden the exact bug I was trying to freeze. I rejected it and moved the seam to the boundary.
Treat generated pins as candidates. Every one needs a matching probe run before it enters the suite.
8. Limitations and Who Should Skip This
This approach costs child processes. For a function taking a large binary input, the matrix is slow and the hashing is wasteful.
It also assumes the legacy behavior is worth preserving. If the current output is the bug, a golden master locks the bug in. Characterize only what a caller depends on.
Skip this if your target needs shared database state. Parallel child processes will collide, and the probe results become noise.
Skip it too when the suite already runs under a fixed container image. Half the matrix is then a no-op, and you can check TZ and hash seed by inspection.
Finally, this proves determinism, not correctness. A stable wrong answer is still wrong.
The Two-Line Test
Before you pin any golden master, ask two questions. Does it reproduce twice in the same environment? Does it survive a timezone change?
If either answer is no, the snapshot is a coin flip. Fix the ambient input first, then refactor with confidence.
Run the probe on your least favorite module this week. The VOLATILE line is usually shorter than you fear and more surprising than you expect.
