Prove Each Golden Can Fail Before You Refactor the Messy Repo

A characterization test that cannot fail is documentation. Documentation will not catch your refactor.

That is the whole thesis. Capture behavior first, then prove the capture can break, then make one change. The order matters more than any tool.

This post gives you three small scripts and one habit. The scripts fingerprint a runtime, pin golden behavior, and mutate source to see if the goldens notice. The habit is refusing to refactor a repo you have not run twice.

Conclusion first: three gates, one change

Before editing anything, you need three green lights.

1. A runtime fingerprint you can reproduce on a second machine.
2. A golden baseline that captures exit codes and stdout for a fixed case list.
3. Proof that each golden fails when the code is wrong.

Gate 3 is the one most teams skip. Without it, you refactor against tests that pass no matter what. Free models can now draft gate 2 quickly, which makes gate 3 mandatory, not optional.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 1: Fingerprint the runtime before you trust it

Most "messy repo" failures are environment failures wearing a code costume. You cannot tell a behavior change from a missing env var until you pin the box.

# fingerprint.py — stdlib only. Record what the repo needs in order to run. import hashlib import json import os import platform import subprocess import sys STABLE = ("TZ", "LANG", "PYTHONPATH", "PYTHONHASHSEED") PRESENCE_ONLY = ("PATH", "HOME", "VIRTUAL_ENV", "DATABASE_URL", "REDIS_URL") def h(value: str) -> str: return hashlib.sha256(value.encode()).hexdigest()[:12] def build() -> dict: fp = { "python": sys.version.split()[0], "impl": platform.python_implementation(), "system": platform.system(), "stable_env": {k: h(os.environ.get(k, "<unset>")) for k in STABLE}, "env_present": {k: k in os.environ for k in PRESENCE_ONLY}, "path_entries": len(os.environ.get("PATH", "").split(os.pathsep)), "pip_freeze_sha": None, "pkg_count": None, } proc = subprocess.run([sys.executable, "-m", "pip", "freeze"], capture_output=True, text=True, timeout=120) if proc.returncode == 0: lines = sorted(l for l in proc.stdout.splitlines() if l.strip()) fp["pip_freeze_sha"] = h("\n".join(lines)) fp["pkg_count"] = len(lines) return fp if __name__ == "__main__": payload = build() with open("fingerprint.json", "w") as fh: json.dump(payload, fh, indent=2, sort_keys=True) print(json.dumps(payload, indent=2, sort_keys=True)) 

Run it on the machine that "works". Then run it on the machine that does not. Diff the two JSON files.

Hash TZ, LANG, PYTHONPATH, and PYTHONHASHSEED. Record only presence for secrets and PATH. Never hash values you cannot reproduce.

pkg_count is the cheapest signal you have. A difference of one package explains a surprising number of "random" failures.

Step 2: Put the case list in data, not in code

Goldens rot when they live inside test files. Keep them declarative so a reviewer can read the whole input surface in one screen.

[ {"name": "empty_stdin", "args": [], "stdin": ""}, {"name": "one_good_row", "args": ["--limit", "1"], "stdin": "id,amount\n1,10\n"}, {"name": "bad_amount", "args": [], "stdin": "id,amount\n1,oops\n"} ] 

Three cases are enough for the first pass. Add a case only when you can name the behavior it guards.

# goldens.py — capture observable behavior of one entry point. import hashlib import json import subprocess import sys def run(entry, args, stdin, timeout=30): proc = subprocess.run([sys.executable, entry, *args], input=stdin, capture_output=True, text=True, timeout=timeout) return {"rc": proc.returncode, "stdout": proc.stdout, "stderr_sha": hashlib.sha256(proc.stderr.encode()).hexdigest()[:12]} if __name__ == "__main__": entry, label = sys.argv[1], sys.argv[2] cases = json.load(open("cases.json")) data = {c["name"]: run(entry, c["args"], c["stdin"]) for c in cases} with open(f"goldens.{label}.json", "w") as fh: json.dump(data, fh, indent=2, sort_keys=True) print(f"{label}: {len(cases)} cases on {entry}") 

# compare.py — exit 1 on any behavioral drift. import json import sys before, after = (json.load(open(p)) for p in sys.argv[1:3]) drift = sorted(k for k in before if before[k] != after.get(k)) print("MATCH" if not drift else "DRIFT " + ",".join(drift)) sys.exit(1 if drift else 0) 

Two commands give you the baseline and the gate.

python goldens.py src/report.py before # ... make exactly one change ... python goldens.py src/report.py after && python compare.py goldens.before.json goldens.after.json 

Hash stderr instead of storing it. Stack traces carry line numbers and paths that change on every refactor.

Step 3: Let a free model draft cases, then audit them

Here is where free model access earns its place. Drafting a case list from an unfamiliar module is tedious and low-risk. It is also easy to check, because the output is data, not a diff.

A workable loop:

1. Paste the entry point signature and one existing fixture into the model.
2. Ask for cases.json only. Reject anything that touches source files.
3. Read all three to ten cases yourself. Delete any case you cannot justify.
4. Run the audited list and store the baseline.

Constrain the output format. Declarative data is reviewable in seconds, and it cannot smuggle in a change you did not ask for.

MonkeyCode is an open-source project with free model access and a free server option. The operator's published free allowance at the time of writing is 10,000,000 tokens. It is useful here for two narrow jobs: drafting case lists, and hosting a disposable box so the baseline runs in a clean environment instead of your laptop.

Treat every quota number as perishable. Verify the current terms on the project page before you design a workflow around them.

Step 4: Mutate, or your goldens guard nothing

This is the step that separates a gate from a ritual. Break the code on purpose. Confirm at least one golden notices.

# mutate.py — prove a golden can fail. Usage: mutate.py <file> <old> <new> <entry> import json import pathlib import subprocess import sys target, old, new, entry = pathlib.Path(sys.argv[1]), sys.argv[2], sys.argv[3], sys.argv[4] original = target.read_text() if old not in original: sys.exit(f"mutation site {old!r} not found in {target}") try: target.write_text(original.replace(old, new, 1)) subprocess.run([sys.executable, "goldens.py", entry, "mutant"], check=True) finally: target.write_text(original) before = json.load(open("goldens.before.json")) mutant = json.load(open("goldens.mutant.json")) caught = sorted(k for k in before if before[k] != mutant[k]) print("CAUGHT " + ",".join(caught) if caught else "SILENT — this golden guards nothing") sys.exit(0 if caught else 1) 

python mutate.py src/parse.py "float(" "int(" src/report.py # CAUGHT one_good_row,bad_amount -> goldens are real # SILENT -> rewrite the case list 

The finally block restores the file even when the run crashes. Do not skip it. A half-mutated source tree is how refactors become incidents.

Pick one-token mutations. float( to int(. >= to >. sorted to list. Each should trip a specific, named case.

Record the mutation result in the pull request. "Three cases, three mutations caught" is evidence. "Tests pass" is not.

Step 5: One change, then compare

Now make the smallest safe change. Rerun the goldens. Compare the files.

python goldens.py src/report.py after python compare.py goldens.before.json goldens.after.json 

Drift is information, not failure. Read the diff. Decide whether the change in output was intended.

If you cannot explain every drifted case in one sentence, revert. The change is too large for the evidence you have.

Run fingerprint.py again after a dependency bump. A refactor that also upgrades a package is two changes wearing one commit message.

Decision table

Limitations and who should not use this

This workflow assumes deterministic output. If your module prints timestamps, IDs, or latencies, pin them first or hash only stable fields.

Three cases prove three behaviors. They do not prove correctness. This is a safety net for one small change, not a test suite.

Do not run this on code you cannot legally or safely execute on a third-party machine. Secrets in env vars are recorded as presence only, but the code itself still runs somewhere.

Do not build CI policy on a free tier without confirming current limits. Free allowances change. A workflow that silently stops when a quota moves is worse than no workflow.

Skip this if your team has no reviewer for the case list. Unreviewed model-drafted cases reproduce unexamined assumptions, which is exactly what you are trying to escape.

Checklist before you merge

• fingerprint.json matches between the two machines you care about.
• cases.json was reviewed line by line by a human.
• Every case survived at least one mutation.
• The diff touches one behavior, and you can name it.
• compare.py prints MATCH, or you can explain each drift.

If you want to try the free-box part, MonkeyCode's free model access and free server option are the two pieces used above. Run fingerprint.py there first, before you trust anything else.