A characterization test that passes on old and new code proves nothing.
It only proves the code changed in ways your assertion cannot see.
My rule now: every characterization test must go red on a deliberate behavior change first.
Here is the gate I run before touching a messy repo.
It records real behavior, freezes it into tests, then probes those tests with small forced changes.
A test that survives a probe gets fixed or deleted. Never trusted.
The failure mode: green tests, silent behavior change
Picture a function that totals report rows.
Your test asserts one number: the grand total.
# tests/test_rows.py def test_total(): assert total_for("fixtures/report.csv") == 1842.50 Now refactor the row ordering, or swap rounding from half-even to half-up.
The total may shift by a cent, or stay identical while row order changes.
Either way your gate is blind to the change you just made.
The fix is not more assertions written in advance.
You do not yet know which behaviors matter.
So measure sensitivity instead: inject a known change, watch the test react.
Step 1 — Record real behavior before you reason about it
Do not invent fixtures from reading source code.
Run the code and capture arguments plus outputs.
# trace_capture.py import functools, json, pathlib TRACE = pathlib.Path("traces/rows.jsonl") def traced(fn): @functools.wraps(fn) def wrapper(*args, **kwargs): out = fn(*args, **kwargs) record = {"fn": fn.__name__, "args": list(args), "kwargs": kwargs, "out_repr": repr(out)} with TRACE.open("a") as fh: fh.write(json.dumps(record) + "\n") return out return wrapper Wrap the entry point, run the existing test suite or a scripted job, then unwrap.
Twenty to fifty traces usually beat two hundred hand-written cases.
Two honest caveats. First, repr for floats is stable enough for comparison but ugly for reading.
Second, scrub anything sensitive in args before the file leaves your machine.
Step 2 — Freeze traces into characterization tests
Convert each trace line into one parameterized test.
The test asserts today's behavior, bugs included.
# tests/test_rows_frozen.py import json, pathlib, pytest from report.rows import total_for LINES = pathlib.Path("traces/rows.jsonl").read_text().splitlines() CASES = [json.loads(line) for line in LINES if line.strip()] @pytest.mark.parametrize("case", CASES, ids=lambda c: f"trace-{c['args'][0]}") def test_frozen_behavior(case): got = total_for(*case["args"], **case["kwargs"]) assert repr(got) == case["out_repr"] Run it once. Every case should pass, because you recorded the same code.
If a case fails here, your trace is stale or nondeterministic.
Stop and fix that before any refactor. A flaky gate is not a gate.
Step 3 — Write probes, not more assertions
A probe is a tiny patch that must change observable behavior.
Keep each probe to one line where possible.
| Probe class | Example change | What it should catch |
|---|---|---|
| Constant shift | + 0.01 on a total | Numeric blindness |
| Boundary swap | >= becomes > | Edge cases at thresholds |
| Ordering change | sorted(...) removed | Tests asserting only aggregates |
| Error path change | raise becomes return None | Missing failure assertions |
| Format change | str(x) becomes f"{x:.2f}" | Type and shape blindness |
Store probes as data, not as scattered branches in your test file.
# probes.py PROBES = [ {"name": "half_up_drift", "file": "report/rows.py", "old": "return round(total, 2)", "new": "return round(total, 2) + 0.01", "test": "tests/test_rows_frozen.py"}, {"name": "threshold_swap", "file": "report/rows.py", "old": "if qty >= 10:", "new": "if qty > 10:", "test": "tests/test_rows_frozen.py"}, ] Step 4 — Run the sensitivity harness
Never patch your working tree.
Copy the repo, apply one probe, run one test file, record the outcome.
# sensitivity.py import pathlib, shutil, subprocess, sys, tempfile from probes import PROBES IGNORE = shutil.ignore_patterns(".git", "__pycache__", "*.pyc", ".venv") def run_test(repo: pathlib.Path, test: str): cmd = [sys.executable, "-m", "pytest", test, "-x", "-q", "-p", "no:cacheprovider"] return subprocess.run(cmd, cwd=repo, capture_output=True, text=True) def probe(repo: pathlib.Path, p: dict) -> str: with tempfile.TemporaryDirectory() as tmp: work = pathlib.Path(tmp) / "repo" shutil.copytree(repo, work, ignore=IGNORE) target = work / p["file"] src = target.read_text() if p["old"] not in src: return "INVALID: anchor not found" target.write_text(src.replace(p["old"], p["new"], 1)) r = run_test(work, p["test"]) if r.returncode == 0: return "GREEN: test is blind to this change" if r.returncode == 2: return "INVALID: collection or usage error" return "RED: expected" if __name__ == "__main__": root = pathlib.Path(".").resolve() for p in PROBES: print(f"{p['name']:20s} {probe(root, p)}") python sensitivity.py Exit code 2 matters here. A broken patch or import error looks like a failure but proves nothing.
Treat it as INVALID, not as success. This one distinction prevents most false confidence.
Step 5 — Read the report with a decision table
Do not just count red results. Classify each test.
| Observed | Meaning | Action |
|---|---|---|
| RED on its intended probe | Test is sensitive | Keep as gate |
| GREEN on every probe | Blind test | Add an assertion or delete it |
| RED on unrelated probes | Over-fitted test | Loosen brittle exact-match assertions |
| INVALID | Probe or fixture broken | Fix before trusting anything |
| RED before the patch | Stale trace | Re-record, then re-run |
Keep a small table in the PR description. Fill it honestly.
| Test file | Probes run | RED | GREEN | INVALID |
|---|---|---|---|---|
| tests/test_rows_frozen.py | 2 | 1 | 1 | 0 |
One green cell is a finding, not a delay. It tells you exactly where the refactor can go wrong.
Step 6 — Make the smallest safe change
Only now edit production code. One behavior per commit.
Re-run the frozen tests and the sensitivity harness after each commit.
git diff --stat main...HEAD python -m pytest tests/test_rows_frozen.py -q python sensitivity.py If a frozen test goes red during the refactor, stop. Either the refactor changed behavior on purpose or it broke something.
Both cases need a written decision, not a re-recorded trace. Re-recording mid-refactor destroys the gate.
Where a free model and a free server genuinely help
Two parts of this workflow are tedious and untrusted at the same time.
Probe authoring is tedious: reading a diff to find one-line mutations takes minutes per probe.
Probe execution is untrusted: model-drafted patches run real code on a real filesystem.
MonkeyCode's free model access is a reasonable source for probe candidates and extra fixture cases.
You paste a function and ask for behavior-changing one-liners, then review each one.
MonkeyCode's free server option is a reasonable place to run the harness, because a throwaway machine keeps experimental patches off your laptop.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Keep one rule: the model proposes, the harness decides. A probe is credible only when it turns a real test red.
Do not claim the server isolates anything you have not checked yourself. Verify the environment before running private code there.
Limitations and who should skip this
This gate needs a deterministic entry point. Code that hits a live database, a clock, or a third-party API will produce unstable traces.
Freeze those dependencies first, or skip the approach for that module.
Skip it too if the code is safety-critical and freezing bugs needs product sign-off. Characterization tests preserve current behavior, including behavior nobody wants.
Also skip it for one-file scripts. Two hours of harness work for a thirty-line file is a bad trade.
A useful refactor gate is one that has been seen failing.
Break your tests on purpose, in a copy, before you trust them.
If you want to try the harness without touching your machine, the free server option is a fair place to start.
