Tests can pass while the public surface quietly changes. A rename, a new required argument, a fresh exception type — green build, broken callers. The fix is a gate, not more tests.

This is the second half of a messy-repo refactor. First you pin behavior with characterization tests. Then you allow exactly one small change. The gate below makes "small" and "safe" measurable instead of aspirational.

Why tests alone miss the smallest safe change

Characterization tests freeze what the code does for the inputs you recorded. They say nothing about what the module exposes to callers you never imported. A diff can be behavior-identical and API-hostile at the same time.

Two checks close that gap. A public surface manifest, dumped before and after. A diff budget, enforced with the same script. Both are cheap and both fail loudly.

Step 1 — Cut a baseline worktree

Never refactor on a dirty tree. Create a throwaway worktree at the commit you trust.

git worktree add ../repo-base HEAD cd ../repo-base python -m pytest -q tests/characterization | tail -3 python -m pytest --collect-only -q tests/characterization | tail -1

You now have an untouched tree, a pass count, and a collected-test count. Save the collected count. A refactor that "fixes" tests by deleting them will drop that number, and you want to see it drop.

Step 2 — Dump the public surface manifest

This script records exported functions, their argument names, required-argument counts, decorators, and raised exception types. Classes get bases and public methods. Everything private is ignored by convention.

Requires Python 3.9+ for ast.unparse. Save as tools/surface.py.

import ast import json import sys from pathlib import Path def surface_of(path: Path) -> dict: tree = ast.parse(path.read_text(encoding="utf-8")) public = {} for node in tree.body: if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): if node.name.startswith("_"): continue args = node.args public[node.name] = { "kind": "function", "args": [a.arg for a in args.args], "kwonly": [a.arg for a, d in zip(args.kwonlyargs, args.kw_defaults) if d is None], "required": len(args.args) - len(args.defaults), "decorators": [ast.unparse(d) for d in node.decorator_list], "raises": sorted( {ast.unparse(h.type) if h.type else "bare" for h in ast.walk(node) if isinstance(h, ast.Raise)} ), } elif isinstance(node, ast.ClassDef) and not node.name.startswith("_"): public[node.name] = { "kind": "class", "bases": [ast.unparse(b) for b in node.bases], "methods": sorted( n.name for n in node.body if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) and not n.name.startswith("_") ), } return {"module": path.stem, "public": public} def manifest(root: Path) -> dict: return { str(p.relative_to(root)): surface_of(p) for p in sorted(root.rglob("*.py")) if "__pycache__" not in p.parts and not p.name.startswith("surface") } if __name__ == "__main__": root = Path(sys.argv[1]) out = Path(sys.argv[2]) out.write_text(json.dumps(manifest(root), indent=2, sort_keys=True), encoding="utf-8") print(f"wrote {out}")

Run it in the baseline tree. Commit the output or stash it somewhere the candidate tree can read.

cd ../repo-base python tools/surface.py src surface.base.json

Step 3 — Pin behavior with three kinds of test

Keep this boring and fast. Characterization tests are a recording, not a design document.

  • Golden output for the main entry point, byte-for-byte.
  • The exception types and messages the code raises today.
  • One test per side effect you can observe without a network call.

Run them against the baseline until they pass. Then stop. If a test needs a rewrite to pass, the test is wrong, not the code.

Step 4 — Ask for the smallest patch

Now get a candidate diff. Free model access works fine here because the model proposes and the gate disposes. You are not asking for correctness. You are asking for one narrow change you will judge mechanically.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access and free server option referenced here are operator-supplied availability claims; verify current limits in the product itself before relying on them.

A prompt shape that produces reviewable diffs:

Goal: extract the date parsing in report.py into parse_date(). Constraints: - Do not change any public name, signature, or raised exception. - Touch at most 2 files. - Keep the diff under 60 changed lines. - Return only a unified diff. No prose.

Apply the patch to a second worktree, not to your working copy.

git worktree add ../repo-cand HEAD cd ../repo-cand git apply --check candidate.diff && git apply candidate.diff

If you are running the model on the free server option, keep the repo slice small and exclude secrets. A free tier is a convenience, not a trust boundary.

Step 5 — Run the gate

The gate compares two manifests and the diff size. Save as tools/gate.py.

import json import subprocess import sys from pathlib import Path BUDGET_LINES = 60 BUDGET_FILES = 2 def load(p: str) -> dict: return json.loads(Path(p).read_text(encoding="utf-8")) def surface_reasons(base: dict, cand: dict) -> list: reasons = [] for mod, info in base.items(): if mod not in cand: reasons.append(f"module removed: {mod}") continue for name, sig in info["public"].items(): new = cand[mod]["public"].get(name) if new is None: reasons.append(f"public symbol removed: {mod}::{name}") continue if sig["kind"] != new["kind"]: reasons.append(f"kind changed: {mod}::{name}") continue if sig["kind"] == "function": if new["args"][: len(sig["args"])] != sig["args"]: reasons.append(f"arg order changed: {mod}::{name}") if new["required"] > sig["required"]: reasons.append(f"new required parameter: {mod}::{name}") added = set(new["raises"]) - set(sig["raises"]) if added: reasons.append(f"new exception type: {mod}::{name} {sorted(added)}") else: lost = set(sig["methods"]) - set(new["methods"]) if lost: reasons.append(f"method removed: {mod}::{name} {sorted(lost)}") return reasons def diff_budget() -> list: out = subprocess.run( ["git", "diff", "--numstat", "HEAD", "--", "."], capture_output=True, text=True, check=True, ).stdout lines = files = 0 for row in out.splitlines(): add, rem, _ = row.split("\t") if not add.isdigit() or not rem.isdigit(): return ["binary file in diff"] lines += int(add) + int(rem) files += 1 reasons = [] if lines > BUDGET_LINES: reasons.append(f"diff too large: {lines} > {BUDGET_LINES} lines") if files > BUDGET_FILES: reasons.append(f"too many files: {files} > {BUDGET_FILES}") return reasons if __name__ == "__main__": reasons = surface_reasons(load("surface.base.json"), load("surface.candidate.json")) reasons += diff_budget() if reasons: print("REJECT") for r in reasons: print(f" - {r}") sys.exit(1) print("ACCEPT")

Run it in the candidate worktree, after generating the candidate manifest and the tests.

cd ../repo-cand python tools/surface.py src surface.candidate.json python -m pytest -q tests/characterization | tail -1 python tools/gate.py

Step 6 — Read the rejection reasons as data

Three rejections are common and each means something specific.

Reason Usual cause Response
new required parameter Model added a config flag Ask for a default value
new exception type Error handling drifted Pin the old exception in a test
diff too large Task was too broad Split into two refactors

The write-only part is important. Append each rejection to a JSONL log with the prompt hash. After ten attempts you can see which prompt shapes produce oversized diffs, and fix the prompt rather than the model.

When this gate is the wrong tool

Some repos should skip it.

  • Dynamic re-exports through __getattr__ or globals(): the manifest will be empty or wrong.
  • Deliberate public API changes: the gate cannot tell a planned break from an accidental one. Bump the version and update the baseline instead.
  • Generated code: regenerate first, then take the baseline. Otherwise every run rejects.
  • C extensions or code that only exists after import hooks: static parsing misses it.
  • Teams without a golden-output corpus: write those tests first. The gate protects a surface, not a behavior.

Also skip this if the change is really a feature. A feature needs new behavior, and a behavior lock will fight you the whole way.

What you get

One refactor per branch. A baseline manifest you can diff by eye. A hard cap on diff size. A rejection log that improves your prompts over time.

That is the whole loop: record behavior, lock the surface, propose one change, gate it. The model writes the patch. Your harness decides whether the patch ever touches main.

If you want to run steps 4 through 6 without a paid seat, the free model access and free server option are enough to try the loop end to end.