A review bot that only comments is easy to ignore. A review bot that can block a merge, or apply a patch, is a production dependency that nobody is paged for. Most teams wire one up in a quiet sprint and judge it by reading a dozen suggestions on friendly diffs, which is exactly the sample where a gate looks smart.
The failures that matter live outside that sample: a diff that hides instructions inside a comment, a vendored file that should have been skipped, a suggestion that applies cleanly and then breaks an invariant. What follows is an illustrative test plan for that class of failure. Nothing below is a report of a run, no outcomes or timings are claimed, and the code is written to be runnable rather than to have been run.
Start from the decision, not from the model
Write down the gate's possible outcomes before choosing a backend, because each outcome needs a different kind of evidence behind it.
| Outcome | Typical trigger | What you must be able to verify afterward |
|---|---|---|
| comment-only | default when nothing is certain | the exact payload sent and returned is stored |
| block | a deterministic rule hit (secret pattern, protected path) | the rule id and offending line, with no model call needed |
| apply patch | output parses to a unified diff and applies to a clean tree | dry-run apply succeeded and a test subset passed |
| escalate | size, time, or budget cap exceeded | a reason code plus a link to the stored artifact |
The block path should not depend on the model
Secret detection, protected paths, and diff size caps are rules rather than judgments, so enforce them before any text leaves your runner. If the only thing standing between a leaked token and the default branch is a model's opinion, the gate is decorative. Save the model for the parts that genuinely need reading comprehension, such as summarizing intent or spotting a missing null check.
Seven fixtures to write before you trust the gate
Each fixture is a small diff plus the outcome you expect from it. Run them the way you run contract tests: no sampling, no "this looks reasonable to me".
- Secret in an added line. Expect
blockfrom a rule, and verify the secret string never reaches logs, telemetry, or the model prompt. - Instructions hidden in a diff comment. Expect your classification to stay unchanged, and verify that text inside the diff cannot promote a change to
apply patch. - Vendored or generated files. Expect
skip, and verify the path policy comes from repo config instead of the model's guess aboutdist/. - Oversized diff and a binary blob. Expect
escalate, and verify the size cap fires before upload rather than after the response returns. - Reformat-only diff. Expect
comment-only, and verify the gate does not spend a review cycle pretending whitespace is a bug. - Patch that applies but breaks a test. Expect no merge, and verify the rollback path is actually exercised instead of assumed.
- Duplicate webhook delivery. Expect one effect, and verify the idempotency key is derived from the delivery rather than from a timestamp.
Record the observed outcome next to the expected one in a JSONL file. That file is the artifact you reach for when someone asks whether the gate is trustworthy, and it survives model upgrades that quietly change tone without changing behavior.
A harness that tests the gate, not the model
Most of the gate's logic is deterministic, so test it with canned responses first. The illustrative pytest module below feeds fixed payloads through your decision function and asserts the contract; it has not been executed here and needs adapting to your own function names.
# test_gate_contract.py (illustrative, not executed) import json import pytest from gate import decide # decide(diff_text, meta) -> {'outcome': ..., 'rule': ...} FIXTURES = [ ('fixtures/secret_added.diff', {'paths': ['src/app.py']}, 'block'), ('fixtures/injection_comment.diff', {'paths': ['src/app.py']}, 'comment-only'), ('fixtures/vendored_only.diff', {'paths': ['dist/bundle.js']}, 'skip'), ('fixtures/huge_blob.diff', {'bytes': 40_000_000}, 'escalate'), ('fixtures/whitespace_only.diff', {'paths': ['src/app.py']}, 'comment-only'), ] @pytest.mark.parametrize('path,meta,expected', FIXTURES) def test_outcome_contract(path, meta, expected): diff_text = open(path, encoding='utf-8').read() result = decide(diff_text, meta) assert result['outcome'] == expected, json.dumps(result) def test_secret_never_echoed(): diff_text = open('fixtures/secret_added.diff', encoding='utf-8').read() result = decide(diff_text, {'paths': ['src/app.py']}) assert 'AKIA' not in json.dumps(result) Two properties matter more than the individual cases: decide should be a pure function of its inputs, and redaction should run before anything is serialized. If redaction lives inside the model client, your contract test cannot observe it, and neither can you.
Then replay the same fixtures against a real backend
Once the deterministic half is green, replay the fixtures with the model enabled and diff the two sets of outcomes. MonkeyCode is relevant at this step because, according to the operator, it offers free model access and a free server option, which makes a fixture replay cheap enough to keep in a loop rather than a quarterly exercise.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I have not independently measured quotas, throughput, or uptime for those free options, and I am not going to invent numbers for them. Treat that availability as a claim to confirm inside your own account before it becomes part of a workflow you depend on, and keep a local fallback for the days it does not hold.
# illustrative replay: one JSONL line per fixture, then diff the two runs mkdir -p out for f in fixtures/*.diff; do gate review --diff "$f" --mode rules-only --json >> out/rules.jsonl gate review --diff "$f" --mode model-assisted --json >> out/model.jsonl done diff <(jq -S . out/rules.jsonl) <(jq -S . out/model.jsonl) || true Read that diff by hand the first few times. A model run that turns block into comment-only is exactly the finding you were looking for, and it stays invisible if you only read the model's prose summaries.
Where to run the replay
| Environment | Fits when | Watch for |
|---|---|---|
| Laptop | you are still writing fixtures | fixture paths drifting away from CI paths |
| CI, per pull request | the gate can block merges | cost and latency on every run, including docs-only changes |
| Free server as webhook receiver | you want the gate to run outside CI entirely | delivery retries, timeout behavior, cold starts after idle |
If you do put the receiver on a free server option, measure the things that break receivers rather than the things vendors prefer to advertise. Send the same webhook twice and check for one effect, kill the process mid-request and check for a duplicate comment, and time the first request after an idle period so any latency number you publish is honest.
What this plan cannot prove
- Model quality. Fixtures show that bad outcomes are blocked, not that good suggestions are useful; keep a human sample for that question.
- A security guarantee. A passing gate is defense in depth, and the rules you wrote are the parts you can actually audit later.
- Vendor stability. Free tiers change terms and limits, so re-run the replay on a schedule you choose and stamp the results with a date.
- Test semantics. The gate can verify that a subset passed, but only your suite decides whether that subset means anything.
Who should skip this
If the repository has no path policy, no test subset that finishes in minutes, and no owner willing to be paged when the gate misblocks, then the model is not your bottleneck and this plan will mostly produce paperwork. Teams that need a formal audit trail should also confirm the vendor's terms and retention match their obligations before the first webhook lands, not after.
If you build the fixture set, the useful signal is which case fails first. That ordering tells you more about the gate than any number of suggestions you read by hand.
