Same-Model Self-Review Is a Blind Spot: A Cross-Model Review Protocol for Agent PRs

If the model that wrote the pull request also reviews it, you have not added a reviewer. You have added an echo.

That is the uncomfortable part of most agent-PR workflows I see: author model and reviewer model are the same endpoint, prompted differently. The review comes back fluent, confident, and largely in agreement with the diff it was just handed. Nothing in the output is falsifiable, so nothing gets caught.

The fix is structural, not rhetorical. Build a packet instead of pasting a diff. Point a different model at it. Gate the merge on artifacts a human can check line by line.

Why same-model review fails quietly

A reviewer earns its keep by disagreeing in a checkable way. Same-model review rarely does that.

• Correlated priors. Author and reviewer share training data, style habits, and failure patterns. If the author never considered the empty-list case, the reviewer is less likely to invent it.
• Fluency as false evidence. A tidy summary reads like verification. It is not.
• No line citations. An objection that cannot point at file:line cannot be tested, so it can be neither accepted nor rejected.
• Symmetric seam errors. The comment says 30s, the code says 60. The same model in both roles tends to repeat whichever it wrote first.

A hypothesis worth testing, not asserting: cross-model review produces more falsifiable objections per PR than self-review at the same wall-clock cost. The measurement plan at the end tells you whether that holds in your repo.

Give the reviewer a packet, not a diff

A raw diff strips the context a reviewer needs: which files exist, which are tests, whether assertions were deleted, whether CI permissions moved. Humans rebuild that by scrolling. Models do not rebuild it — they guess.

So build the packet first.

#!/usr/bin/env python3 """review_packet.py - build an audit packet for an agent-generated PR. Stdlib only; run it inside a checkout of the PR branch. python review_packet.py --base origin/main > packet.md python review_packet.py --base origin/main --strict; echo "exit=$?" """ from __future__ import annotations import argparse import re import subprocess import sys from dataclasses import dataclass, field TEST_RE = re.compile( r"(^|/)(tests?|__tests__)/|(^|/)test_[^/]*\.py$|_test\.(py|go)$|\.(test|spec)\.[cm]?[jt]sx?$" ) RISKY = [ (re.compile(r"(^|/)(Dockerfile|docker-compose\.ya?ml)$"), "container/build"), (re.compile(r"\.(tf|tfvars)$"), "infrastructure"), (re.compile(r"(^|/)migrations?/"), "schema migration"), (re.compile(r"(^|/)\.github/workflows/"), "CI permissions"), (re.compile(r"(^|/)(settings|config)\.(py|ya?ml|json|toml)$"), "configuration"), ] ASSERT_RE = re.compile(r"\b(assert|expect|should)\b") def git(*args: str) -> str: return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout @dataclass class Packet: base: str files: list[tuple[str, str]] # (status, path)  stats: dict[str, tuple[int, int]] # path -> (added, removed)  tests: list[str] = field(default_factory=list) risky: list[tuple[str, str]] = field(default_factory=list) weakened: list[tuple[str, int]] = field(default_factory=list) def build(base: str) -> Packet: files: list[tuple[str, str]] = [] for line in git("diff", "--name-status", f"{base}...HEAD").splitlines(): parts = line.split("\t") if len(parts) >= 2: files.append((parts[0][0], parts[-1])) stats: dict[str, tuple[int, int]] = {} for line in git("diff", "--numstat", f"{base}...HEAD").splitlines(): added, removed, path = line.split("\t", 2) stats[path] = (int(added) if added.isdigit() else 0, int(removed) if removed.isdigit() else 0) pkt = Packet(base=base, files=files, stats=stats) for _status, path in files: if TEST_RE.search(path): pkt.tests.append(path) removed_asserts = 0 for line in git("diff", "-U0", f"{base}...HEAD", "--", path).splitlines(): if line.startswith("-") and not line.startswith("---") and ASSERT_RE.search(line): removed_asserts += 1 if removed_asserts: pkt.weakened.append((path, removed_asserts)) for pattern, label in RISKY: if pattern.search(path): pkt.risky.append((label, path)) break return pkt def render(pkt: Packet) -> str: added = sum(a for a, _ in pkt.stats.values()) removed = sum(r for _, r in pkt.stats.values()) out = [ "# Review packet", "", f"- base: `{pkt.base}`", f"- files changed: {len(pkt.files)} (+{added} / -{removed} lines)", f"- test files touched: {len(pkt.tests)}", f"- risky surfaces touched: {len(pkt.risky)}", "", "## Changed files", "", "| status | path | + | - |", "|---|---|---|---|", ] for status, path in pkt.files: a, r = pkt.stats.get(path, (0, 0)) out.append(f"| {status} | `{path}` | {a} | {r} |") out += ["", "## Risky surfaces", ""] out += [f"- {label}: `{path}`" for label, path in pkt.risky] or ["- none detected"] out += ["", "## Test delta", ""] if not pkt.tests: out.append("- **no test file touched**") for path in pkt.tests: out.append(f"- `{path}`") for path, n in pkt.weakened: out.append(f"- **{n} assertion line(s) removed in `{path}`**") return "\n".join(out) + "\n" def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--base", default="origin/main") ap.add_argument("--strict", action="store_true", help="exit non-zero on gate failures") args = ap.parse_args() pkt = build(args.base) sys.stdout.write(render(pkt)) gates = [] if pkt.risky and not pkt.tests: gates.append("risky surface touched with no test change") if pkt.weakened: gates.append("assertions removed from test files") for g in gates: print(f"GATE FAIL: {g}", file=sys.stderr) return 2 if (args.strict and gates) else 0 if __name__ == "__main__": raise SystemExit(main()) 

Run it on the PR branch:

git fetch origin main git switch pr-1234 python review_packet.py --base origin/main --strict > packet.md 

Three things the packet buys you that a pasted diff does not:

1. Assertion deletions are counted, not summarised. A PR that turns assert x == 3 into assert x shows up as a number you can reject.
2. Risky surfaces are named. CI workflow, Dockerfile, migration, config default. These are the files where agent PRs cause the most expensive class of incident.
3. The gate is machine-checkable. --strict exits non-zero, so a human decision is required before merge instead of being implied by momentum.

Caveat on the script: I have not run it against your repository. Renames produce awkward numstat paths, binary files report - instead of a count, and submodules are ignored. Treat the first run as a test of the script itself.

Trust, test, revert

Once the packet exists, every PR gets one of three dispositions. "Read harder" is not a disposition.

Trust only what the packet proves mechanically. Everything else gets a test or a revert — no exceptions for a polite PR description.

The reviewer contract

Give the second model a contract, not an invitation. Constraints are what make its output auditable.

You are reviewing a pull request you did not write. Inputs: packet.md, the full diff, and the output of the project's test command. 1. Do not summarise the diff. Summaries hide uncertainty. 2. Every objection cites file:line from the diff you were given. 3. Split each objection into CLAIM (what the code asserts) and EVIDENCE (what proves it). 4. If a CLAIM has no test, mark it "unproven" and name the cheapest test that would settle it. 5. If you cannot cite a line, output "insufficient context" and stop. 6. Output at most five objections, ordered by blast radius. 7. Never approve. You produce objections; a human produces the verdict. 

Rule 7 matters most. A reviewer that emits LGTM has converted a human decision into a rubber stamp.

Where to run it

Two practical reasons to keep review off your own machine: you should not execute agent-authored code on the laptop holding your SSH keys, and packet building plus a test run needs a checkout and a toolchain, which is exactly what a disposable box provides.

MonkeyCode is a fit for that shape of work. It offers free model access and a free server option, so the author model and the reviewer model can differ without provisioning a second paid endpoint, and the packet build can run on a machine you are willing to throw away. The operator states a free token allowance (10M tokens at the time of writing) alongside the free server option; quotas, model availability, and server specs are precisely the kind of terms that change without notice, so confirm the current ones on the project's site before wiring either into CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I would not do: paste a private repository into any third-party model, free or not, without your organisation's approval. Use it on OSS repos, synthetic reproductions, or code already cleared for external processing.

Who should not use this

• Teams under data-residency or regulated-data constraints. The review step sends code off-box.
• Anyone who needs an SLA for the review step. A free tier is a convenience, not a guarantee; keep a fallback path.
• Repos whose test command is not trustworthy yet. A packet plus a broken suite produces confident nonsense.
• Security-sensitive changes. Cross-model review reduces correlated blind spots; it does not replace threat modelling or a human with context.

Prove it helps in twenty PRs

Keep a log and compute. Do not vibe-check.

Two numbers decide it: confirmed-valid objections per reviewer-minute, and the false-objection rate. If cross-model review does not beat self-review on the first metric across twenty PRs, drop it — the second model is then costing you latency for nothing.

Pair this with planted bugs, the way you would calibrate any reviewer: seed three known defects into a throwaway branch and record which reviewer finds them and which reviewer congratulates you. If both miss all three, your protocol is not the bottleneck; your test suite is.

Takeaway

Self-review by the author model is the cheapest thing to set up and the easiest thing to mistake for coverage. Change one variable: make the reviewer a different model, hand it a packet with numbered evidence, and force the verdict through a table with three rows. Everything else — which endpoint, which box, which quota — is replaceable.