Let a Model-Backed Check Earn Its Blocking Rights: A Three-Rung CI Gate Ladder

A model-backed check should not block a merge on day one. Give it three rungs, a ledger, and a promotion rule, and it will eventually tell you whether it deserves that authority — with data instead of opinions.

Teams usually pick one of two bad defaults. Either the model runs as a comment bot nobody reads, or it gets wired straight into a required check and the next month disappears into flaky red Xs. This post is the middle path: a CI/CD integration where hooks, fixtures, and flake control decide when a model check graduates from advisory to blocking.

The ladder at a glance

Each rung has a different job, a different cost profile, and a different exit criterion. You move up only when the rung below is boring.

Step 1 — Put the first reviewer on the developer's machine

Rung 1 exists so a bad diff dies in a terminal, not in a queue. It is advisory, it is fast, and it never blocks anything.

A pre-push hook is the cheapest place to try it. This one is deliberately dumb: it reviews the diff against the merge base, caps the payload, and treats an oversized diff as skipped, not passed.

#!/usr/bin/env bash # .githooks/pre-push — rung 1, advisory only set -euo pipefail BASE="${BASE_REF:-origin/main}" DIFF=$(git diff --unified=0 "$BASE"...HEAD -- . ':!vendor' ':!*.lock') if [ -z "$DIFF" ]; then exit 0; fi if [ "${#DIFF}" -gt 60000 ]; then echo "advisory review skipped: diff too large (skip is not a pass)" exit 0 fi payload=$(jq -n \ --arg m "$REVIEW_MODEL" \ --arg s "$(cat prompts/review.md)" \ --arg d "$DIFF" \ '{model:$m, temperature:0, messages:[{role:"system",content:$s},{role:"user",content:$d}]}') curl -sS --max-time 25 "$ADVISOR_BASE/chat/completions" \ -H "Authorization: Bearer $ADVISOR_API_KEY" \ -H "Content-Type: application/json" \ -d "$payload" | jq -r '.choices[0].message.content' 

Enable it with git config core.hooksPath .githooks. Keep the key in your shell environment, never in the repo, and make sure the hook exits 0 on network failure — a timeout on a train is not a code review.

This is where free access actually matters. MonkeyCode offers free model access (the operator describes the free tier as roughly 10 million tokens) and a free server option, which makes rung 1 affordable to leave switched on for an entire team while you collect evidence. Check the project's current terms before you build a budget assumption on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 2 — Write a verdict ledger, or you have no history

A model check with no ledger is folklore. Every run writes exactly one line, including failures, and the line carries a replay key so you can reproduce it later.

{"ts":"2026-11-09T09:14:03Z","sha":"a91f2c7","merge_base":"4e0b118","prompt_sha":"7d3a0f9","model":"pinned-at-r2","status":"completed","verdict":"fail","findings":2,"duration_ms":4180,"ci_url":"..."} 

Three fields do the heavy lifting. prompt_sha pins the instructions, merge_base pins what the diff was measured against, and status distinguishes completed from timeout and skipped. If any of the three is missing, the run cannot count toward promotion.

Commit the ledger writer as part of the CI job, not as a separate service. One printf of JSON to an artifact is enough at this stage; you can move it to object storage when it grows.

Step 3 — Make promotion a data question

Rung 3 is the only rung allowed to block. It should be granted by a script, not by someone's good impression of last week's results.

#!/usr/bin/env python3 """Promote or hold a model-backed gate based on its own ledger.""" import json, pathlib, sys MIN_STREAK = 30 MIN_GOLD = 20 MIN_AGREEMENT = 0.90 def lines(path): for raw in pathlib.Path(path).read_text().splitlines(): if raw.strip(): yield json.loads(raw) def hold(reason): print(f"HOLD: {reason}") return 1 def main(ledger_path, gold_path): runs = list(lines(ledger_path)) if not runs: return hold("empty ledger") keys = {(r["prompt_sha"], r["model"]) for r in runs} if len(keys) != 1: return hold(f"{len(keys)} prompt/model pairs; pin exactly one") streak = 0 for run in reversed(runs): if run["status"] != "completed": break streak += 1 if streak < MIN_STREAK: return hold(f"streak {streak} < {MIN_STREAK}") replayable = {(r["sha"], r["prompt_sha"]) for r in runs} gold = [g for g in lines(gold_path) if (g["sha"], g["prompt_sha"]) in replayable] if len(gold) < MIN_GOLD: return hold(f"replayable gold cases {len(gold)} < {MIN_GOLD}") false_pass = [g for g in gold if g["human_verdict"] == "fail" and g["model_verdict"] == "pass"] if false_pass: return hold(f"{len(false_pass)} false-pass on human-fail gold") rate = sum(g["model_verdict"] == g["human_verdict"] for g in gold) / len(gold) if rate < MIN_AGREEMENT: return hold(f"agreement {rate:.0%} < {MIN_AGREEMENT:.0%}") print(f"PROMOTE: streak={streak} agreement={rate:.0%} gold={len(gold)}") return 0 if __name__ == "__main__": sys.exit(main(sys.argv[1], sys.argv[2])) 

The gold set is the part people skip. Hand-label about twenty diffs — some clean, some with a real bug — and store the human verdict next to the diff. A gate that never says fail on a known-bad diff is not a gate; it is a linter for vibes.

Step 4 — Control flake by quarantining, not retrying

Retrying a model check until it turns green teaches you nothing and hides the real problem. Instead, treat disagreement as a defect in the gate.

1. When the same sha and prompt_sha produce two different verdicts, replay it once more with the same inputs.
2. If all three runs disagree, mark the gate quarantined in the ledger and drop it from required checks by config, not by clicking through the UI.
3. Attach the three run IDs to one issue. Do not open three.
4. The gate returns to rung 2 until a streak of 30 clean runs is rebuilt.

python3 gate_promote.py ledger.jsonl gold.jsonl || \ gh label create gate-quarantine --force 

Note the asymmetry that keeps this honest: skipped and timeout never count as pass, and they break the streak. That single rule prevents a gate from being promoted on the back of runs that never happened.

What this proves, and what it does not

A promoted gate tells you one narrow thing: on the diffs your team actually pushes, this pinned prompt and model agreed with human reviewers often enough to be useful. It says nothing about a new prompt, a new model, or a diff shape you never recorded.

Also budget for the boring costs. Large diffs get skipped, so the gate covers the median PR and not the monster refactor. Latency adds to every push once the hook is on. And free tiers are terms, not physics — if the free model access or free server option changes, rung 1 gets more expensive and your ledger needs a model field that reflects the change.

Who should not adopt this

If you merge fewer than a handful of PRs a week, the ledger never reaches thirty runs and rung 3 will never be reached. If your team has no merge queue, skip rung 3 entirely and stop at an advisory PR check. And if the check would gate anything security- or compliance-critical, do not use a sampling model for it at all — use a deterministic rule and a human signature.

Start at rung 1 this week. Turn the hook on, write the ledger lines, and let the promotion script tell you when the check has earned its red X. If you want to try the free access path first, the project's own page is the place to confirm the current terms before you wire anything into your queue.