A reviewer opens a pull request whose description reads like a closing argument: three files touched, suite green, one flaky test skipped for reasons. You check out the branch, run the command printed in the description, and two tests fail before the thirty-second mark. Nothing malicious happened here, and no model lied in any interesting sense — the summary simply outlived the transcript it came from.
That gap between what an agent reports and what a machine can prove is the single most useful thing to instrument when you work on a free tier. Cheap generation invites many attempts, and many attempts multiply the number of self-written summaries you have to audit. Below are four claims developers repeat about those summaries, the evidence that undermines each one, and a corrected model you can encode as a small script you keep in the repo.
Myth 1: If the output says the tests passed, the tests passed
The tempting reading is that an agent's prose is a report from the test runner. It is not; it is a paraphrase of stdout that survives truncation, sampling, and whatever the harness chose to keep. Shell semantics do a lot of quiet damage here, and the most common offender is a pipeline, because a pipeline returns the status of its last command rather than the status of the interesting one.
# Reports success whenever tail succeeds — the test result is discarded. pytest -q 2>&1 | tail -n 20 # The same command with the status of the real process preserved. set -o pipefail pytest -q 2>&1 | tail -n 20 echo "pytest exit=${PIPESTATUS[0]}" Agents also write || true, wrap commands in subshells that swallow status, or quietly re-quote a result they read from an earlier turn in the session. The corrected mental model is blunt: treat stdout as a hint and the process exit status as the fact, and capture that status in a place the model cannot rewrite.
Myth 2: A clean diff describes the change
A git diff that looks tidy is evidence about tracked files, which is a narrower category than most people assume. Untracked files never appear, and generated artifacts frequently land in ignored directories where a reviewer will never see them. A test that passes only because a stray fixture file happens to exist on the author's machine is not a passing test.
# What actually changed, including the parts diff hides. git diff --stat HEAD git status --porcelain # '??' lines are untracked and invisible to diff git diff --stat --cached # staged work, if the agent committed anything The corrected model is that a change has three surfaces — tracked modifications, staged content, and untracked additions — and any verification that ignores one of them is a partial audit. Record all three in the same place, at the same moment, or you will be reconstructing them from memory later.
Myth 3: Re-running the command reproduces the run
Re-running a command reproduces your intent, not your environment. An unpinned install resolves whatever the registry serves that afternoon, and a free-tier sandbox that is recreated between sessions will resolve differently from your laptop. Interpreter minor versions and base image digests drift the same way, quietly, without producing any message that looks like an error.
# Freeze the inputs you would otherwise re-resolve. python3 -V >> run-receipt.json node -v >> run-receipt.json python3 -m pip freeze > constraints.txt # or: npm ci with a committed lockfile sha256sum constraints.txt package-lock.json 2>/dev/null >> run-receipt.json The corrected model separates generation from resolution. A model call is nondeterministic by design, so the only reproducibility you can buy is in the environment around it: pinned dependencies, a recorded interpreter version, and a lockfile hash you can diff between runs.
Myth 4: Verification is the expensive part
Teams new to free tiers sometimes skip verification to conserve allowance, which inverts the economics. Deterministic checks — exit codes, file inventories, lockfile hashes — are cheap and repeatable, while the expensive operation is the generation step you are trying to justify. Verifying every attempt costs a rounding error compared with the tokens spent producing attempts.
The artifact: a run receipt you can diff
The script below is the smallest thing I would keep in a repository that accepts agent-authored changes. It runs a list of commands, records each exit status and duration, inventories untracked files, and writes a single JSON file you can attach to a review or diff between two runs. It is deliberately boring, has no dependencies, and refuses to pass when an untracked file is present.
#!/usr/bin/env python3 """Generate a run receipt: what changed, what ran, what exited non-zero.""" import json, pathlib, subprocess, sys, time def run(cmd: str) -> dict: t0 = time.monotonic() p = subprocess.run(cmd, shell=True, capture_output=True, text=True) out = (p.stdout + p.stderr).strip().splitlines() return { "cmd": cmd, "exit": p.returncode, "seconds": round(time.monotonic() - t0, 2), "tail": out[-5:], } def git(*args: str) -> str: return subprocess.run(["git", *args], capture_output=True, text=True).stdout.strip() status = git("status", "--porcelain") untracked = [line[3:] for line in status.splitlines() if line.startswith("??")] receipt = { "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "head": git("rev-parse", "HEAD"), "dirty": bool(status), "untracked": untracked, "diffstat": git("diff", "--stat", "HEAD").splitlines()[-1:], "checks": [run(c) for c in sys.argv[1:]], } receipt["ok"] = all(c["exit"] == 0 for c in receipt["checks"]) and not untracked pathlib.Path("run-receipt.json").write_text(json.dumps(receipt, indent=2) + "\n") print(json.dumps(receipt, indent=2)) sys.exit(0 if receipt["ok"] else 1) Usage is a single line, and the exit status of the wrapper is the exit status of the run, which means it composes with whatever already gates your branch:
python3 receipt.py "pytest -q" "ruff check ." || echo "receipt says no" Read the receipt in three passes rather than one. First, check ok and every exit field, since anything non-zero ends the discussion. Second, look at untracked, because an empty list is the claim most likely to be false and the cheapest to verify. Third, compare head and diffstat against the description you were given — if the prose mentions two files and the receipt shows five, the prose is the thing that gets edited.
| Signal in the receipt | What it supports | What to do next |
|---|---|---|
ok: false, exit != 0 | The run failed on at least one check | Reject the summary; do not re-prompt before reading tail |
untracked non-empty | The environment carries files the diff omits | Inspect each path; decide tracked, ignored, or deleted |
dirty: true, head unchanged | Work exists in the tree but not in history | Diff the tree, then commit deliberately |
| All checks pass, receipt attached | A claim you can audit without trusting prose | Review the change itself, not the narrative |
Where the free tier fits the loop
This workflow is designed to be run once per attempt rather than once per release, which is exactly the pattern that free tiers make affordable. MonkeyCode is relevant here for two operator-stated reasons: free model access and a free server option, which together mean the generation and the receipt can run in the same remote workspace instead of on your laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator describes a free allowance of 10,000,000 tokens alongside the free server option. I have not independently audited that figure, the model list, the hardware, or how long the offer lasts, so treat all of it as a claim to confirm on the product page before you plan work around it. Nothing in the script above depends on MonkeyCode specifically; it runs against any sandbox that gives you a shell and a git checkout.
Who should not adopt this
Skip the receipt if your changes are already gated by a CI pipeline that captures exit codes, artifact inventories, and environment digests on its own — you would be duplicating a control you already trust. Skip it as well if you are doing exploratory work where the goal is to learn what a model can do rather than to accept a patch, because the extra ceremony will slow down the part that matters. And do not use a passing receipt as a substitute for reading the diff, since the script proves that something ran, not that it was the right thing to run.
The habit worth keeping is smaller than any tool: before you repeat an agent's claim in a review comment, ask which field in the receipt would have to be wrong for that claim to be false. If there is no such field, you have a paragraph rather than a result. If you want to try the loop end to end, the free tier is enough to run the receipt a few dozen times before you spend anything deciding whether it is worth keeping.
