A demo that only runs while someone else's quota lasts is a claim, not evidence. So here is the whole thesis up front: if you cannot replay your LLM run with the network unplugged, you cannot debug it, benchmark it, or hand it to a reviewer.
Free model access makes experimentation cheap. It also makes your experiment fragile, because the run depends on a quota you do not control and an endpoint you did not build. The fix is old and boring: record every call once, replay it forever.
This tutorial builds that recorder from zero — a call spine, a hash-chained cassette, an offline replayer, and a tamper test — with a verification step at each stage. All code below is a complete runnable sketch. I have not benchmarked it on your machine, so treat the verification steps as the actual product and the code as scaffolding.
Step 1 — Collapse every model call into one function
Before you record anything, you need a single choke point. If five modules each build their own request, you have five things to intercept and none of them will stay in sync.
So write the spine first:
# spine.py import hashlib, json, os from dataclasses import dataclass, field @dataclass class Call: prompt: str model: str params: dict = field(default_factory=dict) def call_key(c: Call) -> str: blob = json.dumps( {"p": c.prompt, "m": c.model, "x": c.params}, sort_keys=True, ensure_ascii=False, ) return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16] def live(c: Call) -> str: """The only place in your codebase that touches the network.""" raise NotImplementedError("wire this to your client") The key is a hash of prompt + model + params. That matters more than it looks: change one sampling parameter and the key changes, so a stale cassette can never silently answer a question you did not ask.
Verify: call call_key twice on the same Call and once on a copy with params={"temperature": 0.2} instead of 0.7. You want two identical strings and one different one. If all three match, your params are not reaching the hash.
Step 2 — Append to a cassette with a hash chain
A JSON file of outputs is not a record. Someone edits a field, the demo "improves," and nobody notices for a month. So chain each record to the previous one:
# cassette.py import hashlib, json, time class Recorder: def __init__(self, path): self.path = path self.prev = "genesis" def append(self, key: str, output: str, meta: dict | None = None) -> None: rec = { "prev": self.prev, "key": key, "output": output, "meta": meta or {}, "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), } body = json.dumps( {k: rec[k] for k in ("prev", "key", "output", "meta")}, sort_keys=True, ensure_ascii=False, ) rec["hash"] = hashlib.sha256(body.encode("utf-8")).hexdigest() with open(self.path, "a", encoding="utf-8") as f: f.write(json.dumps(rec, sort_keys=True, ensure_ascii=False) + "\n") self.prev = rec["hash"] One JSON object per line, append-only, each line carrying the previous line's hash. Boring on purpose.
Verify: run it twice on a fresh file and tail -n 2 run.jsonl. The second record's prev must equal the first record's hash. If they are equal to "genesis" twice, your recorder is being re-instantiated somewhere you did not expect.
Step 3 — Wrap the spine so record and replay share one path
Here is where most homegrown harnesses fork and rot. Keep one entry point, two modes:
# harness.py import json, os from cassette import Recorder from spine import Call, call_key, live class Cassette: def __init__(self, path, mode): assert mode in ("record", "replay", "auto") self.mode, self.path = mode, path self.rec = Recorder(path) if mode != "replay" else None self.index = self._load() if mode != "record" else {} def _load(self): out = {} if not os.path.exists(self.path): return out with open(self.path, encoding="utf-8") as f: for line in f: rec = json.loads(line) out[rec["key"]] = rec["output"] return out def ask(self, c: Call) -> str: k = call_key(c) if self.mode == "replay": if k not in self.index: raise KeyError(f"cassette miss: {k} — your prompt changed, not the model") return self.index[k] if self.mode == "auto" and k in self.index: return self.index[k] out = live(c) self.rec.append(k, out, meta={"model": c.model}) self.index[k] = out return out The KeyError is the feature. A cache miss during replay is a loud, honest failure instead of a quiet fallback to the network — which is exactly how "offline" benchmarks end up secretly online.
Verify: record a two-call run, then python bench.py --replay with Wi-Fi off. Then edit one word of one prompt and replay again. You want a crash naming the missing key. No crash means you are not actually reading the cassette.
Step 4 — Prove the chain, then try to break it
An integrity check you never attack is decoration. Write the verifier and then vandalise a copy:
# verify.py import hashlib, json, sys def verify(path): prev = "genesis" with open(path, encoding="utf-8") as f: for n, line in enumerate(f, 1): rec = json.loads(line) if rec["prev"] != prev: return f"line {n}: prev mismatch" body = json.dumps( {k: rec[k] for k in ("prev", "key", "output", "meta")}, sort_keys=True, ensure_ascii=False, ) if hashlib.sha256(body.encode("utf-8")).hexdigest() != rec["hash"]: return f"line {n}: content edited after append" prev = rec["hash"] return "ok" if __name__ == "__main__": print(verify(sys.argv[1])) Verify:
cp run.jsonl tampered.jsonl sed -i 's/"output": "./"output": "X/' tampered.jsonl # change one character python verify.py run.jsonl # -> ok python verify.py tampered.jsonl # -> line N: content edited after append If both print ok, your sed did not match and you proved nothing. Re-run the check on the tampered file before you believe the green.
Step 5 — Put the replayer where reviewers can reach it
Now the payoff. The recorder runs once against free model access; the replayer runs anywhere, forever, at zero cost. That is the split worth building around.
This is also the one place where MonkeyCode's free model access and free server option genuinely participate in the method rather than decorate it: exploration burns quota, so you explore in --record mode on the free models, and then you host the pure-replay demo on the free server so a reviewer can click through your eval without touching a paid endpoint. The project publishes its own current allowance and server terms — read those numbers on the project's site before you plan a budget, because quotas change and a blog post from last week is not a source.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Verify: a minimal replay server you can run locally first, then move:
# serve.py — replay only, no network, no keys import json, os from http.server import BaseHTTPRequestHandler, HTTPServer from harness import Cassette from spine import Call CASSETTE = Cassette(os.environ.get("CASSETTE", "run.jsonl"), "replay") class H(BaseHTTPRequestHandler): def do_POST(self): body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) try: out = CASSETTE.ask(Call(body["prompt"], body["model"], body.get("params", {}))) code, payload = 200, {"output": out} except KeyError as e: code, payload = 422, {"error": str(e)} raw = json.dumps(payload).encode() self.send_response(code) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(raw))) self.end_headers() self.wfile.write(raw) HTTPServer(("0.0.0.0", 8080), H).serve_forever() Check it with the dumbest possible client:
python serve.py & curl -s localhost:8080 -d '{"prompt":"hello","model":"any"}' -H 'Content-Type: application/json' # expect 422 + a named missing key, because you never recorded that prompt A 422 on an unknown prompt is a passing test. A 200 with invented text is a failing system.
When to record, when to go live
| Situation | Mode | Why |
|---|---|---|
| Prompt and parser iteration | replay | Free, instant, and diffable |
| First run on a new model | record | You need the real output once |
| Reviewers or CI | replay | No keys, no quota, no flakiness |
| Suspected model drift | record into a new file | Never overwrite the old cassette |
| Cost or latency benchmarks | neither | Cassettes contain no timing data |
Where this breaks
- Cassettes are not evidence of quality. They prove a run happened, not that the output is good. Pair them with a scored set, or you are just archiving noise.
- They rot. A prompt edit invalidates every downstream key. That is the correct behaviour and it will still annoy you.
- They are not signed. A hash chain detects edits by someone who forgot to re-chain; it does not stop someone who re-chains deliberately. If you need provenance, sign the tail hash.
- They leak. Prompts and outputs land in plaintext. Scrub secrets before writing, not after publishing.
- Sampling is lost.
temperature > 0means the live path was never deterministic to begin with. Replay is reproducible; the original call was not.
Who should skip this
If your whole pipeline is one prompt in a notebook, a cassette file is overhead — just paste the output into a fixture. If your calls are genuinely non-reproducible by design, such as live tool use against the open internet, record the tool outputs, not the model, or this whole structure will fight you. And if you never intend to hand a run to another human, you do not need a chain; you need a screenshot.
Everyone else: record once, replay always. Then ask yourself the only question that matters — could a stranger reproduce your result from your repo alone, with the network down and no keys in their shell?
If the answer is no, you do not have a benchmark yet. You have a demo with good timing.
