"All tests pass."
If you use an AI coding agent, you've probably read that sentence several times today.
It sounds like proof. But it's only a claim.
So I built receipts, an open-source Claude Code plugin with one simple principle:
No receipts, no "done."
Then I tried to prove my own idea wrong.
The results were surprising. And some of them weren't particularly flattering to my plugin.
I'm publishing those too.
The Problem: Confidence Isn't Evidence
An AI coding agent can confidently report "all tests pass" even when:
It never ran the tests.
It ran them, then edited another file, making the previous green run stale.
It ran pytest | tail -5, hiding a failed command's exit code.
It tested one file and claimed the entire suite passed.
It simply reported the wrong number of passing tests.
A model can't reliably audit its own actions.
But the execution transcript can.
I replayed 4.6 GB of my own Claude Code history and found 233 success claims.
I manually reviewed every flagged claim against its transcript.
11 of those 233 claims had no valid supporting evidence — roughly 1 in 20.
In one case, Claude reported 131 passing tests when the only test run in that turn showed 114.
In another, C# files were modified after Claude had already declared "9/9 tests pass."
These weren't hypothetical failure scenarios. They came from actual development sessions.
What I Built: receipts
The plugin does three main things.
- Establishes verification rules at session start.
Claude receives clear instructions, including the project's actual full-suite test command.
- Identifies what an edit might break.
When Claude modifies a shared module, receipts identifies potentially affected files or projects across JavaScript, TypeScript, Python, and .NET.
- Audits success claims against execution evidence.
When Claude finishes a turn and claims something passed, receipts checks the transcript:
Did the relevant command actually run?
Did it succeed?
Was it executed after the last relevant code change?
If the evidence is missing or stale, the plugin explains why.
It can also flag weakened tests, interrupt repeated failing commands, maintain an evidence ledger, and scan previous sessions.
It has no external package dependencies, makes no network calls, and runs locally using Node.js 18 or newer.
Then I Tried to Prove It Wasn't Necessary
Building a tool that feels useful is easy.
Demonstrating that it actually improves outcomes is harder.
So I ran 1,260 live Claude Code sessions across exploratory and pre-registered A/B experiments using Haiku, Sonnet, and Opus.
The methodology included:
Identical prompts, settings, and fresh repository copies for each experimental arm.
Hidden acceptance tests introduced only after each session finished.
Pre-registered hypotheses, sample sizes, and decision rules for confirmatory runs.
SHA-256 hashes to identify the exact code versions being tested.
Saved session transcripts and reproducible evaluation reports.
The key question was simple:
Does receipts actually help Claude leave behind working code?
On tasks involving cross-package changes, the answer was yes.
For the released version, the share of sessions that left a working repository increased from:
43% without receipts → 78% with receipts
The difference was statistically significant (Holm-corrected p = 0.00007).
Sessions ending with a failing test suite dropped from 57% to 22%.
Four separate pre-registered runs reproduced the improvement.
The Model-Level Results Were Surprising
On the final experiment's cross-package tasks, each model faced 24 trials per condition.
Opus: 3/24 successful without receipts → 19/24 with receipts.
Haiku: 5/24 → 13/24.
Sonnet: Already solved 23/24 without additional help.
On these particular tasks, the more capable model wasn't necessarily the more careful one.
These results shouldn't be interpreted as a general ranking of the models. They describe behavior on a specific set of controlled software engineering tasks.
The Twist: Four Sentences Nearly Matched the Plugin
This is where the experiment became more interesting.
I took the same verification rules and placed them directly into a CLAUDE.md file.
No plugin.
No hooks.
No additional enforcement machinery.
Just the instructions.
The result?
receipts plugin: 78% success
CLAUDE.md instructions alone: 76% success
There was no statistically detectable difference between the two approaches in these experiments (p = 1.0).
That doesn't prove the approaches are universally equivalent.
But it does suggest something important:
The measured improvement in task success came primarily from the instructions, not the plugin's enforcement machinery.
So here are those four sentences.
You can use them for free. Just replace npm test with your project's actual full-suite command.
Before you say tests/build/lint/typecheck pass or that something works, run that check AFTER your last edit.
The pipefail instruction applies to compatible shells such as Bash; adapt it if you're using PowerShell or another shell.
If your only goal is to improve final test verification behavior, try these instructions before installing anything.
The Uncomfortable Results I Didn't Hide
Not every experiment supported the plugin.
The end-of-turn audit alone showed no statistically measurable improvement in the short experimental tasks.
Why?
Because none of those sessions made a false success claim. Even sessions that ended with failing tests reported their failures honestly.
The audit had nothing to catch.
On my actual development history, its limitations became clearer.
Out of 35 flagged claims:
11 were genuine problems.
19 were false positives.
5 couldn't be conclusively evaluated from the transcript.
That's only 11 correct flags out of 30 decidable cases.
This is why receipts warns by default rather than automatically forcing Claude to retry.
I also tested five real bug fixes from open-source repositories.
Every experimental arm solved every task.
There was no remaining task-success improvement to measure.
And there are important limitations: the controlled failure scenarios were relatively small and the experiments ran on Windows.
These results do not establish the same effect across every repository, operating system, or long-running development workflow.
So Why Does the Plugin Still Exist?
Because improving an agent's behavior and independently verifying what happened are two different problems.
Four sentences can encourage better behavior.
But they don't provide an independent record of execution.
The plugin still offers capabilities those instructions cannot provide on their own:
Detecting unsupported success claims in long sessions.
Identifying stale verification after relevant code changes.
Flagging potentially weakened tests.
Interrupting repeated failing-command loops.
Recording actual verification evidence.
Scanning historical Claude Code sessions.
Generating verification summaries for pull requests.
After several rounds of improvements, the final audit flagged 0 of 216 new, honest sessions in the held-out experiment.
And across 504 plugin-enabled sessions, the hooks recorded zero crashes.
Those results don't make the audit perfect. They show measurable progress toward making it more reliable.
Three Things I Learned
- Try the simplest solution first.
Four sentences captured nearly all of the measured task-success benefit.
A complicated tool isn't automatically a better solution.
- Evaluate your own tools as critically as someone else's.
Pre-registration, hidden acceptance tests, and reproducible experiments make it much harder to convince yourself that something works when it doesn't.
- Treat "done" as a claim, not a guarantee.
What actually ran?
When did it run?
Did anything change afterward?
And can the evidence support the conclusion?
Try It Yourself
receipts is open source.
To install it in Claude Code, run:
/plugin marketplace add Furkiak/claude-receipts
/plugin install receipts@receipts
Then scan your own session history:
/receipts:scan
You might be surprised by what you find.
Or skip the plugin entirely and use the four sentences above.
Both options are free.
The complete methodology, test results, pre-registration records, and instructions for reproducing the experiments are available on GitHub:
https://github.com/furkiak/claude-receipts
I would rather publish a result that challenges my own tool than make a stronger claim the evidence doesn't support.
What's the last thing an AI coding agent told you was "done" that turned out not to be?
receipts is an independent open-source project and is not affiliated with Anthropic.
