I run Claude Code most of the day, often several agents at once. For about two months I logged every session on my machine and asked one boring question of each "done" message: did a passing check actually run before the agent said it?
A "done" message here means the agent claimed tests pass, a bug is fixed, a build is green, or something is deployed. A passing check means a test, build or lint command ran in that same turn and exited 0.
752 of 1,521 done-claims had no passing check behind them. That's 49%.
What the unbacked ones look like
Most of them aren't lies exactly. They fall into three buckets:
- Stale green. The agent ran the tests, then kept editing, then reported success off the old run. This is the biggest bucket by far.
- Fixed, never rebuilt. It changed the code the compiler complained about and called the build fixed without building again.
- Green with red above it. A small number said "all tests pass" with a failing exit code a few lines up in the same turn. Those are the ones that actually cost me time.
What helped
The thing that moved the number most was boring: a stop hook that refuses to let a turn end on a success claim unless a passing check ran in that turn. The agent then just runs the tests, which is what I wanted in the first place.
It's annoying on docs-only turns, and deciding what counts as a "check" in a monorepo takes some tuning, but it turned "done" back into something I can trust without rereading every diff.
The tool
I keep this data with corescout, a local-first Rust service (and MCP server) I built that watches how my machine and my coding agents work together and records which claims were never verified, which failures keep recurring, and which fixes actually worked. It's MIT licensed: https://github.com/Mattbusel/corescout
I'm curious how other people handle this. Do you treat CI as the only source of truth, make the agent paste command output, use a second model as a judge, or just read every diff?
