A Test Run I Aborted Came Back and Corrupted My Evidence

From the agentdrill v0.1.0 release, part of agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses.

The case finished at 12:09 with complete evidence. At 12:49 its assertions file was empty. Nothing was running. Or so I believed.

Somewhere in the background, a sweep I had aborted hours earlier was still alive, patiently re-running cases. Between 12:45 and 12:49 it wrote over the crew results and truncated an assertions file mid-case. The process I thought I had killed had killed my evidence instead.

This is the kind of thing that only happens to test data, because we treat test data as if it were not data. The full forensics are in the agentdrill v0.1.0 field test report, issues I-10 and I-11.

Two writers, one directory, no locks

The harness truncates a case's logs at the start of each run. That is correct behavior for a fresh run. It is catastrophic behavior when two runs touch the same case directory. The orphaned sweep started cases fresh, which wiped the assertions files, then got torn down partway, which left some of them empty.

The case it hit, FT-B-CREW-EXAMPLES, had a healthy verdict from 12:09 with all five assertions embedded in it. At 12:49 the backing file was blank. A pass verdict with no assertions behind it is a claim without proof, and it sat there looking exactly like every other green row.

The only reason it got caught: a grounding gate. The harness re-derives the report from disk, and a PASS with zero grounded assertions is flagged as over-claimed. It flagged exactly one case. That is the entire safety net between "corrupted evidence" and "shipped as proof", and it worked because it reads the raw files instead of trusting the summary.

The second landmine was a default

The same day, a different foot-gun surfaced. Running a single case directly defaulted its results to a directory called results/all/, next to the per-suite directories. So a quick one-case re-run created a duplicate case directory, a second copy of the truth that would drift from the first with every subsequent run.

Defaults that write files are quiet landmines. The second run does not fail. It forks the record. And a forked record is worse than a lost one, because now there are two answers and no marker saying which is canonical.

The fix in lib.sh is my favorite change of the whole run, mostly because of its tone. There is no default run ID anymore. If a runner starts without an explicit destination, the harness dies before writing anything, announcing FATAL: FT_RUN_ID is not set and refusing to write to a default results dir. A stray all/ directory can never be created silently again. run-case.sh now derives the correct suite directory from the case registry before anything else happens.

What recovery looked like

Three things made this recoverable instead of fatal:

1. The verdict carried embedded evidence. The 12:09 verdict listed its assertions inline, so the proof existed in two places. When one was truncated, the other still testified.
2. Git held the pre-corruption state. The evidence was committed, so recovery meant regenerating from source, not mourning.
3. The grounding gate pointed at the wound. One over-claimed PASS was the signal, and it was precise. No scanning, no guessing. The gate named the case.

The evidence was regenerated in place, inside the correct suite directory, and the grounding gate went back to green with zero over-claims across all 73 units.

The lesson is about data, not tests

I keep re-deriving the same conclusion from this run: test evidence is production data. It needs the same engineering we would never skip for a database:

• Unique write namespaces. No two writers in one directory, ever. Parallel runners get their own branches of the results tree.
• No default write paths. If the destination is ambiguous, refuse to start. A hard failure at launch is infinitely cheaper than a silent fork at 12:49.
• Re-derive conclusions from raw files. The summary is a cache. The grounding gate reads what actually happened and compares.
• Abort must mean abort. A killed parent with living children is a ghost, and ghosts write files.

Tip: if two processes can write your results directory, make the second one crash at launch, not at 12:49.

What did not work: process hygiene, full stop. The sweep should have died with the tool call that started it. Process groups exist for exactly this, and we were not using them. Every other control was a net that caught the failure after it happened. This one was the only control that could have prevented it.

If two processes wrote your test results directory tonight, would tomorrow's report know? For us the answer was "only because a gate re-reads the raw files", which is a good answer but an accidental one. I would rather have the answer be "impossible by construction", which is what the FATAL guard now gives us.

Code and receipts: agentdrill · field test report · the guard that refuses a default results dir · the grounding gate

About agentdrill

This story is one of seven from the agentdrill v0.1.0 field test. agentdrill is the adversarial-testing engine in agentic-suite: open-source, harness-agnostic security tools for AI-agent defenses. It runs versioned attack packs against your deployed stack in CI and publishes honest results, misses included.

pip install agentsec-agentdrill # v0.1.0 is live on PyPI agentdrill quickstart # first honest result in ~15 minutes, bundled demo 

Dig deeper: the v0.1.0 field test report (73/73 units, every pass grounded), the release notes, and the rest of the suite at github.com/agentic-suite: agentdrill drills the defenses, agentwatch records the evidence.