Capture that depends on the model's cooperation is not evidence

We build Chron, a tool that records AI coding sessions locally so there is an audit trail of what an agent actually did. It passed 12,000 downloads recently. The records are hash chained, they live on your own machine, and nothing is sent anywhere on its own.

The principle has not moved since the first release: Chron records what happened. It does not certify that what happened was correct.

Which means the interesting failures are not in what we record. They are in what we do not.

The problem with asking a model to log

Chron's primary capture path is an MCP server. The agent calls log_message, Chron writes a row, the row joins a hash chain. This works.

It works when the model calls the tool.

When it does not, you get a perfectly good answer from the agent and no record of it. Nothing in the database distinguishes that from a tool that was never installed, or one that failed silently, or a server that was configured but never attached. We recorded one such result honestly as:

reply produced; no Chron session recorded; cause unestablished

Capture that depends on the model choosing to cooperate is not evidence. It is a hope with a timestamp. You cannot build an audit claim on an instruction in a system prompt, because the thing being audited is the same thing being asked to do the logging.

A reviewer reads silences, not records

Someone with a security services background asked us a question in a comment thread that turned out to be the sharpest piece of feedback we have had:

Does Chron make intentional redaction distinguishable from missing capture? For a tool call that handles sensitive data, a reviewer needs to know whether the content was withheld or never recorded.

The answer is no. And checking the code to answer it properly was worse than I expected.

A missing entry for a tool call currently means one of three things, and nothing on the record says which:

1. The content was deliberately withheld as sensitive.
2. Nothing was captured at all.
3. Capture ran, but the secret detection pass never finished.

Here is what we found when we looked instead of assuming:

• Content is stored verbatim. The insert path normalises text so the hashed bytes and the stored bytes match, which is not the same as redacting it. "Intentionally withheld" is not a state our capture path produces, so there was never anything to distinguish from absence.
• The detections table records a sighting, not a withholding. Its columns are a type, a masked value and a timestamp. It says a secret was seen here. It never says content was withheld here.
• The redaction helpers persist nothing. scanPrompt() and rehydratePrompt() take no database handle at all. Redaction through that path leaves no audit record.
• Tool calls are not scanned. logToolCall, logToolResult and logCodeChange never call the detection writer. Only message logging does. The exact case in the question gets no detection, no masked record, nothing.

So the question was not "is this feature complete". It was "does your evidence model have a hole in it", and it did.

We had the same disease in our own tests

This is the part worth writing up, because it generalises past our product.

The last patch release took seven candidates. Six were held. Nearly every hold was the same defect wearing a different costume:

A check that confirms something is present, and reports it as a claim that the thing works.

Some specimens.

The read-back that accepted nothing. A packaged test logged a 2MB message, read it back, and asserted the response contained a digit. QA substituted an empty message list with session_id: "synthetic-0". The 0 satisfied it. The check reported storage that never happened.

We strengthened it to require a unique marker, the role, and the stored length. QA flipped one character in the middle of the 2MB payload. Marker intact, role intact, length identical. It passed again.

Only the third version actually asserted the thing we meant:

} else if (mine.content !== content) { const at = [...content].findIndex((c, i) => mine.content[i] !== c); failures.push(`came back altered, first differing at index ${at}`); } 

Twice I strengthened a check just far enough to defeat the last counterexample, rather than asking what the check was for. The assertion was always "the bytes that came back are the bytes that went in". Everything before it was an approximation.

The reader that rewrote its evidence. A probe whose job was to confirm nothing had been written to a database opened it directly. Against a killed-writer fixture, SQLite recovered the write-ahead log in order to read it: the main file grew, and the log was checkpointed to nothing. A tool for proving absence of writes had rewritten the thing it was inspecting. PRAGMA query_only does not help. Neither does mode=ro. Both checkpoint. You have to copy the database and its sidecars and query the copy.

Four gate fixtures that passed against deliberately broken builds. Each was supposed to catch a boundary bug. We only found out they were worthless by validating them against an artifact known to contain the bug, and watching them pass.

That last one became the rule: a new check is not coverage until it has failed against a known-bad artifact. Disarm the fix, watch the test go red, then keep the test. If you cannot make it fail, you have not written a test, you have written a comment.

Measure the advisory, do not read its title

One more, because it cost me a wrong conclusion in public.

A dependency advisory described memory exhaustion while validating oversized constrained collections. I assumed it applied to us, wrote up a mitigation, and was corrected. So I measured, on 200,000 elements:

The constrained case differs by about sixty times. The unconstrained case does not differ at all, because validating every entry of an unbounded collection is the required work, not a defect. Our schemas declare no constrained collection, so the advisory's mechanism never applied to them. The real exposure was somewhere else entirely, and the fix I had proposed first would have remediated nothing while looking responsible.

Also worth knowing: the new option that does mitigate it ships disabled by default. Upgrading the dependency changes nothing until you set it.

Where this goes

Two directions, both pointed at the same problem.

Capture comes out of the model's hands. A wrapper writes the journal itself, cooperating model or not, and verification compares the journal records to the projected session field by field rather than trusting a column.

Withholding becomes a recorded event, sitting next to the action and its result, with the sensitive value never exposed. A reviewer must be able to separate "withheld", "never captured" and "detection did not finish" without being shown what was hidden. And a generated summary has to stay traceable to those records instead of becoming the basis for inferring what happened. A summary that silently collapses "withheld" and "never captured" is worse than no summary.

Finding this class of bug repeatedly in our own instruments is, perversely, the strongest signal we are now pointed at the right problem. The failure we are asking people to trust us to catch is exactly the failure we kept shipping into our own test suite.

If your agent went quiet during a sensitive operation, could you prove today whether that was discretion or a gap?