I maintain two small open-source CLIs for working with AI agents. aisync is a local CLI for Multi-Agent Context Management, Coordination & Observability across AI coding tools. TRACES.md is a plain-text format for recording what an agent did, with a validator that is supposed to catch a trace that hides a failure. This post tests that validator: which hidden failures it catches, and which it lets through.

I used a temporary HOME and the v0.1.5 tag of TRACES.md (spec traces/0.1.1). It is not on PyPI, so I ran it from git:

export HOME="$(mktemp -d)" traces() { uvx --from git+https://github.com/MaxonT/TRACES.md@v0.1.5 traces "$@"; } traces init --agent claude-code --task "demo run" 

A recorded failure is a valid trace

traces run -- python3 -c "import sys; sys.exit(3)" 

The step in the file reads shell: python3 -c 'import sys; sys.exit(3)' | run python3 -c ... | fail: exit 3. traces run returns the command's own exit code to the shell (3 here, 0 for a command that succeeds). traces validate says valid: 1 files, 0 errors, 0 warnings. A failure that is written down counts as a good record.

The CLI refuses a hidden failure

traces append -t "shell: pytest" -a "run the tests" -s ok -d "exit 1, 2 failed" 
traces: step refused: error[E040]: hidden failure: status is ok but the result reports exit code 1; record it as 'fail' 

Exit code 1, and the file still had one step afterwards.

Lines written straight into the file

The check is a validator, so I also wrote single-step trace files by hand, the way a script or a person could. One file per case, each validated alone:

Result written in the file validate exit
ok: exit 1, 2 failed error E040, hidden failure 1
ok: 2 tests failed warning W002 only 0 (1 with --strict)
ok: returncode 1 valid, no warnings 0
first step is "retry the tests", no earlier fail error E041, the failure being retried is missing 1
ok: exit 0, 12 passed in place of a deleted fail step valid 0

The third row is the gap. The exit-code check is a regular expression in traces/validate.py (EXIT_NONZERO_RE), and it looks for the word exit followed by a non-zero number. returncode 1 has no such word and no failure word either, so nothing fires.

The last row is a limit by construction. A validator reads one file. If a fail step is deleted and the next step is not labelled as a retry, the file is consistent with itself. E041 only helps when the later step says "retry", "again" or similar.

What stats shows

On a trace with one fail and one ok step:

1 traces, 2 steps: 1 ok, 1 fail, 0 partial (failure rate 50.0%) 1 of 1 traces contain at least one failure 

It also lists failures per agent, per tool and per command.

What this does not show

  • I wrote every step myself, with a script. No agent produced these traces, so I measured the validator and nothing else.
  • All checks are regular expressions over text. I did not look for other phrasings that slip through, only this one.
  • I did not change anything. Whether returncode should count is a decision for the spec. I opened https://github.com/MaxonT/TRACES.md/issues/22 with a table of the phrasings I tried afterwards: rc=1, return code 1 and status 1 also pass, exited 1 is caught.
  • One version (v0.1.5), one machine (macOS, arm64), uv. I did not test the JSONL format, traces fmt, the schema, or traces status.

If you validate agent logs

Do not treat a clean validator run as proof that nothing failed. It tells you the file is consistent with its own rules. Record the exit code from the process, as traces run does, instead of typing it.

This post and the test runs were written and run by an AI agent (Claude Code) for the maintainer. The outputs above are copied from those runs.