A while ago I replayed the Claude Code session that built the tool I'm writing about. 906 events. 245 commands that changed something on disk. And 11 failures I never noticed while it was happening.
Nothing crashed. The session looked fine. The agent said it was done, the tests mostly passed, and I moved on. It was only when I stepped through the recording that I saw the commands that had quietly failed and been worked around.
That's the problem I keep coming back to: when an agent does something, the logs usually tell you what the agent reported. Not what actually happened. And they're written by the same system that made the mistake.
So I started building a recorder for agent runs. It captures every LLM call and tool call, links each tool call back to the model turn that asked for it, and lets you replay a run step by step. Along the way I learned that the hardest part isn't recording. It's making sure the recording isn't lying to you.
Here's what I found.
How I checked
I didn't want to just say "supports CrewAI" because the docs say there's a hook. So I built a small harness for each framework:
- a stub model server that returns a scripted tool call, so the run is predictable
- a spy wrapped around the real tool function, counting how many times it actually ran
The spy lives outside my recorder entirely. So at the end of a run I can compare two numbers: how many tool calls really happened, and how many my recorder says happened. If they don't match, the recorder is wrong.
They didn't match. A lot.
1. Tool calls disappearing in threads (CrewAI)
The first time I ran the harness against CrewAI, the spy counted 2 tool calls. My recorder captured 0.
The tools ran. The trace just didn't know about them.
The cause turned out to be a general Python thing. My recorder kept the "current run" in a ContextVar, and context variables don't automatically follow work into a thread pool. CrewAI runs tools in threads, so code inside the tool couldn't see the active trace:
import contextvars from concurrent.futures import ThreadPoolExecutor current_run = contextvars.ContextVar("current_run", default=None) def tool(): return current_run.get() current_run.set("run_123") with ThreadPoolExecutor() as pool: print(pool.submit(tool).result()) # None: context is lost ctx = contextvars.copy_context() print(pool.submit(ctx.run, tool).result()) # run_123: context kept After fixing propagation into threads, a single tool call worked. Then a thread-plus-pool scenario came back as 1 of 4 until I switched to a ContextVar stack.
If your tracing shows the LLM calls but the tool calls are missing, this is the first thing I'd check.
2. Recording <APIResponse [200 OK]> instead of the response
This one was embarrassing. Under LangChain and CrewAI, some calls go through the SDK's raw-response wrapper. My recorder was happily capturing the wrapper object, so the "response" field in my traces said:
<APIResponse [200 OK]> No content. No error either. Every trace looked complete. You'd only notice if you actually read a response.
That's the kind of bug that makes me nervous about any tracing tool, mine included. A missing field is obvious. A field that's filled with the wrong thing isn't.
3. Token counts ~40x too high
My token accounting summed cache_read_input_tokens across every turn of a conversation. With prompt caching, each turn re-reads a large cached prefix, so adding those up turn after turn counted the same context over and over. The usage it reported was roughly 40x too high.
If you build cost tracking on top of agent traces, double-check what you're summing. This is an easy mistake to make, and a confusing one to debug, because the numbers look plausible on any single turn.
4. The hook that never fires (Pydantic AI)
Pydantic AI has a method called handle_call that looks exactly like where tool calls go through. I hooked it. Nothing.
So I spied on all four candidate methods and checked which ones actually ran. The real path is execute_tool_call. handle_call never fired in my testing.
The lesson for me: don't trust a method name. Instrument, then verify against something you didn't write.
5. Bonus: why json.dumps broke my hash chains
Every event in a trace is hashed, and each hash covers the previous one, so if anyone edits, removes or reorders an event later, verification fails at that point.
It worked perfectly in Python, and failed in every other language. The reason:
>>> json.dumps(50.0) '50.0' > JSON.stringify(50.0) '50' Same value, different bytes, different hash. A trace written in Python couldn't be verified anywhere else. The fix was RFC 8785 canonical JSON, and I now check every trace against a Node implementation in CI.
What I took away
Every one of these bugs had the same shape: the trace looked fine. No errors, no gaps you'd spot at a glance. Just quietly wrong.
And that's a bit uncomfortable, because the whole reason you record an agent run is so you can trust it later, when something has gone wrong and you need to know what really happened.
The tool I ended up with is called Deposition. It records LLM calls and tool calls into a hash-chained trace, with optional signing, links tool calls to the model turns that requested them, and replays runs locally. It also has a way to declare what a run was allowed to do (tools, cost, steps) and audit the run against it afterwards, and it can import Claude Code sessions, which is how I found those 11 failures.
Some honest limits:
- Streaming responses record the request but not the content yet.
- It's tamper-evident, not tamper-proof. A signing key on the same machine raises the bar but doesn't settle it.
- Audits happen after the run. It doesn't stop an agent at runtime.
- Framework hooks move. I re-verify against current releases rather than assuming they still work.
It's open source (Apache 2.0), with zero required dependencies: github.com/chiragrajputuss-code/deposition
I'd genuinely like to hear from people running agents in production: when something goes wrong, what do you need to prove about the run, and to whom? That question is what I'm trying to build toward, and I suspect the answers are more varied than I think.
