I recorded 87 Claude Code sessions to find out where the money goes

I wanted to know why AI coding agents get expensive on real codebases, so I stopped guessing and started recording. Every session in this post is a real Claude Code run (Claude Sonnet 5), saved as a full transcript with the cost Claude Code reported. 87 sessions in total, on two open-source repositories.

Along the way I built knock, an open-source MCP server that gives the agent a map of the codebase. The first two versions saved nothing. This post is about why, and what changed when they finally did.

The short version

On a 4,776-file codebase, the same investigation, 8 runs per side:

	Claude Code	Claude Code + knock
Median cost	$0.42	$0.25 (−40%)
Median tool calls	26	13
Median model time	142 s	86 s
Correct answers	8 of 8	8 of 8
Runs that spun up a sub-agent	5 of 8	0 of 8

But that's the end of the story. The beginning was a loss.

Attempt 1: a memory layer. It lost.

The first version of knock was a knowledge layer: it indexed docs, commit messages and code into notes the agent could search, and let the agent save findings for the next session.

I tested it on Flask with a two-day scenario. On Monday, a bug question in a fresh session: "Users get logged out every time we rotate SECRET_KEY, even though the old key is in SECRET_KEY_FALLBACKS. Why?" On Tuesday, in a new session, a small related edit.

The first recording was not kind:

Monday	Claude Code	+ knock v1
Tool calls	10	25
Cost	$0.21	$0.39

knock made it worse. The transcripts showed three bugs. Commit messages were cut to their subject line, so the one commit that explained the cause was useless. Search results filled up with off-topic notes. And code could only be found by name. I fixed all three and re-recorded, 3 runs per side this time.

Monday: $0.28 without knock, $0.27 with it. A wash. Tuesday: knock was $0.03 more expensive.

Attempt 2: a code map. It also lost.

Notes felt like the wrong shape. An agent that's about to change code wants the code's structure, not someone's summary of it.

So version 2 replaced notes with structure. One map call takes the task and returns a skeleton of the code that matters: function signatures, exact line ranges, what each function calls and who calls it, plus the commits that explain why the code changed. Then open reads exact code, and remember saves findings.

On Flask: $0.25 without knock, $0.29 with it on Monday. The same on Tuesday. No win again.

This time the transcripts made the reason obvious: map and open replaced Claude's own grep and Read almost one for one. Flask is about 200 files, and the model already knows it well. Finding the code was never the expensive part, so a better way to find code couldn't save anything. Claude even re-checked the fix commit with git show in every Monday session, though the map had already quoted that commit's explanation.

The test that mattered: a big, unfamiliar codebase

So I moved to Saleor, an e-commerce platform with 4,776 files, in a version released days before the test. The question spans a channel model, two Celery tasks, a custom beat schedule, settings and GraphQL validation:

"In one of our channels, fully paid checkouts never turn into orders automatically, even though automatic completion is enabled. Some of them are a few weeks old. What decides which checkouts get completed automatically, and where is that code?"

This is where the money went. Here are two of the recorded runs, side by side:

Without a map, Claude Code often hands the search to a sub-agent, which then explores on its own: grep, read, grep again. One of those sub-agents made 29 tool calls to answer a single question. With knock, map pointed straight at the task, the settings and the scheduler, and Claude confirmed the details with open instead of exploring.

Finding 1: you pay per step, not per token

Every step an agent takes sends the whole conversation back to the model. Prompt caching makes the repeated part cheaper, but not free, and the conversation keeps growing. So a 30-step investigation doesn't cost 30 small amounts. It costs 30 increasingly large ones.

That's why knock is built to cut steps, not just shrink what each step returns. Without knock, the most expensive runs were the ones that used a sub-agent. With knock, none did.

Finding 2: one pair of runs can lie

After I posted the first results, someone on X pointed out that agent runs on the same question can swing a lot, so a single pair can flatter either side. They were right. So I recorded 5 more runs per side and pooled them with the first 3:

Two things show up:

The spread halves. Without knock, runs split into two kinds: Claude either delegated to a sub-agent (23 to 37 tool calls, $0.39 to $0.65) or searched directly (8 to 20 calls). With knock it never delegated, and costs stayed between $0.16 and $0.38.
It's not a win every time. The cheapest run without knock ($0.22) beat knock's median. If I'd only recorded that one pair, I'd have reported no difference. With 3 runs, the saving looked like 49%. With 8, it's 40%. That's the number I trust.

Finding 3: the tool has to know when to step aside

knock also lost a test, and it kept losing it. The day after the Saleor investigation, a new session got a small edit: log how many checkouts the background job queued. Without knock, one regex search found the file: 3 tool calls, about $0.08. With knock it cost $0.14.

That gap is structural. Claude Code has to read a file with its own Read tool before it can edit it, and loading an MCP server's tools takes a step too. So an easy edit with knock takes at least 4 steps where 3 would do.

I narrowed the gap with an "edit mode" that returns the exact lines to read, but couldn't close it. The real fix was guidance. knock now tells the agent: use me to explore; for a small edit you can find with one grep, skip me. In the next batch, Claude followed it. It skipped knock for all three small Flask edits, and the whole two-day Saleor workflow (investigate, then fix) came out 39% cheaper ($0.70 → $0.43).

On small Flask the two-day workflow was only 14% cheaper, and the ranges overlap, so I don't count that as a win. knock pays off when exploring is expensive: big or unfamiliar codebases.

How knock works

Indexes locally, with zero AI calls. Saleor took 76 seconds the first time, and a refresh with no changes takes 0.3 s. Nothing leaves your machine.
Stays fresh. Before each answer it checks which files changed (about 35 ms on Saleor) and re-indexes them first. If more than 50 changed, it says the map is stale instead.
map returns the functions that matter for a task, their exact line ranges, who calls what, and why the code changed (from git history).
remember saves what the agent learned for the next session, and the note is flagged stale automatically when the code it cites changes.

It's one command to add it to Claude Code (it needs uv):

claude mcp add knock -- uvx knock-mcp serve --repo . --toolset map

Caveats

One model (Claude Sonnet 5 in Claude Code), two repositories, and a handful of scenarios. 3 to 8 runs per condition is still a small sample.
The Saleor scenario isn't untouched: I fixed a ranking bug while checking knock's answers for these exact tasks, before recording.
Costs and times are as reported by Claude Code. Tested on macOS and Linux, not yet on Windows.

Every recorded result, including the losses, is in the benchmark write-up.