About 3x faster. That is the whole claim, and it sits in the last line of the agent's summary: "Replaced the regex with a hand-rolled scanner. ParseLine is now about 3x faster, all tests pass." Above it you have a diff and a green test run.

The number is a sentence. Nothing in the session timed the old code and the new code under the same conditions and compared them. Maybe the agent ran a benchmark once, on a warm cache, while your other agent was compiling in the next terminal. Maybe it ran nothing and estimated.

Published research says you should check. When the authors of the GSO benchmark gave leading coding agents 102 real optimization tasks from 10 codebases, the agents succeeded less than 5% of the time. A success there means a patch that passes the correctness tests and reaches at least 95% of the speedup an expert developer got (GSO). The paper names "lazy optimization" and trouble finding the actual bottleneck among the failure modes. It dates from 2025 and the models have moved since, but the lesson holds: a claimed speedup is a hypothesis until you measure it.

The fix is a measurement, and most people skip its last step: reading whether the difference is real.

Put the base and the agent's commit side by side

Say the agent worked on a branch called hermes/speed-up-parseline. You want two folders: one with the exact commit the agent started from, one with the agent's latest commit. git worktree gives you both without a second clone.

cd ~/code/app BR=hermes/speed-up-parseline BASE=$(git merge-base main "$BR") git worktree add --detach ../app-base "$BASE" git worktree add --detach ../app-agent "$BR"

Use the merge base, not main. If main moved while the agent worked, comparing against it mixes someone else's changes into your result. The merge base is the commit the agent's branch split from, so the only difference between the two folders is the agent's work.

The --detach on the agent side matters. By default, git worktree add refuses to check out a branch that another worktree already has, and the agent's own worktree has that branch. A detached checkout of the same commit gets around that, and your builds in ../app-agent never touch the files the agent is still editing.

One limit: this measures what is committed. If the agent left its change uncommitted, ask it to commit first, or build the agent side from its own worktree.

Time the whole program with hyperfine

If the claim is about a command (a CLI, a build step, a converter), build both sides and let hyperfine run them. These examples assume a Go program with its entry point in ./cmd/app. Swap in your own build command.

mkdir -p ~/bench (cd ../app-base && go build -o ~/bench/base ./cmd/app) (cd ../app-agent && go build -o ~/bench/agent ./cmd/app)

Then run both against the same input:

hyperfine --warmup 3 --runs 30 \ --prepare 'rm -f out.json' \ --export-markdown bench.md \ --export-json bench.json \ -n base "$HOME/bench/base convert big.csv out.json" \ -n agent "$HOME/bench/agent convert big.csv out.json"

What each flag does, from the hyperfine docs:

  • --warmup 3 runs each command three times before measuring, which fills disk caches so the first timed run isn't the odd one out.
  • --prepare runs before every timed run. Here it deletes the output file so each run starts from the same state. To measure cold-cache behavior on Linux instead, the README's example drops the page cache in --prepare.
  • --runs 30 fixes the sample size. Without it, hyperfine does at least 10 runs and aims for about three seconds of measuring.
  • -n gives each command a short name for the report.
  • --export-markdown writes a table with the mean, the ± spread, and a "Change" and "Factor" column against the reference.

Order matters. Since hyperfine 2.0.0, released on October 7, 2026, the first command is always the reference and --reference has been removed for now, so put base first. Version 2 also runs commands without a shell by default, which is why $HOME sits inside double quotes: your own shell expands it before hyperfine sees the string. --prepare still runs through a shell.

The mean and ± alone don't tell you whether the gap is noise. The hyperfine repo ships analysis scripts for the JSON export (scripts README), including a Welch's t-test for exactly two benchmarks. With uv installed:

uvx --from 'git+https://github.com/sharkdp/hyperfine' \ welch_ttest bench.json

It ends with one of two lines, either "A statistically significant difference in mean value was detected (p < 0.05)." or "No statistically significant difference in mean value was detected (p >= 0.05)." The second one means the agent's "3x faster" didn't survive 30 runs.

Benchmark the Go function with benchstat

When the claim is about one function, time that function. Write the benchmark with b.Loop, which the testing package added in Go 1.24. It resets the timer on the first call, so setup above the loop isn't measured, and it keeps the compiler from optimizing the call away.

package csvx import "testing" var line = []byte("42,acme,2026-11-02,paid,19.99") func BenchmarkParseLine(b *testing.B) { b.ReportAllocs() for b.Loop() { ParseLine(line) } }

The same benchmark file has to exist on both sides. If the agent wrote it as part of its change, copy it into ../app-base/csvx/ before you build. A benchmark that differs between the two folders compares two different things.

The benchstat docs suggest at least 10 runs per side, ideally 20, and interleaving the old and new runs so noise from your machine spreads evenly across both. Compiling each side once with go test -c makes interleaving cheap:

(cd ../app-base && go test -c -o ~/bench/base.test ./csvx) (cd ../app-agent && go test -c -o ~/bench/agent.test ./csvx) rm -f old.txt new.txt for i in $(seq 20); do ~/bench/base.test -test.run='^$' \ -test.bench=ParseLine -test.benchmem >> old.txt ~/bench/agent.test -test.run='^$' \ -test.bench=ParseLine -test.benchmem >> new.txt done benchstat old.txt new.txt

-test.run='^$' skips the unit tests so only the benchmark runs, and -test.benchmem adds allocation counts. Each pass of the loop adds one sample per side, so you end with 20 each. If you don't have benchstat yet, go install golang.org/x/perf/cmd/benchstat@latest installs it.

Read the interval before the headline

This is one line from the benchstat documentation's own example, with the column spacing trimmed:

Encode/format=json-48 1.718µ ± 1% 1.423µ ± 1% -17.20% (p=0.000 n=10)

Each column is the median for that file with the ± range of a 95% confidence interval. The percentage is the change from the first file to the second. p is how likely a difference this large is to come from noise alone, and n is the sample count. When benchstat prints ~ instead of a percentage, it found no statistically significant difference. The default threshold is p < 0.05.

Two runners crossing a finish line almost together, with a magnifying glass held over the photo-finish strip
A photo finish needs a closer look before anyone gets the medal.

The same docs give a few rules for reading the result:

  1. ~ means no measurable speedup. Whatever the agent wrote, the data doesn't support it.
  2. Don't rerun until it turns significant. Rerunning until you get the answer you wanted is multiple testing, and it biases the result.
  3. Expect false alarms. At p < 0.05, about 5% of benchmarks show a significant change that isn't real, so if you compare 20 benchmarks at once, one of them will probably light up by chance.

The docs also warn that significant isn't the same as large: with enough low-noise data, a tiny change can pass the test. So decide what size of win is worth the extra code before you look, and then check the claim itself. "About 3x faster" is −67% on time. A real result of −17% means the code got faster and the agent's sentence was wrong. Note both, because you will read that agent's next claim too. And speed only counts if the code is still correct: GSO counts a patch as a success only when it also passes the correctness tests, and you should hold your branch to the same bar.

Send the numbers back in Hermes

I build Hermes IDE, so weigh this part with that in mind. Nothing above needs it. The git, hyperfine and benchstat steps work in any terminal.

What Hermes changes is where the task already lives. When you start a task with ⌘N (Ctrl+Shift+N on Windows and Linux), the agent runs on its own hermes/ branch in its own worktree, which is the $BR you passed to git merge-base above. The Worktrees tab in the Review Desk shows which folder belongs to which task, so you know what to build.

The Hermes Review Desk Worktrees tab listing one worktree per task with its hermes/ branch, task, age and disk size
The Review Desk's Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size, and the free disk space.

Once you have the numbers, open the Review Desk with ⌘G (Ctrl+Shift+G) and switch to By turn. Find the turn where the agent rewrote ParseLine, and put a line comment on the changed line with the benchstat output pasted in. The Review Desk sends that comment to the agent that wrote the line, so it lands in the right session even with three agents running.

The Hermes Review Desk in By turn view, showing two turns of a Claude Code task, a diff of src/db.ts and a drafted commit message
The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns listed (T1: 3 files +69 −1, T2: 1 file +1 −0), turn 1's diff of src/db.ts adding a refunded status, the changed-files list, and a commit message drafted from the turns.

A comment that works, with your own numbers filled in: "benchstat shows ~ for BenchmarkParseLine at n=20 per side, output below. The scanner adds code and no measurable gain. Revert this turn unless you can show a benchmark where it wins." The agent gets a measurement to answer, and the next claim it makes has to come with one.

If the win is real, use Land: it commits, opens a pull request or merges locally into the branch the task started from, and you can undo it. Either way, remove the two measurement folders when you're done. If you copied the benchmark file into ../app-base/csvx/, delete that copy first, or remove will refuse:

git worktree remove ../app-base git worktree remove ../app-agent

The same check covers modified files. Leave --force off and git worktree remove won't delete work you forgot about.

Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. Download it here, and if you find something that doesn't behave the way I described, open an issue on GitHub.

If this was useful

Most of this post is Go's own tooling doing the work: the testing package, b.Loop, go test -c and the flags behind them. If you want the language, the runtime and the standard library underneath those tools explained from the ground up, that is what The Complete Guide to Go Programming covers. It is the first book of the Thinking in Go series.

Thinking in Go — the 2-book series on Go programming and hexagonal architecture