One hand-written agent harness, five switches, the same model and the same task. Removing the context budget or the error feedback dropped completion from 3/3 to 0/3.

How much of an agent is the loop, and how much is the scaffolding around it? I wanted a number, so I measured it.

I wrote one harness from scratch - no agent framework - with five places where behaviour can be injected, then switched each one off and watched what the wire did. All 21 runs (7 cells x 3) went through a local recorder proxy, so every number below is recomputed from the recorded traffic instead of from memory.

What happens with a bare loop

Short version: with the bare loop, all three runs stopped and nothing was ever published.

Here is the output of the analysis script - completion rate, mean calls and mean tokens per cell (I left out the model-name line at the top):

cell n ok% calls tokens oob-final bad-pub tool-err stalls ------------------------------------------------------------------- full 3 100% 6.3 10528 0 0 3 0 naive 3 0% 2.0 2089 3 0 0 0 no_s1 3 0% 3.0 3636 3 0 3 0 no_s2 3 0% 2.0 2079 3 0 0 0 no_s3 3 67% 6.3 10118 1 0 3 0 no_s4 3 100% 6.3 10908 0 0 3 0 no_s5 3 100% 7.3 12859 0 0 4 0 

What the columns mean:

• oob-final: runs whose final output fell outside the 95-105 word band (a run that never produced a final output counts here too)

• bad-pub: digests published while out of band / stalls: runs stopped by a repeated identical search

full is every seam on, naive is every seam off, no_sX is full minus seam X.

The table states the story, but you cannot see what breaks without the inside. Here is the inside.

What a seam is

I call the places where a harness can inject behaviour that a bare loop does not have "seams". There are five of them.

flowchart TD A([Start]) --> B{S2 context_budget} B -- prompt fits --> C([Model call]) B -- too big, seam ON --> B1[Gist older tool results] B1 --> C B -- too big, seam OFF --> X([Stop: over_budget]) C --> D{Any tool calls?} D -- no --> E{S3 verify_loop} E -- final in band and published --> F([Status: ok]) E -- otherwise, seam ON --> E1[Revision requested] E1 --> C E -- otherwise, seam OFF --> F D -- yes --> G{S5 stop_condition} G -- same search twice, seam ON --> H([Stop: stalled]) G -- no, or seam OFF --> I{S1 error_feedback} I -- tool failed, seam ON --> I1[Error returned to the model] I1 --> J{S4 publish_gate} I -- tool failed, seam OFF --> K([Stop: failed]) I -- tool ok --> J J -- out of band, seam ON --> J1[Publish denied] J1 --> I1 J -- in band --> L([Published, loop continues]) L --> C

The spec names are the ones used in SPEC.md:

• S1 error_feedback: return a tool failure to the model as an error

• S2 context_budget: trim the prompt until it fits the context window

• S3 verify_loop: check the final output at the end of the run

• S4 publish_gate: check the one call that has an outside effect (publishing)

• S5 stop_condition: stop the run when the same search repeats

The task itself is simple - read a document, write a digest of about 100 words, publish it - with two traps planted on purpose: a reference to a page 3 that does not exist (S1's trap), and the rule that the task is not complete until the digest has been published (S3's trap).

Two seams carry the whole run

Two of the five, when switched off, took completion from 100% to 0%. S2 first: the context window is set to 2,000 tokens, and after two pages the assembled prompt estimates 2,442 tokens (identical in all three runs).

With S2 on, old tool results are replaced by a short gist so the request still goes out. With S2 off, the harness refuses to send and the run ends.

That is naive and no_s2: all three runs stopped the same way, on the same wire.

S1 next. The trap page 3 does not exist, so the fetch always fails. With S1 on, that failure comes back to the model as a tool result and the model carries on with pages 1 and 2. With S1 off, the exception escapes the loop and the run dies on the spot.

That is no_s1: S2's trimming keeps the request alive, and all three runs still end at that point.

full : search->search->(trim)->(return error)->digest->publish->done 3/3 naive: search->search->(over budget, refuse to send)->stop 0/3 no_s1: search->search->(trim)->(error kills the run)->stop 0/3 

The token counts make the same point. full averaged 10,528 tokens per completed run; naive spent 2,089 and produced nothing.

The naive harness is not "cheaper but weaker" - it never arrives. The ~10k tokens of the harness are not a markup on the same work, they are the price of being able to do the work at all.

Three seams with no measurable effect (honest nulls)

The other three showed no effect at n=3. I am listing them instead of hiding them.

• S4 publish_gate: zero bad publishes in every cell. There was no bad publish for the gate to block

• S5 stop_condition: the same search with the same arguments never happened in any of the 21 runs, so the trigger never fired

• S3 verify_loop: every run that emitted a final message was inside the word band, so the end-of-run check never asked for a revision

The reason is simple: in most runs the model counted the words itself with check_word_count before publishing.

"A seam that never got a turn" is not the same as "a seam that is useless". n=3 cannot settle that difference.

n=3 is a trend check, not a statistical claim. That is all this section claims.

The failure none of the five seams catches

The most expensive failure in the grid escaped all five seams. It was run 2 of no_s3: the model called check_word_count four times while adjusting, and published an in-band digest. Then it never emitted a final message, and the loop burned to the 8-turn cap - 14,049 tokens - with nothing to show for it.

The publish had already happened, so S4 had nothing to block; there was no repeated search, so S5 never fired; and S3 only fires on a final message, so by construction it cannot catch this shape.

A loop-level "no progress, no finish" detector does not exist in these five seams. That is what the measurement says.

Related prior work measured a harness that shortens old tool results and responds to stalled work (arXiv:2608.26218). Its conclusion was that model and harness should be evaluated together, as one solver.

https://arxiv.org/abs/2608.26218

Honest limitations

• One model, one task, one tool set. These numbers describe this harness on this task, not harnesses in general

• The 2,000-token window is emulated harness-side. The run stops because the harness refuses to send, not because the provider returned a 400

• On a deployment with a wide window the naive loop would finish, and the comparison moves from "survives" to cost and quality

• The success criterion is mechanical (95-105 words and published). Whether the digest is any good is not measured

• n=3. The 67% cell (no_s3) flips on a single run

Wrapping up

• The scaffolding around the loop is worth building, but only two of the five seams did anything at n=3: fitting the context window and returning tool failures to the model

• The most dangerous failure (burning turns without finishing) lives outside those five seams. The next seam to add is a no-progress detector

• Every number is recomputed from the recordings. In the repo, python3 runs/analyze.py --check is the gate

https://github.com/sunnydachs/harness-showdown

This is a personal OSS project, provided without warranty. Use at your own risk — issues and improvement ideas are welcome.