Before I Could Evaluate My Coding-Agent Harness, I Had to Isolate It

I started writing an article about the relationship between coding-agent models and the harnesses around them. I wanted to explain something I had learned from comparing a few configurations, but the results never gave me a clear enough answer to finish it.

Instead, I built yuurei, a CLI for running coding agents with isolated, versionable configurations, without swapping out my everyday global setup.

This is the story of the article I did not finish, the comparison behind it, and why I decided that isolation and evaluation should be separate tools.

Adapted from my Japanese article, originally published on September 14, 2026.

Was my harness holding the model back?

Here, by harness, I mean the workflow instructions I had accumulated in skills and files such as CLAUDE.md and AGENTS.md: understand the code first, review the design, stop when something looks risky, ask another agent to verify the result.

I wanted to try the newly released GPT-6 Astra, so I casually asked Codex, which I rarely used at the time, to refactor one of my open-source projects. The output was better than I expected.

Could this be better than what I was getting from Claude Code, where I had put much more work into the harness?

I ran three configurations, using the same repository, starting commit, and prompt:

pstack is a general-purpose collection of skills for coding agents, covering tasks such as investigation, design, and verification. “Only pstack” means I used those skills without my custom harness.

My hypothesis was that the harness might be setting a ceiling on the agent's work. Perhaps all those constraints were good at giving it reasons to stop, but less good at helping it look for a safe alternative after stopping.

That would have been something I could fix. I had written the harness.

The comparison did not support that hypothesis. In PR3, removing the custom harness narrowed the agent's exploration. It did not produce dedicated tests or reach a design decision.

The result was disappointing because the explanation I had hoped to find was in something I could change myself. Instead, differences between the models and execution environments remained possible explanations.

I ran each condition only once: n = 1 per condition. This was not enough to establish a general effect or separate model effects from runtime effects. What I could say was narrower: in this comparison, taking away my custom harness did not produce the improvement I expected.

I still wanted to keep comparing

An inconclusive result did not make the question less interesting.

Advice that helped an earlier model can become unnecessary as models improve. My harness had grown through repeated additions, and I could not say how much each part still contributed.

I wanted to keep comparing configurations so I could make my own development output more consistent. To do that, I needed the comparisons to be practical to repeat.

Even these three runs had been painful to set up.

A Git worktree only solves half the problem

For the repository, git worktree gave me separate working trees at the same starting point.

The harness was harder. Skills, instruction files, settings, and hooks were spread across several places. Each runtime had its own configuration locations and loading rules.

With Claude Code, I could point CLAUDE_CONFIG_DIR at another directory. But I could not simply assume that this isolated everything. I had to check the Setting sources section of /status each time to see what had actually been loaded.

I used procedures and scripts to switch out the custom parts of my Claude Code harness. Restoring my normal environment afterward still took time because I had not put everything back correctly.

I also started the Codex run assuming pstack would be absent because I rarely used Codex. Only afterward did I realize it had been active there too.

That was one comparison with three runs. Removing individual harness components would quickly give me ten or twenty variants. Every variant would add another round of backing up, swapping, restarting, and restoring configuration.

I could see myself getting stuck maintaining the comparison environment before I ever reached the thing I wanted to measure.

Making the harness a versioned input

That is why I built yuurei.

For each run, it creates a temporary execution cell and runs the coding agent with the selected profile, keeping the runtime's configuration separate from my normal global setup.

The basic model has three parts:

Profiles live under .yuurei/, so I can keep them in version control. For me, that is the most useful part: a harness becomes a named artifact I can preserve instead of a collection of files I continuously overwrite.

Here is a minimal example using the current CLI. It requires Node.js 22 or later, an installed Claude Code runtime, and an explicit authentication environment variable supported by the adapter; the README explains the authentication options.

npm install -g yuurei mkdir yuurei-demo cd yuurei-demo yuurei init yuurei doctor yuurei profile list yuurei run hello 

The default scaffold selects Claude Code and creates a small greeting task. It is a way to check the setup, rather than a refactoring experiment. Authentication must be ready before the run; an existing interactive login alone is not necessarily sufficient.

After a run, I can inspect its record using the run ID printed by the CLI:

yuurei trace show <run-id> 

Each run has a trace.json recording information such as the runtime, execution outcome, timeout status, profile and task digests, and isolation verification result. Logs and the trace are retained under .yuurei/runs/<run-id>/.

The digests let me check which configuration was used. They do not establish that the resulting code is correct.

That distinction mattered in the original comparison. The PR that looked most autonomous also incorrectly reported that its own checks had passed. An agent's report needs verification. So does the infrastructure I use to measure it.

There is also a specific limit to the word isolation here. Yuurei separates configuration and environment variables; its default isolation level uses a temporary HOME. It is not an OS sandbox, and code executed by the agent still has the access available to my user account. Likewise, reproducible configuration does not imply identical agent behavior across runs.

Why I separated isolation from evaluation

My first plan was to build one tool that both isolated runs and evaluated them.

I abandoned that plan because I had not worked out the evaluation design.

The differences that stood out in the original comparison concerned behavior, rather than simply the quality of the final artifact:

• Did the agent detect a risky change?
• Did it detect the risk and deliberately leave the change alone? That is different from never noticing it.
• After deciding against that change, did it look for another improvement?
• Did it hand the decision back to a human?

A PR diff cannot answer those questions on its own. I did not yet know how to capture them as structured observations.

If I put evaluation into the same tool, I would have to settle those questions before I could release the isolation mechanism. An unresolved evaluation design would block a piece of infrastructure whose responsibility was already clear.

I wanted the dependency to run the other way: an evaluator could use yuurei's execution records, while yuurei would not need to know which evaluator consumed them.

So I kept yuurei responsible for preparing isolated executions and emitting machine-readable records. A separate tool would decide what to evaluate and how to judge it.

The trace records execution facts. It does not turn those facts into a judgment that an agent made a good decision, nor does it automatically capture all the behavioral evidence a future evaluator might need.

This separation also has a practical benefit: evaluation criteria will change. What I consider useful behavior may change as models improve. I do not want every revision to those criteria to require rebuilding the isolation layer.

I could make the execution setup repeatable while leaving the evaluation questions open.

What the original version did—and did not do

At the time of the Japanese article, the v0.3 scope was single-runtime execution for Claude Code and Codex. Each runtime × model × harness × task combination was treated independently.

That scope did not include:

• Porting a harness across runtimes.
• Automatically scoring output quality.
• A cost or ROI dashboard.
• OS-level sandboxing.
• Team authentication.

There was also no concept for grouping repeated executions of the same profile and task. Agent behavior varies from run to run, so repetition would be necessary before drawing stronger conclusions.

Implementation update, November 5, 2026: I checked this adaptation against v1.4.0. Yuurei now also supports OpenCode. Later additions include experimental repository seeding (--seed-repo) and opt-in pre-run observation (--observe). Pre-run observation records information about the prepared environment; it does not resolve the behavioral evaluation questions above. These additions do not change the original experiment or its results. The current public contract describes the implementation's guarantees and limits.

The question I still wanted to test

One possibility remained open: could an accumulated harness prevent me from getting improvements that a newer model would otherwise provide?

The original comparison could not answer that. PR2 and PR3 both used Sonnet 5. I had changed the harness while keeping the model fixed. Instructions written to compensate for an older model's weaknesses might behave differently when the model itself changes.

Testing that requires the reverse comparison: preserve the harness and change the model.

It also requires the older harness to still exist when I want to run the comparison. If I keep editing it in place, it may already be gone by the time the next model arrives.

That was what yuurei was preparing me to do. I had not yet measured the effect I wanted to measure. I had built a way to preserve and rerun the configurations I would need to investigate it.