My agents kept dropping what resisted. 10 of my 19 vetoes were about that.

On September 30, an agent was building a catalogue of every setting in our back-office, nearly three thousand keys. The goal I had set that day: each key either recorded with the command that writes it, or dropped with a verified proof, and none left open. Dropping a key meant declaring that no customer action could ever write it, so there was no command to find. Its score counted a dropped key the same way as a recorded one.

That afternoon, the score climbed two points in half an hour, and most of the keys it closed were drops: the keys of two retired modules, with a proof from the code and no second look. A few minutes later, the agent proposed a few dozen more drops, each with its proof. Those came to me as a question and weren't counted.

My answer the next morning was that a second agent would check both batches, the drops already counted and the ones waiting for me. Most of the new proposals held up. A handful were reachable after all, and a few stayed open as uncertain. The first batch was reachable too, through a path the first reading had missed. The score went down, then back up once the agent had recorded those keys for real, which for the retired modules meant creating their sections on a test app and deleting them afterwards.

In the previous post I wrote about the six days a fleet of coding agents then spent on the refactor, and said that ten of the 19 precedents I rejected were about one thing. It's the same thing as on September 30, with more agents.

Why it kept happening

Both times the target was completeness, and both times it accepted a drop. In the catalogue, a dropped key counted as closed. In the refactor, every unit had the same exit criterion for coverage: each of its controls linked to an option of a function, or excluded with a written reason in the unit's spec, the file that declares its functions. Excluding a control was the refactor's drop: the unit stopped answering for it. The rules also said, from before the first unit, that excluding a control or a unit was a question for me.

Only the criterion had a check. The checks a unit had to pass skipped any control its spec listed as excluded, and until day five nothing read the reasons. So when something resisted, whether a control the test bench couldn't play, a clean fix that sat outside the unit or a case that looked out of scope, the shortest way to a green unit was one more line in the spec. From inside the unit, that line looked like finishing the job rather than a decision about scope.

The ten I rejected

How a precedent worked: when a worker met a case like one I had already answered, it applied the same answer, and I validated or rejected it afterwards. The ten exclusions I rejected came from seven different workers, every day from Saturday to Wednesday:

Three of them took an exclusion I had validated earlier as their precedent, and every one came with a reason. What I kept asking back, in one form or another, was which unit would handle the thing once it was dropped. My other rejections were mostly requests for more context, and every rejected case went back to work.

On day five, one of the workers went through every control the units had left outside their functions. Some six hundred were accounted for: another function carried them, or they wrote nothing, so they had no effect on the app. More than five hundred were still on the old code path we were retiring, and no decision of mine covered any of them. The question that reached my dashboard offered three options, each with what it would leave behind: cover everything, cover only the most used and riskiest and list the rest, or accept all of them as limits of this phase. It recommended covering everything, and that's what I chose.

What held up

What worked was handling a drop like a result: it needs a proof, and the proof can't rest on the word of the agent that made the drop.

The catalogue is where it worked first. Every key is accounted for: recorded with its command, or dropped. Each drop says why, points to the file and line in the code that proves it, and has been checked, by a second agent for most of them and live on a test app for the others. The coverage figure counts only the recorded keys, and the drops sit in their own file next to it.

In the refactor, the same idea became code on day five, right after the audit. A check reads the reason of every excluded control and accepts three answers: another function carries it, and the check finds the binding in the code; a decision of mine covers it, and the check finds that decision by its number; or it has no effect on the app, said in one of a few exact phrasings. That third answer still rests on the worker's own sentence, only in a form I fixed, and it covers about half of the excluded controls. Anything else fails. At the end of the run, no excluded control was left without a valid answer, and nearly all of them were either carried by another function or had no effect on the app.

What changed for me

• A drop is accepted once it has a check that doesn't rest on the word of the agent that made it. In the catalogue that held for every key from the next morning on. In the refactor it held once the check was in code, except for "no effect", which still takes the worker's word in a fixed form.
• "Impossible" is a drop too. Since then, a case is impossible only with its own execution trace in the code, file and line. An argument about a whole family of cases doesn't count.
• Next time the exclusion check exists before the first unit. Here it arrived on day five, though the rule had been in writing since before day one. And "no effect" will need a proof of its own.

All of this ran on test apps, and nothing goes to production without me.

If you run agents against a coverage or completeness target, what checks the things they leave out?