"All fixed." I've heard that from my AI agent, opened the app, and found the same margin more times than I'd like. (I wrote this with help from Claude and checked every number myself.)
I do most of my work with Claude Code now, and the agent often finds problems on its own. It says something like "this is also broken, want me to fix it?" And almost every time, my reply is the same question: "So what goes wrong if we leave it, and what gets better if we fix it?" When it says it fixed something, I ask: "Really? How did you check?" I got tired of asking, so I made a skill for it.
The result first
Same situation, same model (Haiku). The agent has fixed a search box and is reporting whether its promise, "requests drop from 12 to 1", came true.
Without the skill, it admitted the count only dropped to 2. Then it added "if typing doesn't pause, it drops to 1", which it never measured. With the skill, it marked that row ❌ and only explained where the "1" came from.
The skill does two things:
- When proposing a fix, fill four fields: what's wrong, how to fix it, what gets better, and the expected result.
- After the fix, check every promise with a measured as-is / to-be table.
The field that matters most is the expected result. Instead of "it'll be more stable", the agent has to write something you can check by running it, like "select 1 returns one row". That sentence later becomes a row in the to-be table.
It started with one bug
In a team project, the agent found a problem in a data script. A function that reads the database in read-only mode was missing a CLI flag, so it stopped and asked for confirmation whenever an agent ran it. The location and the fix were both right.
What it didn't say was why it mattered, so I asked. The answer was bigger than the report:
- two scripts that use this function had never run in an agent session
- the "read-only" function let a table-creating statement (
select … into) straight through
Neither was in the first report. If I hadn't asked, I'd have filed it as "one missing flag" and moved on.
That "all fixed" from the top had the same root. It claimed a fix without a measured "before" to compare against.
I measured my history before writing the skill
One annoying evening isn't enough reason for a rule, so I checked whether this was a real pattern.
I collected everything the agent had sent me over about 40 days, across 87 projects. That came to 248 bug or fix proposals and 326 "fixed it" reports. I sampled 40 of each at random and had Sonnet grade them.
82% said where the problem was and how to fix it. Only 29% said what would get better, and only 18% gave a result you could check. After a fix, about one report in four compared against a measured "before".
So I wasn't being picky. (Okay, maybe a little.)
Did it actually help? 72 answers
Writing the skill took minutes. Knowing whether it changed anything took longer.
I wrote six situations:
- three proposals: the data script bug, a web search box that sends a request for every keystroke, and an Android label that shows "yesterday" for something that happened this morning
- three reports after fixing each of them
Each report hides a trap: a promise from the proposal that the logs show wasn't met, or was never measured. The goal was to catch answers that draw a nice table and tick every box.
Every situation ran once without the skill and once with SKILL.md added, on Sonnet and Haiku, three times each. That's 72 answers. My global settings already include this rule, so every run had settings and CLAUDE.md turned off.
With the skill, "what gets better" and "expected result" showed up in all 18 runs, and every trap was caught. "Doesn't claim what wasn't measured" stopped at 78%. Some habits survive a skill, it seems.
The part I liked most: Haiku with the skill scored higher than Sonnet without it, at about one eighth of the cost per answer.
Making the grading trustworthy
A model grading a model is easy to fool, so I added two checks:
- Quoted evidence. For each item, the grader has to quote the answer. If the quote isn't really in the answer, code flips that item to a fail.
- A second grading. The same 72 answers were graded again with a different prompt.
The two gradings agreed on 446 of 468 judgments (95.3%), and the second one alone gave the same conclusion.
Using it for real
This is the report I got after re-proposing that first bug in the new format and approving it. The "before" values came from temporarily reverting the fix and running the same commands.
My favorite row is the last one. Nobody ran it from a human terminal, so it stays ⚠️ instead of ✅. Before, I'd probably have gotten "all done".
What I took away
My part in this was setting the direction and deciding at each fork.
The biggest lesson was a small one. Once "what gets better" became a field, the agent filled it every time. I had just never said I wanted it.
What do you keep asking your agent over and over? It might fit in one field too.
GitHub: https://github.com/ESeungJun/fix-proposal
LinkedIn: https://www.linkedin.com/in/승준-이-53707a328/
