Builders Week is over, but a reader's suggestion left one question open.

Quick recap: Build Buddy asks a local Gemma 3 4B to fix C# compiler errors, then checks each fix by rebuilding and running the program. A fix only counts if it builds and prints the expected output.

On my six samples, Gemma scored 5/6. The one failure was CS0161: the fix compiled, but added two extra Console.WriteLine calls. A reader suggested treating that as a known failure case and changing the prompt. One sentence, "Keep the program's behaviour and console output exactly the same," took it to 6/6.

The problem with that result

I wrote the rule after seeing the six samples fail, and I tested it on the same six. That's the classic way to fool yourself. A fix tuned on the test can pass the test without being any good.

So I added four new samples the rule had never seen, each with one planted error and an expected.txt. I checked every answer key by running a hand-written correct fix first, because on Day 1 one of my own test cases was broken.

Held-out sample Planted error Should print
unassigned local CS0165: total used before it's set 12
max missing return CS0161: Max doesn't always return 7
int to bool CS0029: an int used as a bool True
string argument CS1503: "4" passed where an int is expected 16

Then I ran the same model on them, with and without the rule.

The result: no difference

Sample set Without the rule With the rule
Original six (the rule was written for these) 5/6 6/6
Four held-out samples 3/4 3/4

The rule fixed the problem it was written for, and on new problems it did nothing measurable. Not harmful, just narrow. That's what a held-out test is for.

Two failures that look nothing alike

CS0165 failed both times, in completely different ways:

  • Without the rule, Gemma "fixed" the loop by unrolling it into total += 3; total += 4; total += 5;, and still never set total = 0. Same error, new code.
  • With the rule, it returned the original broken code unchanged.

Same score, different mistakes. Looking only at the pass rate would have hidden that.

A wrong answer hiding inside working code

Without the rule, Gemma's CS0161 fix was correct: it added return b;. But it also added a comment:

// The method implicitly returns null if no explicit return statement is reached.

That's false: an int can't be null. The program prints 7, so the behaviour check passes, but anyone reading the comment learns something wrong. It's the same lesson as the wrong explanations earlier in the week: the code can pass while the words are wrong, and my checks only test the code.

What I'm taking away

  • Test a fix on cases it wasn't tuned on. Otherwise 6/6 can mean "fits these six," not "works."
  • Look at how things fail, not just how often. Two runs scored the same and failed differently.
  • Checking the code isn't checking the explanation. That's still an open problem for Build Buddy.

Limits

Four held-out samples, one model, one run each. Enough to stop me claiming the rule works in general; not enough to say much more.

Results: results-v2.csv and results-v2-keep-output.csv (held-out rows start with heldout-). Samples: samples-heldout.

Thanks again to the reader whose comments started all this.

How it was built: the held-out samples and this analysis were worked through with Claude; I ran every test on my laptop, and every number comes from those runs.