Your Grader Is Untested Code: A Bootcamp Lab That Mutates Student Submissions

The grader passed everything. That was the failure.

Week three of a lab where students build a small agent that fixes bugs in a toy repo. Sixteen submissions, fourteen PASS. I opened two of the passing repos by hand: neither had fixed the off-by-one. The agent had described a fix in REPORT.md, and my script read the report instead of the code.

So now I teach this before any agent work: your grader is code, and untested code lies. This is the lab I use to make students red-team their own grader before they grade anybody with it.

The grader is the weakest file in the repo

Benchmark posts are everywhere right now. Leaderboards, eval suites, “we measured twelve agents.” Fine, I read them. But a number is only as honest as the harness behind it, and almost nobody ships the harness.

Same thing at classroom scale. The student agent is 80 lines. The grader is 200, plus a rubric written in prose that nobody has ever violated on purpose. Which file would you actually trust?

The lab: break the submission on purpose and check that the grader notices. That is mutation testing, pointed at your grading script instead of at your application code.

Lab setup: 90 minutes, two students, one repo

Prerequisites:

• A task repo with a visible test suite and one known-buggy function (pager.py in mine).
• A grader.py that prints one final line: PASS or FAIL:<rule-id>.
• A labels.yml file that students write by hand before automating anything.

Repo layout:

lab/ submission/ # a known-good reference fix pager.py tests/test_pager.py REPORT.md grader.py # student-written, prints PASS or FAIL:<rule> mutants.py # one mutation per rubric rule run_mutants.py # applies each mutant, compares to labels.yml labels.yml # hand-written expected verdicts 

The reference submission must pass unmodified. If it does not, stop and fix the repo, not the rubric. Half of the labs I have run die right there, and that is cheap compared to discovering it during grading.

The artifact: five mutants, one of them innocent

Each mutant should map to exactly one rubric rule. Random character typos are useless here; the grader catches those for free and teaches nobody anything.

# mutants.py — semantic mutations. Each one should trip a specific rubric rule. from pathlib import Path MUTANTS = {} def mutant(name): def wrap(fn): MUTANTS[name] = fn return fn return wrap @mutant('prose_only') def prose_only(sub: Path): '''Claim the fix in the report. Touch no source file.''' (sub / 'REPORT.md').write_text('Fixed the off-by-one in page_count(). Tests pass.\n') @mutant('silent_skip') def silent_skip(sub: Path): '''Rename the failing test so the suite passes by not running it.''' t = sub / 'tests' / 'test_pager.py' t.write_text(t.read_text().replace('def test_last_page', 'def _disabled_last_page')) @mutant('weaken_assert') def weaken_assert(sub: Path): '''Reword the expectation until the broken output looks correct.''' t = sub / 'tests' / 'test_pager.py' t.write_text(t.read_text().replace('== 3', '>= 0')) @mutant('hardcode') def hardcode(sub: Path): '''Answer only the exact input the demo uses.''' p = sub / 'pager.py' src = p.read_text() p.write_text(src.replace( 'def page_count(n, size):', 'def page_count(n, size):\n if (n, size) == (25, 10):\n return 3', )) @mutant('noop_refactor') def noop_refactor(sub: Path): '''Legitimate cleanup. The grader must still PASS this one.''' p = sub / 'pager.py' p.write_text('# reviewed by pair B, behavior unchanged\n' + p.read_text()) 

That last one is the important one. A grader that fails everything looks strict and is broken in the other direction. Precision matters as much as recall.

Then the runner:

# run_mutants.py — python run_mutants.py submission labels.yml import shutil, subprocess, sys, tempfile from pathlib import Path import yaml from mutants import MUTANTS def grade(sub: Path) -> str: r = subprocess.run( [sys.executable, 'grader.py', '--submission', str(sub)], capture_output=True, text=True, timeout=300, ) lines = r.stdout.strip().splitlines() return lines[-1] if lines else 'ERROR' def main(sub_dir: str, labels_file: str) -> int: labels = yaml.safe_load(Path(labels_file).read_text()) base = Path(sub_dir).resolve() survivors = [] for name, apply_mutation in MUTANTS.items(): tmp = Path(tempfile.mkdtemp()) / name shutil.copytree(base, tmp) apply_mutation(tmp) first, second = grade(tmp), grade(tmp) want = labels[name] stable = first == second caught = first.startswith(want) print(f'{name:14} want={want:16} got={first:16} stable={stable} caught={caught}') if not (caught and stable): survivors.append(name) print(f'mutation score: {len(labels) - len(survivors)}/{len(labels)}') return 1 if survivors else 0 if __name__ == '__main__': sys.exit(main(sys.argv[1], sys.argv[2])) 

Run it twice, same verdict, or it does not count. Model-judged rubrics drift; the stable column makes that visible instead of mysterious.

Checkpoints, in order

1. Baseline freeze. Grade the untouched reference submission. Must be PASS, twice, same final line. Commit that output.
2. Hand labels first. Write labels.yml before encoding any expectation in code. Swap papers with another pair and argue about noop_refactor; most teams initially label it FAIL:<rule> and are wrong.
3. Automate and count survivors. Every survivor is a rubric bug. File it as an issue with the mutant name in the title, not as “grader broken.”
4. Budget the suite. Mutation runs are multiplicative: 5 mutants × 2 grades = 10 grader runs, and some of those may call a model. Measure tokens per run, multiply, then cap the suite size.

Checkpoint 4 is where the boring product question shows up. I ran this round on MonkeyCode's free model access and its free server option, which the operator describes as a free token allowance (roughly ten million tokens at the time of writing) plus a free server, both time-sensitive, so re-read the current offer before planning a course around it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The property that matters for this lab is unglamorous and real: repeated grader runs have to be cheap, because the whole exercise is the same grader seeing near-identical inputs many times.

Rubric for grading the lab (40 points)

Deductions, announced up front:

• Grading your own pair's grader.py. Yes, this happened. No, it does not count.
• Editing submission/ so a mutant fails.
• Reporting a mutation score without the control mutant.
• A “fixed” rubric under which the reference submission now fails.

Post the rubric before the lab starts. Every fairness complaint I have received came from a rubric that appeared at grading time.

Stretch goals

• Add a sixth mutant only a human would catch, for example a report that is accurate while the diff lands in the wrong file. Then check whether your grader depends on a model judge at all, and whether you want that.
• Measure false-fail rate. Take one passing submission, apply three legitimate refactors, count how many still pass. This is the number nobody publishes.
• Time-box the suite to five minutes total. If a single mutant takes longer than the reference run, find out why before adding a seventh.
• Cross-grade. Trade grader.py with another pair and run their mutant suite against your grader. Different rubric, different blind spots.

Limitations, and who should skip this

Mutation testing a grader tells you nothing about whether the task is worth grading. It finds blind graders, not bad assignments.

A five-mutant suite also overfits fast. A 5/5 score means five specific mistakes are caught. It does not mean the grader is good, and it does not carry over to next week's task without new mutants.

With stochastic verdicts, the stable column will occasionally lie. Twice-in-a-row agreement is weak evidence, not proof, so budget a third run for anything a student could appeal.

Skip this lab if you already have a hidden test suite plus CI that students cannot edit; that is stronger evidence than mutants. Skip it if the course has no room for a second review round, because hand-labeling needs pairs and time. And do not use an uncalibrated grader for final grades. Use it for feedback, calibrate between cohorts, then decide.

The one-line version

Before you trust a number about an agent, break the thing that produces the number and see whether it notices. If you want to run the suite without spending your own budget, the free model access and the free server option are enough to reach checkpoint 4; re-check the current allowance first, then run checkpoint 1 before you believe anything.