The Take-Home Where the Candidate Must Break Their Own PR

Nine submissions sit in a folder. Every one compiles, every one has a test file, every one is fine. You score them all a 7 and learn almost nothing about how any of these people think when the code stops cooperating.

The problem is not the candidates. It is that the task rewards completion. Someone can read your prompt, satisfy each bullet, and never once be forced to decide what "correct" means when the happy path is gone.

So flip it. Ask for the fix, and ask for the test that kills the fix. Then grade how well they argued with themselves.

The prompt

Keep it to one screen. Everything below is the whole assignment.

Repo: <link>. GET /orders returns 200 with an empty list when the primary database is unreachable. Users see an empty account. Submit three files: 1. fix.patch — a patch against main. Small is fine. 2. attack.test — a test that FAILS on main and PASSES on your patch. It must not import or inspect your patch. 3. rubric.md — at most 200 words: what your attack proves, what it deliberately does not prove, and one thing a reviewer should try that would falsify your fix. Time box: 90 minutes. No new dependencies. Network is off during grading. 

The third file is where the signal lives. Anyone can write a green test. Far fewer people can tell you the boundary of their own evidence.

What you actually grade

Do not read the patch first. Read rubric.md, predict what the attack does, then read the attack. The gap between your prediction and their test is the measurement.

Notice what is missing: style, file layout, commit hygiene. Those are reviewable in fifteen minutes of conversation and are mostly noise at this stage.

The harness

Language-agnostic enough to adapt. It clones a pristine copy of the task repo, drops the candidate's attack in, and runs three checks: the attack must fail on untouched code, pass after the patch, and the existing suite must stay green.

#!/usr/bin/env bash # grade.sh — one submission in, one scorecard out. set -euo pipefail SUB="${1:?usage: grade.sh <submission-dir>}" REPO_URL="${REPO_URL:?set REPO_URL to the pristine task repo}" NAME="$(basename "$SUB")" WORK="$(mktemp -d)" mkdir -p scorecard run_tests () { # $1 = log label docker run --rm -v "$WORK/repo":/app -w /app --network none -e CI=1 \ node:20 bash -lc "npm ci --silent && npm test --silent" \ >"$WORK/$1.log" 2>&1 } git clone --quiet "$REPO_URL" "$WORK/repo" cp "$SUB"/attack.* "$WORK/repo/tests/" # A. The attack must PASS on untouched main — it reproduces a real bug. if run_tests attack_pristine; then A=pass; else A=fail; fi # B. Apply the patch. The attack must now FAIL, i.e. the bug is gone. git -C "$WORK/repo" apply "$SUB/fix.patch" if run_tests attack_patched; then B=fail; else B=pass; fi # C. Their patch must not break what already worked. rm -f "$WORK/repo"/tests/attack.* if run_tests suite_patched; then C=pass; else C=fail; fi printf '{"candidate":"%s","attack_reproduces_bug":"%s","attack_dies_on_fix":"%s","suite_still_green":"%s"}\n' \ "$NAME" "$A" "$B" "$C" | tee "scorecard/$NAME.json" 

Two rules keep this honest. Pin the base image and the lockfile, or check A and check B will disagree for reasons that have nothing to do with the candidate. And run each submission twice before you score it — a flaky attack is a finding, not a nuisance.

A reference answer, small on purpose

The bug is a lie in the shape of a 200. Downstream code cannot distinguish "you have no orders" from "we could not reach the database," so the fix is mostly about status codes and log shape.

// fix.patch (abridged) — routes/orders.js app.get('/orders', async (req, res) => { try { const rows = await db.query('select * from orders limit 50'); res.json({ orders: rows, next: null }); } catch (err) { req.log.error({ err }, 'orders_query_failed'); res.status(503).json({ error: 'orders_unavailable' }); } }); 

// tests/attack.test.js — fails on main, passes on the patch import request from 'supertest'; import { app } from '../app.js'; import { db } from '../db.js'; test('a dead database must not look like an empty account', async () => { db.failNext(new Error('ECONNREFUSED')); const res = await request(app).get('/orders'); expect(res.status).toBe(503); expect(res.body).not.toHaveProperty('orders'); }); 

A strong rubric.md for this patch says something like: "This proves we no longer return 200 when the pool rejects a connection. It does not prove anything about timeouts, partial reads, or the client's retry behavior — a reviewer should kill the DB mid-query, not before it, and check we do not double-charge on retry." That is a person who knows where their evidence ends.

Where a free model and a free server actually help

Grading is a batch job, and batch jobs want somewhere to run that is not your laptop. MonkeyCode offers free model access and a free server option, both operator-supplied availability — I have not audited the quotas, hardware, or how long either stays free, so treat them as conveniences to test, not as a capacity plan. The free server option is the useful half here: it gives the harness above a host that is not your machine, which matters when nine submissions each spin up a container and the fan turns into a jet engine.

The model half fits a narrow step. After checks A, B, and C return, you can ask a model to compare the candidate's rubric.md against their actual attack and flag contradictions — "claims to cover timeouts, test sets no timeout." That is a text-versus-text check with a human deciding the verdict, not an autograder. Anything that assigns the final score to a model is a different article and probably a mistake.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Failure modes you will see by round three

The attack asserts almost nothing. expect(res.status).not.toBe(200) passes on the patched code, on a crashed server, and on a 404 from a typo in the route. Ask for the exact status and the exact body shape.

The patch special-cases the test input. A patch that checks if (process.env.CI) is not a fix, it is a confession. Run check C to catch the collateral damage, and read the diff for branches keyed on test fixtures.

The fix deletes the feature. Making /orders return 503 always does satisfy every check above. Add one line to the prompt: the happy path test in the existing suite must stay untouched, and the patch may not modify it.

The rubric is modesty theater. "This is not production ready and could be improved" is not a limitation; it is a shrug with punctuation. A limitation names a specific failure the candidate chose not to handle.

Nothing fails on main. If check A returns fail, the candidate wrote a test for behavior that already works. Sometimes the bug is real but their repro is wrong; sometimes they fixed the bug and forgot the patch is part of the submission. Either way, one short reply sorts it out, and their answer tells you more than the test did.

When not to run this

It is a poor fit for junior hiring, where the goal is to see whether someone can get to a working solution at all, and a poor fit for roles that never touch failure paths — pure UI work with no async edges will feel like a trick question. It also costs you a harness you have to maintain, and a broken harness is worse than no take-home because it silently scores everyone the same.

If you already have a strong onsite loop, skip this entirely. The interview is faster and you can ask the falsification question out loud.

But if your submissions all look like sevens, this is the cheapest change you can make: keep the same repo, add one file to the deliverable, and let the harness do the reading. Run one submission through it this week before you rewrite anything else.