Benchmark Budgets Are Part of the Grader: A Paired Cost and Latency Gate for AI C++ Patches

Consider a pull request that turns a 12-line C++ change into a 40-line patch. Every golden case passes. CI is green. Two days later the review queue tells a different story: the patch needed three retries per case, pushed p95 compile time up by a third, and burned a quarter more tokens than the model it replaced. The grader never noticed, because it only asked whether the tests passed.

That gap is the subject of this article. The latest benchmarking challenge wrap-up is a useful reminder that leaderboards are snapshots, not budgets. For AI-generated C++ patches, a passing score has a second dimension: tokens, compile time, and wall-clock time per accepted patch.

The failure mode: a green eval with a red budget

A correctness-only harness treats every passing patch as equal. A patch that compiles in 200 ms and one that compiles in 2 s both get a check mark. A patch that uses 800 tokens and one that uses 8,000 tokens both count as one success.

That is fine for a smoke test. It is not fine for a merge gate. Token allowances are finite, CI minutes are finite, and reviewer attention is finite. When a model swap improves pass rate by two points but doubles the tail latency, the eval has found a trade-off; the harness should report it instead of hiding it.

The gate below is deliberately small. It does not replace correctness tests, prompt lock files, or flaky-case triage. It sits after them and answers one narrow question: did the candidate patch set get more expensive per accepted patch?

The artifact: a paired budget gate

The gate reads two JSONL files, one for the baseline and one for the candidate. Both must share a suite hash and the same case IDs. For each case, the script takes the median over repeats, requires every repeat to pass, and computes candidate-to-baseline ratios. It gates the 95th percentile of per-case wall and compile ratios, the median token ratio, and the 90th percentile of per-case token ratios. Queue wait is reported separately as a warning because shared capacity is noisy.

#!/usr/bin/env python3 '''budget_gate.py: paired cost and latency gate for AI C++ patch evals.''' import argparse import json import sys from collections import defaultdict from pathlib import Path from statistics import median REQUIRED = { 'run_id', 'case_id', 'variant', 'suite_hash', 'server_class', 'passed', 'prompt_tokens', 'completion_tokens', 'compile_ms', 'exec_ms', 'queue_wait_ms', } LIMITS = { 'p95_wall_ratio': 1.20, 'p95_compile_ratio': 1.25, 'median_tokens_ratio': 1.15, 'p90_case_tokens_ratio': 1.50, 'queue_wait_p95_ms': 30_000, 'min_paired_cases': 20, } def pct(values, q): values = sorted(values) if not values: return 0.0 k = (len(values) - 1) * q lo = int(k) hi = min(lo + 1, len(values) - 1) return values[lo] + (values[hi] - values[lo]) * (k - lo) def load(path): rows = [] for n, line in enumerate(Path(path).read_text().splitlines(), 1): if not line.strip(): continue try: row = json.loads(line) except json.JSONDecodeError as exc: raise SystemExit(f'{path}:{n}: {exc}') missing = REQUIRED - set(row) if missing: raise SystemExit(f'{path}:{n}: missing {sorted(missing)}') rows.append(row) return rows def by_case(rows, variant): grouped = defaultdict(list) for row in rows: if row['variant'] == variant: grouped[row['case_id']].append(row) return grouped def case_stats(rows): return { 'passed': all(bool(row['passed']) for row in rows), 'wall': median(float(row['compile_ms']) + float(row['exec_ms']) for row in rows), 'compile': median(float(row['compile_ms']) for row in rows), 'tokens': median(float(row['prompt_tokens']) + float(row['completion_tokens']) for row in rows), 'queue': median(float(row['queue_wait_ms']) for row in rows), } def ratio(candidate, baseline): return candidate / baseline if baseline > 0 else float('inf') def main(): parser = argparse.ArgumentParser() parser.add_argument('--baseline', required=True) parser.add_argument('--candidate', required=True) parser.add_argument('--report', default='budget-report.json') args = parser.parse_args() base = load(args.baseline) cand = load(args.candidate) for name, rows in (('baseline', base), ('candidate', cand)): hashes = {row['suite_hash'] for row in rows} if len(hashes) != 1: raise SystemExit(f'{name}: mixed suite_hash {hashes}') if {row['suite_hash'] for row in base} != {row['suite_hash'] for row in cand}: raise SystemExit('suite_hash mismatch: baseline and candidate are not comparable') base_cases = by_case(base, 'baseline') cand_cases = by_case(cand, 'candidate') paired = sorted(set(base_cases) & set(cand_cases)) missing = sorted(set(base_cases) - set(cand_cases)) extra = sorted(set(cand_cases) - set(base_cases)) if missing: raise SystemExit(f'candidate dropped cases: {missing[:10]}') need = LIMITS['min_paired_cases'] if len(paired) < need: raise SystemExit(f'need at least {need} paired cases, got {len(paired)}') ratios = {'wall': [], 'compile': [], 'tokens': []} candidate_queue = [] failures = [] for case_id in paired: b = case_stats(base_cases[case_id]) c = case_stats(cand_cases[case_id]) candidate_queue.append(c['queue']) if not b['passed']: failures.append(f'{case_id}: baseline did not pass') continue if not c['passed']: failures.append(f'{case_id}: candidate regressed correctness') continue for key in ratios: ratios[key].append(ratio(c[key], b[key])) if failures: report = {'verdict': 'FAIL', 'reason': 'correctness/coverage', 'failures': failures[:20]} Path(args.report).write_text(json.dumps(report, indent=2)) print(json.dumps(report, indent=2)) return 1 checks = { 'p95_wall_ratio': pct(ratios['wall'], 0.95), 'p95_compile_ratio': pct(ratios['compile'], 0.95), 'median_tokens_ratio': median(ratios['tokens']), 'p90_case_tokens_ratio': pct(ratios['tokens'], 0.90), 'queue_wait_p95_ms': pct(candidate_queue, 0.95), } verdict = 'PASS' reasons = [] warnings = [] for key, limit in LIMITS.items(): if key == 'min_paired_cases': continue if key == 'queue_wait_p95_ms': if checks[key] > limit: warnings.append(f'{key}={checks[key]:.0f} > {limit}') elif checks[key] > limit: verdict = 'FAIL' reasons.append(f'{key}={checks[key]:.3f} > {limit}') report = { 'verdict': verdict, 'checks': {key: round(value, 3) for key, value in checks.items()}, 'limits': LIMITS, 'paired_cases': len(paired), 'extra_cases': extra, 'reasons': reasons, 'warnings': warnings, } Path(args.report).write_text(json.dumps(report, indent=2)) print(json.dumps(report, indent=2)) return 0 if verdict == 'PASS' else 1 if __name__ == '__main__': sys.exit(main()) 

A record is one JSON object per line. The minimum fields are shown as a Python template so the shape is explicit; the harness serializes this as JSONL.

record = { 'run_id': '2026-11-07T10:00:00Z', 'case_id': 'cpp-vector-reserve-01', 'variant': 'baseline', 'suite_hash': 'sha256:...', 'server_class': 'free', 'passed': True, 'prompt_tokens': 412, 'completion_tokens': 188, 'compile_ms': 310, 'exec_ms': 42, 'queue_wait_ms': 1800, } 

The run command is intentionally boring:

python3 budget_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --report budget-report.json 

A failing report should be readable in one glance:

verdict=FAIL p95_wall_ratio=1.31 (limit 1.20) median_tokens_ratio=1.08 (limit 1.15) queue_wait_p95_ms=41200 (warn at 30000) 

How the gate decides

The ratios are paired by case, not calculated from aggregate totals. That matters. A candidate can keep the same total token count by trading a cheap case for an expensive one. The paired p90 catches tail hogs; the median catches broad quota burn; the wall and compile gates catch review latency that a pass-rate table never shows.

The queue warning is separate for a reason. On a shared free server, a 40-second queue wait is a capacity signal, not a model-quality signal. It should not fail correctness, but it should stop a team from comparing a laptop baseline with a remote candidate and calling the difference a regression.

A five-step runbook

1. Freeze the suite hash and write it into every row. Do not compare across suite edits; that is a different benchmark with different accepted-patch economics.
2. Run the baseline with at least three repeats per case on the intended server_class. Record the prompt hash and model ID if the harness already has them; the budget gate only needs the fields above.
3. Run the candidate in the same measurement window. For MonkeyCode's free server option, record queue_wait_ms; shared capacity can add minutes that are not model execution time.
4. Emit JSONL and run the gate. Fix correctness failures before reading budget ratios. A candidate that fails two golden cases has no meaningful median token ratio.
5. Archive the report next to the prompt lock file. A later model swap can then be compared with a known budget instead of a memory of one.

Where MonkeyCode fits

MonkeyCode's open-source project is relevant to this workflow because the candidate side can run on its free model access and free server option. The operator-supplied offer at the time of writing includes a 10-million-token free allowance and a free server option. Quotas, capacity, and duration are not permanent; verify current terms before sizing a run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The cost model changes, but the gate does not. Free tokens still run out, and free shared servers still queue. The useful move is to keep baseline and candidate on the same server class, record queue wait separately, and treat a budget regression as a failed eval even when every test is green. The free allowance is enough to try the workflow on a small suite; check the current terms before scaling it.

Limitations and who should not use this

The gate has sharp edges. Token ratios are only comparable when both variants use the same tokenizer; otherwise compare billable units or quota consumption, not raw token counts. Compile time depends on flags, cache state, and machine class, so pin those or the ratio measures the host. Shared free capacity adds variance even with repeats; run baseline and candidate in the same window when possible.

It also starts after correctness. If the candidate fails a golden case, the budget report is noise. If the suite has fewer than 20 paired cases, tail percentiles are unstable; use the gate as a directional check, not a hard merge block. Teams without per-case token and timing telemetry should add that telemetry before adopting the gate. A one-off patch review with no model choice does not need it. And no shared free server should be used to promise a hard real-time SLO; use the free option to find regressions, not to guarantee latency.

What to do next

Run the gate against the last two candidate models on a 20-case suite. A 1.2x p95 wall regression with a flat pass rate is a real finding. So is a 0.98 pass rate with a 2x token burn. Publish the budget report next to the eval score; the pair is more useful than another leaderboard screenshot.