Your LLM judge can cost 9.5x the chatbot it grades. Grade 400 replies a day, not all of them.

You shipped a support chatbot on GPT-5.6 mini, and now you want to know how often it answers wrong. The usual advice is to let a stronger model grade every reply. I priced that setup: each grade costs 9.5 times the reply it grades. At 10,000 replies a day, the bot costs $132 a month to run and $1,260 a month to watch. Most of those grades tell you little that 400 wouldn't.

Grading a reply costs more than writing it

Model APIs bill per token, a chunk of text about three-quarters of an English word. Input tokens are what you send and output tokens are what the model writes back; output costs more. Prices are quoted per million tokens, and the ones vendors publish themselves are called list prices.

An LLM judge is a second model that reads a reply and grades it against your instructions: did the bot answer the question, did it make up a refund policy. Three things make it expensive. It reads everything the bot read, plus the reply, plus the grading instructions. It is usually a stronger model, because a judge no smarter than the bot tends to miss the bot's mistakes. And the usual setup calls it on every reply.

My numbers are assumptions, not measurements. The bot reads 1,000 tokens and writes 200. The judge reads those 1,200 plus 300 tokens of instructions and writes a 100-token verdict. List prices come from the catalog of the LLM gateway I work on, read on 26 October 2026: $0.20 / $1.20 per million for GPT-5.6 mini, $2 / $12 for GPT-5.6. A reply costs $0.00044. Its grade costs $0.0042.

Precision depends on the count, not on the traffic

The user already has the reply, so grading it fixes nothing. What you're after is one number: the share of bad replies. A random sample, a few replies picked by chance, measures that number too. How close it lands is the margin of error: 95% of the time, what you measure is within that many points of the true rate. For a 5% failure rate it equals 1.96 × √(0.05 × 0.95 / n), where n is the number of graded replies. Your traffic isn't in the formula.

Four times the grades buy half the margin. Grading every reply instead of 400 costs 25 times more and narrows "somewhere between 2.9% and 7.1%" to "between 4.6% and 5.4%". At a million replies a day, the same 400 grades still give ±2.1 points, and the judge shrinks to 0.4% of the bot's bill.

Grade a fixed number a day

So pick the precision first, turn it into a daily count, and let the chance of grading follow your traffic:

import math, random GRADES_PER_DAY = 400 def should_grade(replies_yesterday: int) -> bool: """Pick about GRADES_PER_DAY random replies a day, whatever the traffic.""" return random.random() < min(1.0, GRADES_PER_DAY / max(replies_yesterday, 1)) def margin(failure_rate: float, graded: int) -> float: """How far the measured rate can be from the true one, 95% of the time.""" return 1.96 * math.sqrt(failure_rate * (1 - failure_rate) / graded) for n in (200, 400, 1600, 10_000): print(f"{n:>6} grades a day: 5% ± {margin(0.05, n):.1%}") 

It prints the margin column of the table. Call should_grade after each reply with yesterday's reply count, and send the chosen replies to the judge in the background, so no user waits for it. Don't grade the first 400 replies of the day instead: people who write at night ask different questions. If traffic doubles before the count catches up, you grade 800 and pay $3.36 for the day instead of $1.68. Need tighter numbers? Pool a week: 2,800 grades give ±0.8 points at the same daily cost. On a small bot with 2,000 replies a day, 400 grades still cost 1.9 times the bot itself; settle for 200 and ±3 points if that's too much.

What this doesn't solve

A sample measures the rate; it doesn't stop a bad reply. If one must never reach a user, you need a check before every reply goes out, paid on every reply. Rare failures slip through too: at a true rate of 0.1%, two days out of three a 400-reply sample contains none, so catch things like a leaked email address with a cheap code check over all traffic. The judge is a model as well; compare it with 50 replies you graded by hand before trusting its number. And no gateway does this for you: the one I work on logs each call's status code, and a 200 means the model answered, not that it answered right.

The takeaway

Decide how precisely you need to know your failure rate, turn that into a daily number of grades, and grade that many random replies. Your traffic will grow; the judge bill doesn't have to.