A user emailed me a screenshot of two of their reports side by side. One session they'd nailed. One they'd bombed so badly they quit halfway. Both said "7/10."

That's when I learned my LLM-as-a-judge wasn't judging anything. It was printing a seven with extra steps.

I pulled every score the system had ever produced: 1,140 scored answers. 41% were a 7. 27% were an 8. Anything below 5 showed up 3% of the time. My carefully written grading prompt had become a very expensive random number generator with a strong opinion about the number 7.

Here's what I measured, the four fixes I tried, and the one that actually worked.

TL;DR

  • LLM-as-a-judge on an unanchored 1-10 scale collapses into 7s and 8s. Mine put 68% of 1,140 answers there.
  • Telling the model to "be strict" doesn't fix it. It moves the whole pile down one point. Ranking quality barely changed (Spearman 0.41 → 0.44).
  • A 1-5 scale with a written description for every level jumped agreement with my hand grades to 0.68.
  • Binary checklist items, summed in code, hit 0.79, close to the 0.83 agreement between two human graders. Rerun noise dropped from 0.9 to 0.31 points.
  • The cost: scoring got ~75% more expensive per session, and the checklist still mis-scores short, unusual answers.

What was I actually scoring?

The system is Preterview, the interview practice platform I run (full disclosure: I built it). It runs a voice interview, transcribes each answer, and an LLM scores it on a few dimensions before writing a report. If the scores are mush, the report is mush, and the whole product is a chatbot that says "nice job."

Preterview — an interview session in progress

The original judge prompt was what most people write the first time:

Score the candidate's answer from 1 to 10 on: - clarity - specificity - relevance Return JSON: {"clarity": int, "specificity": int, "relevance": int, "feedback": str}

Clean. Readable. Useless.

Why does LLM-as-a-judge cluster scores at 7 out of 10?

Because a number with no definition defaults to the model's prior for "fine," and that prior is about 7. Without anchors, "6" and "8" don't mean anything specific, so the model reaches for the safe middle-high answer that sounds supportive without sounding fake.

A few things stack up:

  1. No anchors. What's the difference between a 6 and a 7 for "specificity"? I never said. The model can't either.
  2. Leniency. Instruction-tuned models are trained to be helpful and pleasant. Handing a struggling person a 3 feels rude. A 7 never offends anyone.
  3. Too many levels. Ten buckets is more resolution than either of us can use consistently. The extra buckets just become noise around the mode.

Here's what my production distribution looked like before any fixes:

How do you test whether an LLM judge is accurate?

Build a small hand-graded calibration set and measure rank agreement against it. You don't need thousands of examples. You need a set where you already know the right ordering.

I took 60 real answers (with permission, anonymized) and graded them myself on a 1-5 scale, deliberately picking 12 per level so the set was spread out. A friend who has run a lot of hiring loops graded them blind. Our agreement (Spearman ρ) was 0.83. That's the ceiling. No judge prompt should be expected to beat two humans agreeing with each other.

Then I ran every prompt variant over all 60 answers, five times each, and tracked three numbers:

  • Spearman ρ against my grades (does it rank answers correctly?)
  • Share of scores at the mode (is it collapsing?)
  • Rerun standard deviation (does the same answer get the same score?)

The baseline prompt on the calibration set: ρ = 0.41, and 71% of answers got a 7 or 8 even though I'd picked the set to be evenly spread. Rerun std dev: 0.9 points. One answer scored 5, 7, 8, 7, 6 across five runs. Same text, same prompt.

Fix 1: "Be strict." Did it work?

No. It shifted every score down about one point and left the ranking almost unchanged. ρ went from 0.41 to 0.44, which is inside the noise.

I added "You are a demanding senior interviewer. Most answers deserve below 6." The mode moved from 7 to 6. That's it. The pile slid left as one block. Bad answers and good answers were still crammed together, just at a lower altitude.

This is the most common fix I see recommended and it's basically a cosmetic filter. You get scores that look harsher, which feels like rigor. It isn't.

Fix 2: A 1-5 scale with anchored levels

This was the first real improvement: ρ jumped to 0.68. I cut the scale to five levels and wrote a concrete, observable description for each one.

specificity: 1 = No concrete example. Speaks in generalities ("I'm a team player"). 2 = Mentions a situation but no actions or outcome. 3 = Situation and actions, but the outcome is vague ("it went well"). 4 = Situation, own actions, and a concrete outcome. 5 = All of 4, plus a measurable result and what they'd do differently.

The trick is that each level describes something the model can check in the transcript, not a vibe like "good" or "excellent." A 3 now means a specific thing is missing.

Fix 3: Anchors plus reference answers

Adding two graded example answers per dimension pushed ρ to 0.74. I included one real answer I'd graded a 2 and one I'd graded a 4, with a one-line explanation each.

The downside: the prompt grew from about 1,100 tokens to about 3,400. Since that block is identical on every call, it caches well, so the real cost increase was small. The bigger issue was that the model started pattern-matching to the examples: answers that looked like my "4" example got 4s even when the content was weaker.

Fix 4: Binary checklists, scored in code

The winner: ρ = 0.79, rerun std dev 0.31, and no pile-up at any single level. Instead of asking for a number, I asked the model a set of yes/no questions and computed the score myself.

from pydantic import BaseModel class Check(BaseModel): passed: bool evidence: str # exact quote from the transcript, or "" if none  class SpecificityChecks(BaseModel): names_a_specific_situation: Check describes_own_actions: Check # "I", not "we"  states_an_outcome: Check outcome_is_measurable: Check includes_a_reflection: Check WEIGHTS = { "names_a_specific_situation": 1, "describes_own_actions": 1, "states_an_outcome": 1, "outcome_is_measurable": 1, "includes_a_reflection": 1, } def score(checks: SpecificityChecks) -> int: total = sum( WEIGHTS[name] for name, c in checks if c.passed and c.evidence.strip() ) return max(1, total) # 0-5 checks → 1-5

Two details mattered more than I expected:

  • Require evidence. A check only counts if the model quotes the transcript. Without that rule, the model happily marked "states an outcome" as true for answers that didn't have one. Forcing a quote killed most of the generous passes.
  • Yes/no is easier than 1-10. "Did they state a measurable result?" has a right answer. "How specific was this, out of 10?" doesn't. Models are far more consistent on the first kind of question, which is exactly why rerun noise dropped by two thirds.

What did the checklist approach break?

Short, unusual, genuinely strong answers. On the calibration set, 5 of 60 answers came out 2 or more levels off my grade, and 4 of those 5 were concise answers from strong candidates.

One answer was two sentences: a precise problem, a precise fix, a number. I graded it a 5. The checklist gave it a 3 because there was no "reflection" and the actions were implied rather than spelled out with "I did X." A checklist rewards structure, and structure correlates with length. I haven't fully solved this. My current patch is a separate check for "complete despite being brief," and it helps on about half of those cases.

Other honest costs:

  • Price. Evidence quotes mean more output tokens. Scoring went from about $0.012 to $0.021 per session.
  • Users noticed. The average score dropped, and in the first week I got a handful of "this feels harsh" emails. Nobody had ever emailed to say a 7 felt too generous. That asymmetry is the whole problem in one sentence.
  • Checklists need maintenance. Every new dimension means writing and validating new checks against the calibration set, not just adding a word to a prompt.

So how do you stop an LLM-as-a-judge from giving everything a 7?

Stop asking it for a number. An LLM-as-a-judge on an unanchored 1-10 scale will cluster around 7 because "7" is the model's default for "acceptable," and telling it to be strict only slides the cluster down. Replace the scale with concrete yes/no checks that each require a quoted piece of evidence, compute the score in your own code, and validate the result against a small hand-graded set where you know the right ranking. In my case that took rank agreement with human grades from 0.41 to 0.79, against a human-vs-human ceiling of 0.83, at the cost of higher token spend and a known blind spot for brilliant short answers.

Written by the developer behind Preterview, an interview prep platform.