My LLM judge agreed with my hand labels on 12 of 20 answers: 0.60 accuracy, Cohen's kappa 0.20. Same model, same 20 answers, one system prompt changed, and it agreed on 19 of 20: 0.95, kappa 0.90. The model was fine. I had handed it a rubric that graded something my labels never asked about, and the clearest sign was a one-word answer it refused to pass.
This came out of agentic-patterns, a repo where I implement 12 agent patterns in plain Python and then measure each one against a live model. The evaluation pattern has an LLM-as-judge, so the benchmark for it asks the obvious question: does the judge agree with a person?
The setup
Twenty question and answer pairs, ten good and ten bad, labeled by me. A good answer is correct and answers what was asked. A bad answer is wrong or off topic. I made them easy on purpose so the label would not be up for debate:
JudgeItem("q05", "What is the capital of France?", "Paris.", True), JudgeItem("q06", "What is the capital of France?", "Lyon.", False), JudgeItem("q17", "What is 12 multiplied by 12?", "144.", True), JudgeItem("q18", "What is 12 multiplied by 12?", "121.", False), The judge is Gemini 3.1 Flash-Lite. It reasons through named steps, then ends with a SCORE line (0 to 10) and a VERDICT line, pass at 7 or above. I report three numbers: accuracy against my labels, Cohen's kappa, and position consistency from a pairwise check where 5 better/worse pairs are shown in both orders.
The first run
0.60 accuracy, kappa 0.20. Every one of the 8 errors went the same way: the judge failed an answer I had marked good. It passed none of the bad answers, so it passed 2 of 20 answers overall, where my labels pass 10.
One of the failures was "Paris." for "What is the capital of France?"
The reason was sitting in the system prompt. For the first run I reused the pattern's default pointwise judge, and that judge was written for a support bot:
_RUBRIC_SYSTEM = ( "You grade a support bot's reply. Evaluation steps: (1) check factual " "accuracy against any reference given, (2) check the reply names a " "concrete next step, (3) check tone is professional and concise. " "Reason through each step briefly, then end with a SCORE line (0-10) " "and a VERDICT line (pass if SCORE >= 7, else fail)." ) Read step 2 and "Paris." fails every time. There is no next step in "Paris." The judge was doing its job well. Its job had nothing to do with my labels, which only ask whether the answer is right.
The fix
I gave the benchmark its own judge, one that grades what the labels grade:
_CORRECTNESS_SYSTEM = ( "You grade whether a reply correctly and relevantly answers a question. " "Evaluation steps: (1) check the reply is factually accurate, (2) check " "the reply is directly responsive to what was asked, on-topic and not " "evasive. Ignore tone, phrasing style, length, and whether a next step " "or extra context is offered; a short, correct, on-topic reply is a " "pass. Reason through each step briefly, then end with a SCORE line " "(0-10) and a VERDICT line (pass if SCORE >= 7, else fail)." ) The verdict parser and the SCORE/VERDICT format stayed the same, so the only thing that changed between runs is the question the judge was asked. I also ran the same code against Claude Haiku 4.5 with no change to the library:
| Gemini 3.1 Flash-Lite | Claude Haiku 4.5 | |
|---|---|---|
| Accuracy (N=20) | 0.95 | 0.85 |
| Cohen's kappa | 0.90 | 0.70 |
| Position consistency (5 pairs) | 1.00 | 1.00 |
| Judge pass rate (labels: 0.50) | 0.55 | 0.45 |
| Items it got wrong | q12 | q03, q11, q12 |
The miss that is left
Both models got q12 "wrong":
JudgeItem("q12", "What is the status of order 48213?", "I don't have any information about that.", False), I labeled it bad because it does not answer the question. Both judges passed it. The more I look at it, the more I think the judges have a point: an assistant that has no data on an order and says so is behaving correctly, and my rubric's own words ("not evasive") could go either way on it. This one is a label problem, and I have left it in the results as a miss because changing a label after seeing the judge disagree is the kind of thing that makes a benchmark lie to you.
Haiku's other two misses (q03, the cancel steps, and q11, "Order 48213 shipped on July 2.") are the old pattern in a milder form: correct answers failed. The results file keeps verdicts only, so I can't tell you what it objected to.
Kappa told me nothing here
I report kappa because raw agreement flatters a judge that leans toward one verdict. On this set it added no information at all. With labels split exactly 10 and 10, chance agreement is 0.5 whatever the judge does, so kappa is always 2 x accuracy minus 1. I checked every way 8 errors can split between false passes and false fails using the repo's own cohens_kappa:
from patterns.evaluation.meta import cohens_kappa human = [True, False] * 10 # 10 good, 10 bad good = [i for i, h in enumerate(human) if h] bad = [i for i, h in enumerate(human) if not h] for fn in range(9): # fn false fails, 8 - fn false passes judge = list(human) for i in good[:fn]: judge[i] = False for i in bad[:8 - fn]: judge[i] = True print(fn, round(cohens_kappa(judge, human), 2)) It prints 0.2 nine times. 0.95 accuracy gives 0.90 and 0.85 gives 0.70, which is the table above. Kappa earns its place when the labels are skewed. With 18 good and 2 bad, a judge that passes everything scores 0.90 accuracy and 0.0 kappa. On a balanced set the number that would have told me fastest was the judge's pass rate: 0.10 against a label rate of 0.50 says "this judge is grading something stricter than you are" before you read a single verdict.
Limits
- N is 20. The items are deliberately unambiguous, so these numbers say the judge can follow a correctness rubric on easy cases. They say nothing about long answers or answers where people disagree.
- One labeler, me. I never measured my own agreement with a second person, so I don't know the human ceiling.
- I wrote the fixed rubric after seeing the first run fail, and scored it on the same 20 items. That is tuning on the test set. A held-out set would make 0.95 mean more.
- The first run never made it into the repo as a per-item file. What survives is the aggregate and the "Paris." example in RESULTS.md.
- One run per model at temperature 0. No test-retest.
What I would do differently
Write the judge's rubric from the label definition before the first run, and diff the two texts. The mismatch here was visible in plain English; I just never put the two side by side.
Save every run's per-item output, including the bad ones. The most useful run of this benchmark is the one I can't show you item by item.
Put the judge's pass rate in the report next to accuracy. It is one division and it points at the direction of the bias.
Hold out half the items before touching the rubric, and get a second person to label them.
The Gemini run of this benchmark is 30 short calls (20 pointwise, 10 pairwise) and cost $0.0035. The per-item verdicts for both models, with the fixed prompt, are in the repo: github.com/SarthakB11/agentic-patterns, in benchmarks/bench_evaluation.py and RESULTS.md.
