Most LLM eval advice collapses into one of two failure modes. Either someone tells you to "just use an LLM judge for everything," or someone hands you a wall of BLEU and ROUGE scores that nobody on the team can interpret. Both are wrong for the same reason: a scorer is a claim about what "correct" means for a specific output, and no single scorer makes that claim well for every case.
I have spent a while building eval tooling around exactly this problem. My CI eval runner, evalgate, ships ten scorers, and my voice eval tool, voiceeval, adds a set of checks that only make sense once you stop reading transcripts and start listening to clocks. This post is the map I wish I had had: what each scorer actually measures, and the case where it is the right tool.
The mental model
An output can be wrong in structurally different ways. It can be the wrong string. It can be malformed. It can be missing a required fact. It can be technically fine but slow, expensive, rude, or hallucinated. Each of those is a different question, and you want the cheapest scorer that answers the question you actually have.
Cheap and deterministic scorers (string and schema checks) are fast, free, and never flake. Expensive and probabilistic scorers (embeddings, LLM judges) catch nuance but cost money and introduce their own variance. The skill is reaching for the deterministic one whenever you can, and only escalating when the question genuinely requires judgment.
The scorer catalog
Here is how I map each scorer to the question it answers.
| Scorer | Use it when | The question it answers |
|---|---|---|
| exact-match | Output is a fixed string: a canned reply, a slug, a classification label | "Is this byte-for-byte right?" |
| regex | Output must follow a shape but not a literal value: an order id, a date, a currency format | "Does this match the pattern?" |
| contains / not-contains | Certain facts must appear, or certain phrases must never appear | "Did it include X? Did it avoid Y?" |
| json-schema | Output is structured data feeding a downstream system | "Is this valid and correctly shaped?" |
| embedding-similarity | Meaning matters but wording is free: summaries, paraphrases | "Is this close in meaning to the reference?" |
| llm-judge | Quality is subjective and rule-based checks cannot express it: tone, helpfulness, reasoning | "Would a careful reader call this good?" |
| rubric | Quality has several weighted dimensions you want scored together | "How does this do across all my criteria?" |
| latency | The call must return within a time budget | "Was it fast enough?" |
| cost | The call must stay within a spend budget | "Was it cheap enough?" |
A few notes from using these in anger.
exact-match is underrated. For anything with a canonical answer, a fixed greeting, a routing label, a boolean, exact-match is perfect and free. People skip it because it feels too simple, then spend judge tokens confirming that "yes" equals "yes."
regex is the honest middle. When you care about shape but not content, a pattern beats a judge every time. An order confirmation should contain something like ORD-\d{6}. You do not need a model to tell you whether a six-digit id is present.
contains gives partial credit. In evalgate the contains scorer can check a list of required substrings and score the fraction present, which is exactly what you want for "the answer must mention the refund amount, the timeframe, and the ticket number." not-contains is its guardrail twin: it fails when a banned phrase, a competitor name, a leaked system instruction, shows up.
json-schema is non-negotiable for tool-calling agents. If your model's output is parsed downstream, a schema check is the difference between a caught regression and a 2am page. Validate structure before you ever score content.
embedding-similarity and llm-judge are the expensive tier. Similarity handles "same meaning, different words." The judge handles everything a rule cannot express. Both are probabilistic, so treat their thresholds as tunable, not sacred. In evalgate these two ship with deterministic offline fallbacks specifically so the test suite can run with zero API keys and never flake in CI.
latency and cost are quality dimensions, not afterthoughts. A correct answer that takes eight seconds or costs a dollar is a regression. Budget scorers turn those into pass/fail signals you can gate on.
Voice needs its own scorers
Here is the part text-focused eval writers miss entirely. If you evaluate a voice agent by reading its transcript, you are evaluating a text agent that happens to have been spoken. Transcripts have no clocks, and no memory of what was actually said before speech-to-text mangled it.
The canonical failure: the caller said "fifteen," the STT heard "fifty," and the agent confidently refunded fifty. Read the transcript and it is flawless. The agent refunded exactly what the transcript says the caller asked for. The transcript agrees with itself perfectly, and the company just lost thirty-five dollars.
So voiceeval adds checks that only exist in the audio-and-time domain:
- misheard_number compares what STT heard against ground truth. "Fifteen" and "fifty" are one unstressed syllable apart, and the agent acts on either with equal confidence. This is the expensive one.
- no_confirmation fails when the agent takes a consequential action, a refund, a booking, a cancellation, without ever reading the value back. In text a misunderstanding costs one turn. In voice it costs the refund.
- slow_response and dead_air read the clock: three seconds of silence is a failed call however good the eventual answer, and dead air is where callers hang up.
- talked_over_user catches barge-in handling, and policy_violation catches actions that exceeded a coded limit.
The lesson generalizes. Your domain probably has a "misheard number," a failure mode that is invisible to generic scorers because it lives in a signal your transcript threw away. Find it, and write a scorer for it.
The one honest caveat
LLM-as-judge is seductive because it feels like it can score anything, and that is exactly why it is dangerous. The judge is itself a model with variance, prompt sensitivity, and its own biases. It will happily reward verbose, confident-sounding answers, and its scores drift when you change the judge model. Never let a judge be your only signal on a case that matters. Pin down whatever you can with deterministic checks first, use the judge for the genuinely subjective residue, and treat its threshold as something you calibrate against human labels rather than a number you trust on faith.
Where this leaves you
Start deterministic. Reach for exact-match, regex, contains, and json-schema before anything else, because they are free, fast, and never flake. Add embedding-similarity and the judge only for the questions that truly need meaning or taste. Wire latency and cost in from day one. And if any part of your system lives outside text, whether that is audio, timing, or images, go find the failure mode your generic scorers cannot see and give it a scorer of its own.
The whole point is not to answer "is this good?" It is to answer "is this worse than it was?" That is the only question CI can answer objectively, and the right set of scorers is what makes the answer trustworthy.
If you want to see the ten scorers wired into a real regression gate, evalgate is on GitHub at github.com/AgentPostmortem/evalgate. Fork it, steal the scorer list, and point it at your own suite.
