If you have put an LLM agent anywhere near customers, you already know the uncomfortable part. You can see everything it does, and you still don’t know whether it’s right.
Our team had full tracing on our support triage agent from day one. We could replay any ticket, read every tool call, watch it reason. What we didn’t have, for the first six weeks, was any way to say before a release whether the new version was better or worse than the old one. We found out in production, when return requests started landing in the wrong queue.
This post is the practical version of how we fixed that: how a 14-engineer team with no ML specialists built an eval set that now gates every change to the agent. Nothing here needs a platform. It needs a spreadsheet, a few hours from the person who knows the domain, and the discipline to run it every time.
Start with real tickets, not invented ones
The first instinct is to write test cases. Don’t. Pull real inputs.
We exported about 200 recent support tickets, removed anything with personal data, and gave them to our support lead, the person who had been answering them by hand for two years.
She picked 40 that between them covered the shapes of question we actually get: where is my order, is this part compatible, I want to return this, the invoice is wrong, I never received it, plus a handful of ambiguous ones where even a human would hesitate.
Forty sounds small. It is small. It was also enough to catch the two failures that mattered in the first month, and small enough that one person could label it in an afternoon, which is the real constraint.
An eval set nobody maintains is worse than none, because it gives you false confidence.
Label the decision, not the prose
For each ticket we recorded three things.
- The correct queue (orders, returns, billing, compatibility, human).
- Whether the agent is allowed to answer at all or must hand off.
- And, where it may answer, the facts the reply must contain and the facts it must not claim.
For a where-is-my-order ticket, the reply must cite the last carrier scan and must not say “on its way” unless that scan exists.
We did not write a model answer. Grading against a model answer makes the eval about wording, and wording is the thing LLMs are good at. The decisions are what go wrong.
The returns failure came from exactly this gap. Our first labels had the routing decision but our examples all used the word “return”. Customers wrote “can I send this back” and “it’s not the one for my machine”, the agent filed them as general enquiries, and because tickets-answered was going up nobody looked.
Those misrouted tickets are now their own block in the set, and anything with return intent routes to a person regardless of what the model thinks.
Make the grader cheap enough to run every time
We grade two ways.
Routing and hand-off decisions are checked by code: did the agent pick the labelled queue, yes or no. That’s a script and it runs in seconds.
The content checks, must-contain and must-not-claim, we grade with a second model call that sees the reply and the labels and returns a pass or fail with a one-line reason. We spot-check about a fifth of those by hand every run, because a grader that nobody audits drifts just like the agent does.
The whole set runs in under two minutes. That matters more than it sounds.
If it took an hour, it would run weekly. Because it takes two minutes, it runs on every prompt change, every model switch, every new tool, and the result is posted to the channel where the change was proposed.
The number everyone looks at is routing accuracy on the 40, and the rule is simple: it does not ship if that number goes down.
What changed in the team
The product manager, not the engineers, owns the set. That was the real shift.
Before, I wrote specs describing behaviour and engineers interpreted them. Now I write the test set and the pass criteria, and the agent’s output is graded against it. It is a different job.
A spec is an act of imagination about something that doesn’t exist. An eval is an act of judgment about outputs that do, and it forces you to decide, in advance and in writing, what correct means for your customers.
It also changed how we scope. Each new workflow we hand to an agent starts with its eval set, funded before the feature.
We said no to the next several agent ideas because nobody had time to build the set, and that was the right call.
In the first months, review load grew faster than drafting time fell, and velocity didn’t move until we narrowed scope to three workflows and gave reviewers a one-page checklist.
If you take one thing from this
Most teams I talk to have observability and no evals. They can watch the agent fail. They can’t catch it first.
The fix does not need a platform or a research team. It needs forty real inputs, the person who knows the answers, three labels per input, and a two-minute script that runs every single time.
Build that before the next feature, not after the next incident.
