You changed one line of your agent's prompt. What else broke?

You tweak one line of your agent's prompt to fix a complaint. The fix works. You ship it.

Two days later, a different user reports something that used to work just fine.

Nobody touched that feature. You changed the prompt, and the agent's behavior shifted everywhere at once. That's why AI agents need evals, and why the evals most teams write miss the bugs that matter.

TL;DR

An eval is a task, a grader, and a number of trials. Not a vibe check.
Grade what the agent did, not just what it said.
Run every task more than once. "Works 90% of the time" fails a lot.

What an eval actually is

An eval gives the agent a task, runs it, and scores the result automatically. Here's one for a scheduling agent:

task: "Book 30 minutes with Priya next Tuesday afternoon" expect: - create_event called once - attendee is priya@example.com - start time is Tuesday, between 12:00 and 17:00 - no existing meeting was moved or deleted trials: 5 pass: all 5 trials

Run your suite after every prompt, model, or tool change, and you'll know what broke before your users do.

Grade all three layers

Most eval suites only check the final reply. But an agent can say "Booked!" while calling the wrong tool, or call the right tool with the wrong date. Check all three: the outcome, the trajectory (tool calls and their arguments), and the state that changed.

Use the cheapest grader that works

Start with plain code. It's fast, free, and never disagrees with itself:

def grade(trace, calendar): events = [c for c in trace.tool_calls if c.name == "create_event"] assert len(events) == 1 assert "priya@example.com" in events[0].args["attendees"] assert calendar.unchanged_except(events[0].result_id)

Use an LLM judge only for things code can't check, like tone or whether an answer is actually helpful. Then have a human review a sample of the judge's scores now and then, so you know the judge is grading the way you would.

Run it more than once

Agents are non-deterministic. If a task passes 90% of the time, the chance it passes five times in a row is only 59%. For anything user-facing, "it worked when I tried it" isn't good enough. Measure how often it works every time.

Where your first 20 evals come from

Don't invent test cases. Mine them:

Real failures: every bug report and bad transcript becomes an eval.
Messy inputs: vague dates, missing details, typos, two requests in one message.
Happy paths: a few normal requests that must keep working, so a fix can't quietly break real users.

Twenty real cases beat two hundred made-up ones.

Keep two suites

Regression evals check things that already work. They should pass almost 100% of the time, and a failure blocks the release.

Capability evals check things the agent can't do reliably yet. A low score is expected. Watching it climb tells you whether your changes are actually helping.