I pasted a deliberately useless agent trajectory into my eval harness and it handed me 82/100 on quality anyway.

That was demo mode, and I knew it — demo mode returns the same fixed sample numbers for any trajectory you paste in, which is exactly what a demo should do. But the number still made me pause, because I had spent the previous month treating "a score" and "a measurement" as the same thing. A demo score that wandered around would have told me even less, so I went back and made the harness say only what it actually knows.

Here is the shape of it.

Three dimensions, because "good" is not a metric

The first version of my grading criteria was one field called "quality". Useless — I was writing a rubric with one word in it and then arguing with myself about what it meant.

So the harness scores three dimensions, each 0-100 with a one-line justification:

  • Quality — is the output correct and useful for the task?
  • Faithfulness — is it grounded in what the agent actually did, without invention?
  • Task success — did it reach the goal it was given?

Each run lets me pick one dimension or all three, and the grading prompt asks for a per-axis score, an overall pass-rate style summary, and a regression flag when the trajectory looks off-task. That last one is what I use most: an off-task flag on a run that scored well tells me more than the score does.

What it grades is the trajectory, not the prompt

The input is three things: a description of my agent, the run itself (steps plus the final output), and the scoring criteria. Not the system prompt. Not the model name. The trajectory, because that is the thing that actually happened and the thing I want to compare across releases.

Anything that emits a trajectory works — a support bot, a tool-using agent, a multi-step workflow. If you only have the final answer, you do not have much to grade.

The 14 checks that run on every output

This is the part I did not expect to matter as much as it does. Every output, including the model's, runs through a deterministic rule layer before I see it. The ruleset is versioned — agent-eval@2026-10-06 — and each run returns its runId and the ruleset version, so a score is always traceable to the rules that produced it.

Fourteen rules, grouped into completeness, quality, faithfulness, and honesty. A few of the ones that changed my own behaviour:

Rule What it checks
EVAL-02 a quality score is actually present
EVAL-04 no fabricated perfect-accuracy or always-correct claims
EVAL-05 a regression flag is present, against a baseline
EVAL-07 a public agent benchmark is named, not just "it went fine"
EVAL-12 the grading method is disclosed — LLM-as-judge, or human review
EVAL-13 tool selection and multi-turn trajectory are scored, not just the reply

EVAL-13 is the one I would keep if I had to keep only one. Grading only the final answer hides the interesting failure: the agent picked the wrong tool three steps in and then wrote a very confident paragraph about it.

Why I trust the rule layer more than the model grade

Because the grader is another model. That is unavoidable — a rubric written in English has to be applied to English output somehow — but it means the model-assisted grade is one more fallible thing in the chain, and I should not pretend otherwise.

The rule layer is different in kind. It cannot be talked into being wrong by a well-written answer, because it is checking the shape of the answer: is the score there, is the regression flag there, is task success defined, is the judge method named, is the trajectory actually scored. When the model grade and the rules disagree, I know which one to believe.

One thing I keep honest in the interface: if no model is configured, the API returns an error instead of quietly serving a mock. And demo mode labels itself as demo in the response, which is exactly the thing my 82/100 incident was about.

Pricing

Free tier, 3 eval runs per day and 15 per month. Pro is $29/mo for 40 runs per day and 300 per month, with saved history and export, or $290/yr. Enterprise is custom — BYOK, higher caps, SSO that is roadmap-ready rather than shipped today.

It is a hosted web tool, and the web form is backed by a REST endpoint if you want to score runs from your own pipeline: AgentGrade.

What AgentGrade is not

It scores outputs against the criteria you give it. That is the whole contract.

  • It is not a correctness oracle. A high quality score means the output met the rubric, not that the rubric was right.
  • It does not prove a task is safe. A trajectory that looks clean tells you nothing about what it would do with a different prompt, a different user, or a different piece of data.
  • It cannot tell you the model is right. It can tell you whether the answer matched the trace.
  • It is decision-support, not a certificate. If you need a certified or attested result, this is the wrong shape of tool.
  • The rubrics and reports are in English, so a trajectory in another language grades less directly comparable.
  • Automated score diffing across releases is on the roadmap. Today I compare by reading two reports side by side, which is the weak link I am trying to remove.

FAQ

Q: What does AgentGrade actually evaluate?
A: Agent trajectories against task success, faithfulness, and quality rubrics, plus regression checks. You paste a description of the agent, the run (steps plus final output), and your scoring criteria.

Q: How are the scores produced?
A: Rule-based checks plus a model-assisted grader. Every run returns a runId and a ruleset version, so you can tie a score back to the exact rules that produced it. The deterministic layer checks the output's structure; the model grade applies the rubric.

Q: Can I use my own rubric?
A: Every run today uses the built-in Quality, Faithfulness, and Task success rubrics. You can paste your own scoring criteria into the input field. Custom rubric import is on the Enterprise roadmap.

Q: Does it catch regressions?
A: Each run flags regression and off-task signals, and saves the score to your history so you can compare across releases. Automated score diffing is still on the roadmap, so the comparison is manual for now.

Q: Is it only for chatbots?
A: No. Any agent that emits a trajectory works — steps plus tool calls. If you only have the final answer, the input is too thin to grade.

References