AI Evals: Stop Trusting Your Agent's “Done”

AI agents are getting pretty good at doing things.

But an agent can complete a task without actually doing it correctly.

Imagine this scenario.

You give your coding agent a Jira ticket:

Add a credit card limit update feature to the payment service.

The agent reads the ticket, searches the repository, finds the relevant service, implements the changes, writes tests, and reports:

"Implementation completed. All tests passed."

Looks good, right?

But let's look a little closer.

• Did it find the correct payment service?
• Did it follow the existing architecture?
• Did it validate the user's authorization?
• Did it accidentally modify unrelated files?
• Did it actually execute the tests?
• Did it use the correct MCP tools?
• Could it have completed the task with fewer steps?

The agent says it's done. But how do we know?

This is where Evals (Evaluations) come in.

I think Evals are becoming one of the most important parts of AI engineering, especially as we move from simple LLM applications to autonomous agents and multi-agent workflows.

Let's look at how we can actually implement them.

1. Evals Are Not Just Another Type of Test

Traditional software testing is built around predictable behavior.

For example:

const result = calculateCreditCardLimit(10000, 5000); expect(result).toBe(5000); 

We know the input, we know the expected output, and we can verify the result.

AI systems are different.

The same task can produce different outputs depending on the model, prompt, context, available tools, and execution environment.

Consider asking an AI agent:

"Find the credit card limit update implementation and explain how it works."

There might be several valid ways to investigate the codebase.

One agent might search by endpoint name. Another might start from the controller. Both could eventually find the correct implementation.

Requiring exactly the same execution sequence would be unnecessarily restrictive.

Instead, we need to evaluate whether the agent reached the correct result and whether its execution was reasonable.

I find it useful to think about Evals as another layer in our engineering workflow.

These approaches complement each other.

We still need unit tests, integration tests, and end-to-end tests. Evals help us understand whether the AI system can reliably produce the changes those tests are supposed to verify.

2. Start With Real Tasks, Not Random Prompts

Before building an evaluation system, we need a collection of tasks to evaluate.

Let's continue with our credit card example.

Imagine we have a coding agent connected to Jira, GitLab, and an internal codebase through MCP.

I would start with a small dataset containing different types of engineering tasks.

[ { "id": "CC-001", "category": "code-discovery", "input": "Find the credit card limit update implementation.", "expected": { "behavior": "Identify the correct controller and service.", "mustNotModifyFiles": true } }, { "id": "CC-002", "category": "feature-implementation", "input": "Add authorization validation to credit card limit updates.", "expected": { "behavior": "Validate card ownership before updating the limit.", "testsRequired": true } }, { "id": "CC-003", "category": "bug-fixing", "input": "Fix the issue where invalid credit card limits are accepted.", "expected": { "behavior": "Reject invalid limits and preserve existing behavior.", "testsRequired": true } }, { "id": "CC-004", "category": "architecture-analysis", "input": "Explain the credit card limit update flow.", "expected": { "behavior": "Explain the BFF, API and service interactions.", "mustNotModifyFiles": true } } ] 

These four scenarios already test different capabilities:

• Understanding an existing codebase.
• Implementing a new feature.
• Fixing a bug.
• Understanding system architecture.

But don't stop at happy paths.

Add scenarios such as:

• The Jira ticket has incomplete requirements.
• The relevant API documentation is missing.
• An MCP server returns a timeout.
• The repository contains multiple implementations of the same feature.
• The agent encounters an existing test failure.
• Two agents attempt to modify the same file.
• The user requests an operation without sufficient authorization.

A good evaluation dataset should represent the problems your agent will encounter in real life.

One practice I particularly like is turning production failures into permanent evaluation cases.

If an agent makes a mistake today, we should be able to detect the same mistake in tomorrow's evaluation run.

3. Deterministic Evals: Let Code Check the Result

Not every evaluation needs another AI model.

In fact, if we can verify something with ordinary code, we should.

Let's say our coding agent implements the credit card limit update feature.

We want to check:

• Was the correct service modified?
• Was authorization validation added?
• Were tests created?
• Did the tests actually pass?
• Were unrelated files modified?

We can write a simple evaluator.

type AgentResult = { modifiedFiles: string[]; toolCalls: string[]; testsPassed: boolean; }; function evaluateCreditCardFeature(result: AgentResult) { return { correctServiceModified: result.modifiedFiles.some( file => file.includes("payment-service") ), testsAdded: result.modifiedFiles.some( file => file.endsWith(".spec.ts") ), testsExecuted: result.toolCalls.includes("run_tests"), testsPassed: result.testsPassed }; } 

Now we can evaluate the actual result instead of trusting the agent's final message.

For example:

{ "correctServiceModified": true, "testsAdded": true, "testsExecuted": true, "testsPassed": false } 

The agent did most of the work, but the implementation isn't ready.

This is a much more useful result than simply receiving "Task completed."

Where can deterministic checks help?

For example, after the agent finishes:

yarn nx test payment-service yarn nx run payment-service:lint yarn nx run payment-service:build 

The important detail is that the evaluator executes these commands independently.

We shouldn't accept an agent's statement that tests passed as proof that they actually passed.

4. Tool Selection: Did the Agent Use the Right Tools?

This becomes especially interesting when we start using MCP.

Imagine our agent has access to these tools:

const tools = [ "search_jira", "search_repository", "read_file", "get_api_documentation", "run_tests", "create_merge_request" ]; 

Now give it this task:

"Find the current credit card limit update implementation and explain its behavior."

A reasonable execution might look like:

1. search_repository 2. read_file 3. get_api_documentation 4. Generate explanation 

But imagine this:

1. search_jira 2. create_merge_request 3. search_repository 4. read_file 

The agent has performed unnecessary and potentially dangerous actions.

We can inspect its tool calls and compare them with the expected behavior.

function evaluateToolUsage( actualTools: string[], requiredTools: string[], forbiddenTools: string[] ) { const missing = requiredTools.filter( tool => !actualTools.includes(tool) ); const forbidden = forbiddenTools.filter( tool => actualTools.includes(tool) ); return { passed: missing.length === 0 && forbidden.length === 0, missing, forbidden }; } const result = evaluateToolUsage( [ "search_repository", "read_file", "get_api_documentation" ], [ "search_repository", "read_file" ], [ "create_merge_request" ] ); console.log(result); 

The expected result is a passing evaluation.

Of course, there can be multiple valid execution paths. We don't need to force every agent to call tools in exactly the same order.

What matters is whether it used the necessary tools, avoided forbidden actions, and achieved the intended result.

This is particularly important when building an MCP router with hundreds of available endpoints.

A model might be capable of using tools, but that doesn't mean it will always select the right one.

Tool selection itself should be evaluated.

5. Trajectory Evals: The Final Answer Isn't the Whole Story

A trajectory is the sequence of actions an agent takes while completing a task.

Consider two agents working on the same credit card feature.

Agent A

1. Read Jira task 2. Search repository 3. Identify payment service 4. Read existing implementation 5. Check API documentation 6. Implement changes 7. Run unit tests 8. Fix failing test 9. Report result 

Agent B

1. Read Jira task 2. Search entire repository 3. Read 35 unrelated files 4. Search the same code again 5. Call MCP tool 6 times 6. Generate implementation 7. Skip tests 8. Report success 

Both might produce similar final responses.

But their execution quality is very different.

We should evaluate:

• Number of tool calls.
• Unnecessary file reads.
• Repeated operations.
• Token consumption.
• Recovery from failures.
• Whether the agent verified its changes.

A simple trace evaluator might look like this:

type AgentTrace = { toolCalls: number; filesRead: number; retries: number; testsExecuted: number; durationMs: number; totalTokens: number; }; function evaluateTrajectory(trace: AgentTrace) { return { toolEfficiency: trace.toolCalls <= 20, contextEfficiency: trace.filesRead <= 25, retryStability: trace.retries <= 3, verification: trace.testsExecuted > 0, timeBudget: trace.durationMs <= 120_000, tokenBudget: trace.totalTokens <= 30_000 }; } 

These numbers are just examples. A code-discovery task and a multi-file feature implementation shouldn't have the same execution budget.

The goal isn't to make agents use as few tools as possible.

It's to understand whether the tools and context they consume actually contribute to solving the task.

6. LLM-as-a-Judge: When We Need Another Model to Evaluate the Result

Some things are difficult to verify with deterministic tests.

For example:

"Explain how credit card limit updates work in this codebase."

There can be multiple correct explanations, so exact string matching isn't particularly useful.

We can use another LLM as an evaluator.

The process is straightforward:

1. The agent completes the task.
2. We collect its response.
3. A judge model receives the task, reference information, and response.
4. The judge evaluates the response against a defined rubric.
5. We store the scores and explanation.

Example judge prompt:

You are evaluating an AI coding agent. Task: Explain how credit card limit updates work. Reference information: - The request enters through the BFF. - The payment API handles the request. - The service validates card ownership. - The business layer validates the requested limit. - The repository persists the updated limit. Evaluate using these criteria: 1. Technical correctness: 0-5 2. Completeness: 0-5 3. Relevance: 0-5 4. Unsupported assumptions: 0-5 Do not reward confident language. Penalize invented implementation details. Highlight missing authorization checks. Return structured JSON. 

Possible result:

{ "technical_correctness": 4, "completeness": 3, "relevance": 5, "unsupported_assumptions": 1, "reason": "The answer explains the BFF and service layers but misses the ownership validation." } 

This is useful when comparing different models or prompt configurations.

For example, we could compare a local Qwen model with another coding model using the same tasks and the same evaluation criteria.

However, an LLM judge can also be wrong.

It may prefer a confident but inaccurate answer, overlook a technical detail, or score the same answer differently across runs.

For critical checks, combine LLM judging with deterministic assertions and human review.

7. RAG Evals: Did We Retrieve the Right Information?

Evals are not limited to coding agents.

Consider a RAG system that searches internal engineering documentation.

Question:

"How does the credit card limit update process work?"

The retrieval system returns five documents.

But what if the correct API documentation isn't among them?

The model might still generate a convincing explanation using unrelated documents.

We need to evaluate two different things:

Retrieval quality: Did we find the right information?

Answer quality: Did the model use that information correctly?

Let's define a small dataset:

[ { "question": "How is a credit card limit updated?", "expectedDocuments": [ "credit-card-api.md", "payment-service.md" ], "expectedFacts": [ "ownership validation", "limit validation", "database update" ] }, { "question": "What happens when a limit update fails?", "expectedDocuments": [ "credit-card-error-handling.md" ], "expectedFacts": [ "error response", "validation failure", "retry behavior" ] } ] 

There are several useful retrieval metrics.

Recall@K

Did the correct documents appear in the top K results?

Suppose there are three expected documents and the retriever returns only one of them in its top five results.

[
Recall@5 = \frac{1}{3}
]

Precision@K

How many retrieved documents were relevant?

If only two of the five returned documents are relevant:

[
Precision@5 = \frac{2}{5}
]

MRR (Mean Reciprocal Rank)

How high does the first relevant document appear?

If the first relevant document is at position two:

[
RR = \frac{1}{2}
]

These metrics help us identify whether the problem is in retrieval or generation.

For example, a model might produce the correct answer even though the retriever returned the wrong documents.

That doesn't mean the RAG system is reliable. The model might simply have guessed correctly.

8. Multi-Agent Evals: Individual Agents Can Work, But the System Can Still Fail

Let's take a step further.

Imagine we have a multi-agent development workflow.

A Jira ticket enters the system, and several agents work together.

 Jira Ticket | v Planner Agent | Task Decomposition | +------------+------------+ | | | v v v Backend Frontend Tester Agent Agent Agent | | | v v v API Code UI Code Test Results | | | +------------+------------+ | v Review Agent | v Final Result 

How do we evaluate this?

We should evaluate each agent separately.

But individual performance isn't enough.

We also need to evaluate coordination.

For example:

• Did the frontend agent start before the API contract was ready?
• Did two agents modify the same file?
• Did the planner forget an important requirement?
• Did the reviewer inspect the latest implementation?
• Did the orchestrator recover when one agent failed?

This is where workflow-level evaluation becomes important.

An agent might be excellent at writing code but unreliable when coordinating with other agents.

The same principle applies to distributed agent architectures: communication, task ownership, shared state, and failure handling are all part of the evaluation.

9. Regression Evals: Don't Let Improvements Break Existing Behavior

Here's a situation that will probably look familiar to anyone experimenting with prompts and models.

You update your system prompt to improve backend code generation.

Your backend agent performs better.

But suddenly, it starts modifying more files than necessary during frontend tasks.

Without regression evaluations, you might not notice until production.

A simple workflow:

 Prompt / Model Change | v Run Evals | v Compare Results | +-----+-----+ | | Improved Degraded | | v v Accept Investigate | | v v Deploy Fix / Revert 

For example:

The new prompt improved backend performance but damaged frontend performance and increased token consumption.

This is why every meaningful change to a model, prompt, tool, context strategy, or orchestration logic should trigger relevant evaluations.

I would treat this as part of the normal development lifecycle, just like running unit and integration tests before merging a change.

10. Let's Build a Small Evaluation Runner With TypeScript

So far, we've discussed different evaluation techniques.

Let's put some of them together.

We don't need a complex platform to get started. A simple evaluation runner is enough to demonstrate the idea.

Our runner should:

1. Load evaluation scenarios.
2. Execute the agent.
3. Collect the execution result.
4. Run deterministic checks.
5. Record latency and token usage.
6. Generate a report.

First, define the contracts.

type EvalCase = { id: string; input: string; expected: { requiredTools?: string[]; forbiddenTools?: string[]; mustNotModifyFiles?: boolean; }; }; type AgentExecution = { answer: string; modifiedFiles: string[]; toolCalls: string[]; testsPassed: boolean; durationMs: number; totalTokens: number; }; type EvalResult = { id: string; passed: boolean; failures: string[]; durationMs: number; totalTokens: number; }; 

Now let's implement the evaluator.

async function evaluateCase( testCase: EvalCase, executeAgent: (input: string) => Promise<AgentExecution> ): Promise<EvalResult> { const execution = await executeAgent(testCase.input); const failures: string[] = []; for (const tool of testCase.expected.requiredTools ?? []) { if (!execution.toolCalls.includes(tool)) { failures.push(`Missing tool: ${tool}`); } } for (const tool of testCase.expected.forbiddenTools ?? []) { if (execution.toolCalls.includes(tool)) { failures.push(`Forbidden tool used: ${tool}`); } } if ( testCase.expected.mustNotModifyFiles && execution.modifiedFiles.length > 0 ) { failures.push("Unexpected file modifications"); } return { id: testCase.id, passed: failures.length === 0, failures, durationMs: execution.durationMs, totalTokens: execution.totalTokens }; } 

Finally, run the dataset.

async function runEvaluation( dataset: EvalCase[], executeAgent: (input: string) => Promise<AgentExecution> ) { const results: EvalResult[] = []; for (const testCase of dataset) { results.push( await evaluateCase(testCase, executeAgent) ); } const passed = results.filter( result => result.passed ).length; return { total: results.length, passed, failed: results.length - passed, successRate: passed / results.length, averageTokens: results.reduce((sum, r) => sum + r.totalTokens, 0) / results.length, results }; } 

This is a deliberately small implementation. A production runner would also need isolated workspaces, test-result assertions, execution timeouts, structured traces, repeated runs, and error classification.

Also notice that the evaluator doesn't trust the agent's final response. It checks the actual execution data.

That's the idea.

Evaluation should be an executable part of your engineering process, not a manual review of a successful demo.

11. Open-Source Projects Worth Exploring

You don't have to build everything from scratch. These three projects are good starting points.

1. Promptfoo

https://github.com/promptfoo/promptfoo

Probably one of the easiest ways to start experimenting with Evals.

It supports:

• Prompt and model comparisons.
• Custom JavaScript assertions.
• LLM-as-a-Judge.
• RAG evaluations.
• Coding agent evaluations.
• Cost and latency checks.
• Regression testing.

What makes it particularly interesting for coding agents is that it can evaluate actual agent execution, inspect traces, and run tasks in separate workspaces.

A good starting point:

https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/

2. OpenAI Evals

https://github.com/openai/evals

An open-source framework for evaluating LLMs and LLM-based systems.

It provides a framework for defining evaluation tasks, creating custom evaluators, and running repeatable experiments.

It's useful when you want to build evaluation datasets around your own use cases rather than relying only on generic benchmarks.

3. Inspect AI

https://github.com/UKGovernmentBEIS/inspect_ai

An open-source evaluation framework developed by the UK AI Security Institute.

What makes it interesting is its support for more complex agent evaluations.

It provides building blocks for:

• Datasets.
• Agents and tools.
• Scorers.
• Sandboxed execution.
• Multi-step tasks.Source：https://dev.to/arif_sefa_f2d991d373eb93e/ai-evals-stop-trusting-your-agents-done-334d
  My favorites