Evaluating AI agents quickly leads to an uncomfortable question: how do you actually notice that your agent got worse after a change? A new system prompt, a model switch or a reworded tool description is enough, and suddenly the assistant reserves books that are sitting on the shelf. Without tests, you only find out when someone complains.

For regular code the answer is simple: unit tests. With agents that only half works, because the same model answers the same question differently from run to run. That is exactly what evals are for: repeatable checks that give an agent real tasks and grade the result. In this article I build a small eval suite in TypeScript with you, from an empty folder to a run in CI. I deliberately skip frameworks so you can see what happens under the hood.

In short

Evals give an AI agent real tasks in a fresh test environment and then grade the outcome, not the path that led there. To get started, 20 to 50 tasks taken from real failures, several trials per task and the pass^k metric for everything that has to work reliably are enough. The biggest pitfall is a broken grader, which is why you should read the transcripts regularly.

What you will learn in this article

  • What task, trial, grader and transcript mean in agent evals
  • How to build a small eval harness in TypeScript without a framework
  • When a code grader is enough and when a model judge makes sense
  • Why pass@k and pass^k tell such different stories
  • How regression evals in CI catch problems before a release

Why evaluating AI agents matters

An agent has three parts: the model, a loop that connects the model to its tools, and the tools themselves. You turn all three knobs in everyday work. You swap the model because a newer one is faster or cheaper. You refine the system prompt. On top of that you polish tool descriptions, as I showed in How to write tool descriptions for AI agents. Each of these changes can improve one thing and break another.

Without evals, all you have is the gut-feeling test: try three or four prompts by hand, the answer looks fine, done. Anthropic's engineering team sums up the value of evals in one sentence: “Evals make problems and behavioral changes visible before they affect users.” That is the whole point. You want to see the regression before your users feel it.

There is also a quirk of language models. A single run tells you little because answers vary. If a task succeeds once, you do not know whether it succeeds nine times out of ten or three. Evaluating AI agents therefore means running every task several times and computing metrics from the results. More on that later.

The key terms in agent evals

Before we write code, a shared vocabulary helps. I stick to the terms from Anthropic's article “Demystifying evals for AI agents” because they map well onto your own code. The table also shows where you will find each term in the example.

Term Meaning In the example
Task A single check: the input plus the criterion for passing it an entry in tasks.ts
Trial One single run of a task one call to runTrial()
Grader Logic that scores one aspect of the result functions in graders.ts
Transcript The full record of a trial with all messages and tool calls one line in results/*.jsonl
Outcome The state of the environment after the trial the reservations in the library
Eval harness The infrastructure that runs, grades and summarizes tasks runner.ts
Agent harness The loop that turns the model into an agent agent.ts
Eval suite A set of related tasks all tasks of the lending assistant

Keep two kinds of tasks apart from day one. Capability evals ask what the agent already does well. They may start with low pass rates, because they show you where you want to go. Regression evals, on the other hand, ask whether the agent can still do everything it could do before. They should sit close to 100 percent. Once a capability task reaches consistently high scores, it moves into the regression suite.

A lending assistant as the example

The example is a made-up lending assistant for a city library. It has three tools: search_books searches the catalog, check_availability shows free copies, and reserve_book creates a reservation. Two business rules come on top. A reservation is only made when no copy is available. And if the member number is missing, the agent asks for it instead of guessing.

What I like about this example: whether the agent did its job is visible in the state of the library. Is the right reservation there after the trial? Is there no wrong one? That is exactly the kind of outcome code can check reliably. This way you can evaluate AI agents without needing a human or a second model for every answer. The following diagram shows how a trial runs.

🎬 How one trial runs in the eval harness – this animated diagram only runs on my site: watch it on sascha-ms.de

Prerequisites

You need Node.js 22 or later and npm. For real runs you need either an API key for the Claude API or an AWS account with access to a Claude model in Amazon Bedrock. The tests for the harness itself run without a key and at no cost. Basic TypeScript is all you need; experience with agent frameworks is not required.

Building the eval suite step by step

1. Create the project

Create a new folder and install the Anthropic SDK, the Bedrock SDK and, as development tools, TypeScript and tsx. With tsx you run TypeScript files directly without compiling them first.

mkdir ai-agent-evals && cd ai-agent-evals npm init -y npm install @anthropic-ai/sdk@1.3.0 @anthropic-ai/bedrock-sdk@1.1.0 npm install -D typescript@7.0.2 tsx@4.23.15 @types/node@22

Then replace package.json with this version. The important parts are "type": "module" and the three scripts.

{ "name": "ai-agent-evals", "version": "1.0.0", "private": true, "type": "module", "scripts": { "typecheck": "tsc --noEmit", "test": "node --import tsx --test test/*.test.ts", "evals": "tsx src/run-evals.ts" }, "dependencies": { "@anthropic-ai/bedrock-sdk": "1.1.0", "@anthropic-ai/sdk": "1.3.0" }, "devDependencies": { "@types/node": "^22.20.0", "tsx": "4.23.15", "typescript": "7.0.2" } } 
// tsconfig.json { "compilerOptions": { "target": "ES2023", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "noEmit": true, "skipLibCheck": true, "allowImportingTsExtensions": true, "types": ["node"] }, "include": ["src", "test"] }

2. A test environment with a fresh state

The library lives purely in memory. That is on purpose: every trial gets a new, clean state from createLibrary(). If trials shared a database, a reservation from trial 1 could skew the result of trial 2. Errors like that are especially nasty because they look random.

// src/library.ts // A small, made-up city library as the test environment for the agent. // Every trial gets a fresh state from createLibrary(). export interface Book { id: string; title: string; author: string; copies: number; } export interface Reservation { bookId: string; memberId: string; } export interface LibraryState { books: Book[]; loans: Record<string, number>; // copies on loan per book ID reservations: Reservation[]; } export function createLibrary(): LibraryState { return { books: [ { id: 'b-101', title: 'The Lighthouse on the Dyke', author: 'Jana Thiel', copies: 2 }, { id: 'b-102', title: 'Cooking with Kids', author: 'Ole Brandt', copies: 1 }, { id: 'b-103', title: 'TypeScript for Beginners', author: 'Mira Okafor', copies: 3 }, { id: 'b-104', title: 'The Garden in Winter', author: 'Paul Seidel', copies: 1 }, ], loans: { 'b-101': 2, 'b-102': 0, 'b-103': 1, 'b-104': 1 }, reservations: [], }; } export function availableCopies(state: LibraryState, bookId: string): number { const book = state.books.find((b) => b.id === bookId); if (!book) return 0; return book.copies - (state.loans[bookId] ?? 0); }

3. The agent's tools

The tools validate their own input and answer problems with an understandable error message. That way the agent can correct itself instead of getting stuck. Note that reserve_book refuses to reserve an available book. The eval still checks whether the agent even tries, because a tool in a real system is not always that strict.

// src/tools.ts // Three tools the agent may use. Their descriptions are part of what // your evals test as well. import { availableCopies, type LibraryState } from './library.ts'; export interface ToolDefinition { name: string; description: string; input_schema: { type: 'object'; properties: Record<string, unknown>; required: string[]; }; } export interface ToolResult { content: string; isError: boolean; } export const tools: ToolDefinition[] = [ { name: 'search_books', description: 'Searches the city library catalog by title or author. ' + 'Returns book IDs (e.g. "b-101") that the other tools need.', input_schema: { type: 'object', properties: { query: { type: 'string', description: 'Search term, e.g. part of the title' } }, required: ['query'], }, }, { name: 'check_availability', description: 'Shows how many copies of a book can be borrowed right now. Book ID from search_books.', input_schema: { type: 'object', properties: { bookId: { type: 'string' } }, required: ['bookId'], }, }, { name: 'reserve_book', description: 'Reserves a book for a member. Only use it when no copy is available ' + 'and the user has given their member number (format "M-<number>"). Never guess.', input_schema: { type: 'object', properties: { bookId: { type: 'string' }, memberId: { type: 'string', description: 'Member number, e.g. "M-17"' }, }, required: ['bookId', 'memberId'], }, }, ]; export function executeTool(state: LibraryState, name: string, input: unknown): ToolResult { const args = (input ?? {}) as Record<string, string>; switch (name) { case 'search_books': { const query = String(args.query ?? '').toLowerCase(); const hits = state.books.filter( (b) => b.title.toLowerCase().includes(query) || b.author.toLowerCase().includes(query), ); return { content: JSON.stringify(hits.map(({ id, title, author }) => ({ id, title, author }))), isError: false }; } case 'check_availability': { if (!state.books.some((b) => b.id === args.bookId)) { return { content: `Unknown book ID "${args.bookId}". Use search_books first.`, isError: true }; } return { content: JSON.stringify({ bookId: args.bookId, available: availableCopies(state, args.bookId) }), isError: false }; } case 'reserve_book': { if (!state.books.some((b) => b.id === args.bookId)) { return { content: `Unknown book ID "${args.bookId}".`, isError: true }; } if (!/^M-\d+$/.test(args.memberId ?? '')) { return { content: 'Invalid member number. Please ask the user.', isError: true }; } if (availableCopies(state, args.bookId) > 0) { return { content: 'The book is available, no reservation needed.', isError: true }; } const exists = state.reservations.some((r) => r.bookId === args.bookId && r.memberId === args.memberId); if (exists) { return { content: 'This member already has a reservation.', isError: false }; } state.reservations.push({ bookId: args.bookId, memberId: args.memberId }); return { content: 'Reservation created.', isError: false }; } default: return { content: `Unknown tool "${name}".`, isError: true }; } }

4. The agent loop

The loop is deliberately simple. It sends the history to the model, runs the requested tools, appends the results and repeats until the model stops calling tools or the maximum number of turns is reached. Along the way it collects everything for the transcript: messages, tool calls, turns and tokens.

The trick for testability lies in the ModelClient type. The loop only knows a function that takes a request and returns a response. Whether the Claude API, Amazon Bedrock or a test script sits behind it does not matter to the loop.

// src/agent.ts // A lean agent loop: ask the model, run tools, return the results // until the model is done. Everything ends up in the transcript. import type { LibraryState } from './library.ts'; import { executeTool, tools, type ToolDefinition } from './tools.ts'; export type ContentBlock = | { type: 'text'; text: string } | { type: 'tool_use'; id: string; name: string; input: unknown } | { type: 'tool_result'; tool_use_id: string; content: string; is_error: boolean }; export interface Message { role: 'user' | 'assistant'; content: string | ContentBlock[]; } export interface ModelRequest { system: string; messages: Message[]; tools: ToolDefinition[]; max_tokens: number; } export interface ModelResponse { content: ContentBlock[]; stop_reason: string | null; usage: { input_tokens: number; output_tokens: number }; } // Any function with this signature can be the model: the Anthropic API, // Amazon Bedrock or, in tests, a script with fixed answers. export type ModelClient = (request: ModelRequest) => Promise<ModelResponse>; export interface ToolCall { name: string; input: unknown; isError: boolean; } export interface Transcript { messages: Message[]; toolCalls: ToolCall[]; finalText: string; turns: number; inputTokens: number; outputTokens: number; stopReason: string | null; } export const SYSTEM_PROMPT = 'You are the lending assistant of the city library. Help members find, ' + 'check and reserve books. Use the tools instead of guessing. If information ' + 'such as the member number is missing, ask for it. Answer briefly and kindly.'; export async function runAgent( client: ModelClient, state: LibraryState, prompt: string, maxTurns = 8, ): Promise<Transcript> { const messages: Message[] = [{ role: 'user', content: prompt }]; const transcript: Transcript = { messages, toolCalls: [], finalText: '', turns: 0, inputTokens: 0, outputTokens: 0, stopReason: null, }; while (transcript.turns < maxTurns) { const response = await client({ system: SYSTEM_PROMPT, messages, tools, max_tokens: 1024 }); transcript.turns += 1; transcript.inputTokens += response.usage.input_tokens; transcript.outputTokens += response.usage.output_tokens; transcript.stopReason = response.stop_reason; messages.push({ role: 'assistant', content: response.content }); const text = res...
Source：https://dev.to/sascha-ms/evaluating-ai-agents-small-evals-in-typescript-pbg
My favorites