Most AI agent demos let the model call any tool it likes. That's fine until the model decides a frustrated customer deserves a $5,000 refund.

Prompts like "never refund more than $100" don't enforce anything. Code does. Here's a small pattern we use in AI agent development: tag every tool with a risk level, run every tool call through a policy check, and send risky calls to a human approval queue.

It works with any model provider or agent framework, because it sits between "the model asked for a tool" and "the tool actually runs."

1. Define tools with a risk level 

import { z } from "zod"; type Risk = "read" | "low_write" | "high_write"; interface Tool<I extends z.ZodTypeAny> { name: string; description: string; risk: Risk; input: I; run: (args: z.infer<I>) => Promise<unknown>; } const getOrderStatus: Tool<z.ZodObject<{ orderId: z.ZodString }>> = { name: "get_order_status", description: "Look up the status of an order by ID.", risk: "read", input: z.object({ orderId: z.string().min(1) }), run: async ({ orderId }) => ({ orderId, status: "shipped" }), // call your API here }; const issueRefund: Tool< z.ZodObject<{ orderId: z.ZodString; amount: z.ZodNumber }> > = { name: "issue_refund", description: "Refund part or all of an order.", risk: "high_write", input: z.object({ orderId: z.string().min(1), amount: z.number().positive().max(5000), }), run: async ({ orderId, amount }) => ({ orderId, refunded: amount }), }; const tools = [getOrderStatus, issueRefund] as const;

The schema is your first guardrail. Bad arguments never reach your API.

2. Write the policy as plain code 

type Decision = | { kind: "allow" } | { kind: "needs_approval"; reason: string } | { kind: "deny"; reason: string }; function policy(toolName: string, risk: Risk, args: any): Decision { if (risk === "read" || risk === "low_write") return { kind: "allow" }; if (toolName === "issue_refund") { if (args.amount <= 50) return { kind: "allow" }; return { kind: "needs_approval", reason: `Refund of $${args.amount}` }; } return { kind: "needs_approval", reason: `High-risk tool: ${toolName}` }; }

Keep this boring and testable. Product and compliance teams can read it, which you can't say about a 2,000-word system prompt.

3. Gate every tool call 

interface PendingAction { id: string; toolName: string; args: unknown; reason: string; createdAt: Date; } const approvalQueue: PendingAction[] = []; async function executeToolCall(toolName: string, rawArgs: unknown) { const tool = tools.find((t) => t.name === toolName); if (!tool) return { error: `Unknown tool: ${toolName}` }; const parsed = tool.input.safeParse(rawArgs); if (!parsed.success) { // Give the model a readable error so it can retry with fixed arguments return { error: parsed.error.issues.map((i) => i.message).join("; ") }; } const decision = policy(tool.name, tool.risk, parsed.data); log({ toolName, args: parsed.data, decision }); // trace every step if (decision.kind === "deny") return { error: decision.reason }; if (decision.kind === "needs_approval") { const action: PendingAction = { id: crypto.randomUUID(), toolName, args: parsed.data, reason: decision.reason, createdAt: new Date(), }; approvalQueue.push(action); return { status: "pending_approval", message: "A team member will review this request shortly.", actionId: action.id, }; } return tool.run(parsed.data as any); } function log(entry: Record<string, unknown>) { console.log(JSON.stringify({ at: new Date().toISOString(), ...entry })); }

Your agent loop calls executeToolCall instead of running tools directly. The model gets back either a result, a readable error or a "pending approval" message it can relay to the user.

4. Approve or reject from your back office 

async function approve(actionId: string, approvedBy: string) { const i = approvalQueue.findIndex((a) => a.id === actionId); if (i === -1) throw new Error("Action not found"); const [action] = approvalQueue.splice(i, 1); const tool = tools.find((t) => t.name === action.toolName)!; const result = await tool.run(action.args as any); log({ event: "approved", actionId, approvedBy, result }); return result; }

In production, the queue lives in your database, the approve and reject buttons live in your admin panel or Slack, and each decision is stored for audit.

5. Test the policy like any other code 

import { strict as assert } from "node:assert"; assert.equal(policy("get_order_status", "read", {}).kind, "allow"); assert.equal(policy("issue_refund", "high_write", { amount: 20 }).kind, "allow"); assert.equal( policy("issue_refund", "high_write", { amount: 900 }).kind, "needs_approval", );

Add these to CI, next to your agent eval suite. When someone changes a threshold, the tests tell you.

Why this matters

Agents fail in production for boring reasons: a bad argument, a missing permission check, a tool that does too much. A 60-line gate catches most of them, and it lets you give the agent more autonomy later with real data behind the decision.

If you're planning AI agent development for a business system and want help with the architecture, here's how our team approaches AI agent development services and AI integration.

How do you handle risky tool calls in your agents: approval queues, hard limits, or something else?

This post was written with AI and published by the author.