The demo went perfectly. Stakeholders nodded, someone called it magic, and we released the agent to a small group of real users the following Monday. Within an hour, it had issued a refund against the wrong order, called the same tool eleven times in a row, and replied to a customer in a language they never wrote in.

Nothing was wrong with the model. What failed was everything around it. This post covers what broke, why the demo hid it, and the specific changes that made the agent survive contact with real people.

The Setup

The agent handled support tickets for an e-commerce back office. It read an incoming message, looked up the order through an internal API, drafted a reply, and could issue small refunds through a tool call. The stack was simple: a hosted LLM with function calling, a Python orchestration loop, and a few REST tools.

In the demo, it was flawless. I typed “Where is my order ORD-10293847?” and it fetched the status, wrote a polite answer, and finished in two steps.

Why the Demo Lied to Me

A demo is a controlled experiment where you are the only variable. I had made four unconscious choices that guaranteed success:

  • Clean inputs: I wrote short, well-formed requests with valid order IDs.
  • A cooperative user: I never changed my mind halfway or asked two things at once.
  • A healthy backend: Every API call returned quickly with the expected shape.
  • A forgiving judge: I decided what “good” meant, after the fact.

This gap between a convincing prototype and a dependable system is exactly what custom AI agent development services exist to close. The model call is the easy part. The engineering around it (validation, limits, observability, and testing) is where the real work lives.

Real users violate all four of my demo assumptions within minutes. Here is what that looked like, one failure at a time.

Failure 1: Inputs I Never Imagined

Real tickets arrived as pasted email threads with signatures, quoted replies, and three order numbers. One was a forwarded message in Portuguese. Another contained only “same problem as last time.”

The agent treated all of this as equally trustworthy instructions and picked the first ID it saw, which was in the quoted history rather than the actual complaint.

The fix was a normalization step before the model sees anything: strip quoted replies, detect language, extract candidate IDs, and flag ambiguity. If a message contains more than one candidate order, the agent asks a clarifying question instead of guessing. This single change removed most of our wrong-order incidents.

Failure 2: Tool Calls With Plausible but Wrong Arguments

Function calling gives you structured output, but structured does not mean correct. The model sometimes produced an order ID that matched the format but did not exist, or a refund amount larger than the order total.

I had been trusting arguments because they parsed as JSON. That is not validation. Now every tool call passes through a strict schema, and failures go back to the model as readable errors:

from pydantic import BaseModel, Field, ValidationError class RefundArgs(BaseModel): order_id: str = Field(pattern=r"^ORD-\d{8}$") amount_cents: int = Field(gt=0, le=50_000) reason: str = Field(min_length=5) def run_tool(name: str, raw_args: str) -> dict: if name == "issue_refund": try: args = RefundArgs.model_validate_json(raw_args) except ValidationError as e: return {"error": "invalid_arguments", "details": e.errors()} order = orders_api.get(args.order_id) if order is None: return {"error": "order_not_found"} if args.amount_cents > order.total_cents: return {"error": "amount_exceeds_order_total"} return refunds_api.create(args)

Returning the error as a tool result lets the model correct itself on the next step, which it usually does. The backend checks matter just as much as the schema, because the schema cannot know whether an order exists.

Failure 3: The Infinite Loop

When the orders API timed out, the agent did what any persistent system does: it tried again. And again. Eleven identical calls later, we had burned tokens and hit a rate limit.

A raw agent loop has no concept of “this is not working.” You have to give it one. I added a step budget and a repeated-call detector:

MAX_STEPS = 8 def run_agent(messages): seen_calls = set() for step in range(MAX_STEPS): resp = llm.chat(messages, tools=TOOLS) if not resp.tool_calls: return resp.content for call in resp.tool_calls: signature = (call.name, call.arguments) if signature in seen_calls: return escalate("repeated_tool_call", messages) seen_calls.add(signature) messages.append(run_tool_message(call)) return escalate("step_budget_exceeded", messages)

Escalation matters here. When the agent gives up, it should hand the case to a human with the full context attached, not return a vague apology.

Failure 4: I Could Not See What Happened

When the first bad refund landed, my debugging tool was print. I had final answers but not the path that produced them: which tool ran, with what arguments, how long it took, and what came back.

Agents are non-deterministic pipelines, so you need traces, not just logs. Every step now emits a structured event:

trace.emit({ "run_id": run_id, "step": step, "tool": call.name, "args_hash": hash_args(call.arguments), "latency_ms": elapsed_ms, "tokens_in": usage.input, "tokens_out": usage.output, "outcome": "ok" if "error" not in result else result["error"], })

This turned a bad run into a five-minute investigation instead of a guessing game. We could also see aggregate patterns, such as which tool failed most often and which step consumed the most tokens.

Failure 5: Acting When It Should Have Asked

The most expensive bug was behavioral. When uncertain, the agent acted anyway, because language models are fluent and fluency looks like confidence.

I split tools into two classes. Read-only tools run freely. Anything that changes state, like refunds or cancellations, goes through an approval gate:

RISKY_TOOLS = {"issue_refund", "cancel_order", "update_address"} def run_tool_message(call): if call.name in RISKY_TOOLS and not is_approved(call): ticket_id = review_queue.add(call, context=current_messages()) return tool_msg(call, {"status": "pending_human_review", "ticket": ticket_id}) return tool_msg(call, run_tool(call.name, call.arguments))

For low-risk, low-value refunds, we later relaxed the gate with explicit limits. Starting strict and loosening with evidence is far safer than starting open and tightening after an incident.

The Fix That Mattered Most: Evals From Real Failures

The best change wasn’t in the agent at all. I started saving every failed conversation as a test case and replaying them after each change:

def test_regressions(): for case in load_jsonl("evals/real_failures.jsonl"): result = agent.run(case["input"]) assert case["check"](result), f"Regression: {case['id']}"

My demo had never been a test. It was a performance. A regression set built from real failures is the first thing that tells you whether a prompt tweak helped or quietly broke three other behaviors. It also turns every production incident into permanent protection.

What I’d Do Differently Next Time

If I were starting over, I would change the order of work:

  • Collect real inputs first: Even twenty messy examples beat a polished demo script.
  • Define failure before success: Decide what the agent must never do, then build the guardrails.
  • Validate every tool boundary: Treat model output like untrusted user input.
  • Budget everything: Steps, tokens, time, and money all need ceilings.
  • Instrument from day one: You cannot fix what you cannot replay.
  • Keep humans in the loop for state changes: Remove them only when the data justifies it.

Outside perspective pays off early, too. A friend who does AI consulting asked me one question before launch that I couldn’t answer: “What happens when the agent is wrong?” Had I answered it during design instead of after the incident, most of this post wouldn’t exist.

Conclusion

The agent did not fail because the model was weak. It failed because I had built a demo and called it a product. Production brings messy inputs, flaky dependencies, ambiguous requests, and consequences, and an agent needs engineering for each.

The good news is that these problems are tractable. Normalize inputs, validate tool calls, cap loops, trace everything, gate risky actions, and test against real failures. None of it is glamorous, and all of it is the difference between a demo that impresses and a system people can trust.

What was the first thing that broke when your agent met a real user? Share it in the comments. I’d love to compare failure modes.

Frequently Asked Questions

1. Why do AI agents that work in a demo fail in production?

A demo uses clean inputs, a cooperative user, and a healthy backend, while production brings messy messages, ambiguous requests, slow APIs, and real consequences. The model usually isn’t the problem. What fails is the missing engineering around it: input normalization, argument validation, loop limits, and testing.

2. How do I stop an AI agent from getting stuck in an infinite loop?

Set a hard step budget and track repeated tool calls. If the agent makes the same call with the same arguments twice, or hits the step limit, stop the run and escalate to a human with the full context attached. It also helps to cap tokens, time, and cost per run.

3. Should AI agents be allowed to take actions like issuing refunds automatically?

Start with read-only tools running freely and put anything that changes state, such as refunds, cancellations, or address updates, behind a human approval gate. Once you have real data showing the agent is reliable, relax the gate for low-risk, low-value actions with explicit limits.

4. How do I test an AI agent properly?

Don’t treat the demo as a test. Save every real failed conversation as a regression case and replay the full set after each prompt or code change. This shows whether a tweak helped or quietly broke other behaviors, and it turns each production incident into permanent protection.

5. What should I log when running an AI agent in production?

Log traces, not just final answers. For every step, capture the run ID, which tool was called, hashed arguments, latency, token usage, and the outcome (success or error type). This turns a bad run into a quick investigation instead of a guessing game, and helps you spot patterns like which tool fails most often.