I've been building AI agents for a while, and I keep hitting the same wall.
It's not the model hallucinating. It's not the API being slow. It's that the agent works perfectly in my test environment, with my 20-message context, and then fails in production with a 200-message context, a different tool chain, and a user who asks something I didn't anticipate.
The failure modes are different from traditional software:
- The agent doesn't crash. It just makes a slightly worse decision.
- The tests pass. The context window just got too big.
- The demo works. The real user asks a question that's 5% different, and the whole chain breaks.
I wrote about this before ("retry loops are expensive"), but I'm curious about the broader problem. How do you test an agent for the "messy context" it'll see in production, without shipping it to production first?
What does your test suite look like for agents? Do you have a "context window full of noise" test, or is it still mostly happy-path?
