I've been thinking about what happens to an AI agent when the model underneath it changes. The code doesn't change, but the behavior can. A prompt that worked well on one model may give different answers on another, and a tool call that was reliable may start failing quietly.
How are you handling this today? Some questions I'm curious about:
–When you upgraded or switched models, what broke first?
–Did you have tests for agent behavior, or did you find out from users?
–Are your prompts, tool definitions, and test cases stored together, or in different places?
–Did you build any manual process to catch these changes, and what does it look like?
–What's the last model change that surprised you?
–How do you know when an agent has gotten worse?
–What do you do before you upgrade a model?
–What tool do you wish existed for this?
If you've solved this, I'd like to hear how. If you haven't, I'd like to hear that too. Share what you've run into in the comments.