There's a trap in agent outcome reporting that looks like rigour but isn't.
If you define what "success" means after seeing how the agent resolved an action, you'll fit the definition to the outcome. The report reads real data. The dishonesty is in when the threshold was set.
The threshold trap
When did you decide what "close enough to expected" means?
// Set before dispatch — honest const SUCCESS_THRESHOLD = { stateChanged: true, timestampAfterDispatch: true, matchesExpectedOutput: true } // Tuned after seeing results — not honest // "We decided 2 out of 3 is fine after noticing // timestampAfterDispatch was flaky in production" The second version isn't wrong because it's less strict. It's wrong because the strictness was decided in response to what looked convenient.
How this shows up in StareBrain
Early on, UNRESOLVED outcomes started quietly resolving into SUCCESS when the state check was ambiguous. "The action probably ran, the system looks right, call it done." That's not a threshold — it's a post-hoc justification.
enum class ActionStatus { CONFIRMED, // State verified against pre-defined threshold FAILED, // State verified as unchanged DENIED_UNRESOLVED // State unverifiable — not SUCCESS by default } The rule Set the definition of success before you know what the result will be. If the threshold needs to change based on real production behaviour, change it explicitly — not by quietly fitting it to a distribution you've already seen.
Why this matters more for agents than regular software
A traditional API either returns the expected response or it doesn't. The success definition is in the contract. An agent acts on the world. The world doesn't return a typed response. The more capable the agent, the more ways it has to produce an outcome that looks like success without actually being one.
Set it first.
StareBrain is an Android AI agent: say what you want done, see exactly what it's about to do, confirm before anything runs. Pre-launch — waitlist open.
