TL;DR: Agents built with only two exits, answer or error, decide anyway on the cases they should not. The third exit, abstention, has to be designed in code, tested in the eval set and recorded in the decision log.

The model's confidence is not the exit

The tempting shortcut is to ask the model how sure it is and set a threshold. What comes back is a well-formed sentence, or a number the model generated, not a calibrated probability. It can be one signal among others, but it cannot be the gate. A system that stops only when the model says it is unsure stops when the model happens to phrase things that way.

Designing the third exit

The exit belongs in code, outside the model, where it can be read and tested.

Start with explicit stop conditions, checked before and after the model runs: required fields missing, a validation that fails, a tool result outside the expected range, a value that contradicts the master data. They are cheap, deterministic and easy to explain.

Then give the output a shape that allows abstention. The schema the agent fills in has a "cannot decide" value next to the normal outcomes, with a reason code from a short list. Free text is fine as an explanation, but the code is what routes the case.

Route those cases to a queue with an owner. An abstention that lands nowhere is just a slower failure. Someone has to pick it up, decide, and feed the case back into the eval set.

Finally, record the abstention in the decision log like any other decision: input references, versions, the reason code, who resolved it and how. An auditor asking how the system behaves under uncertainty gets a list, not an anecdote.

Measuring it

An eval set made only of cases with a clear answer cannot tell you whether the third exit works. Add cases where the correct output is "I don't know": incomplete inputs, conflicting rules, documents out of scope. Then track two numbers over time. How often the agent abstains, which is the workload you send to people. And how often it should have abstained and did not, which is the number that matters for risk.

The two pull in opposite directions. Tighten the conditions and the queue grows; loosen them and the silent errors grow. Where to set the balance is a business decision. Write it down with the owner of the process instead of letting the developer tune it quietly.

What this means for a delivery

An agent that never abstains is not accurate. It has not been tested on the hard cases. Put the third exit in the design document, in the output contract and in the acceptance tests, before the first demo.

What does your agent do today when it should not decide?

Articolo originale: https://www.omarbaruzzo.it/en/blog/agente-ai-deve-poter-dire-non-so