Friday AI Question. This week's question is one I think most teams never ask out loud:

When your AI feature doesn't know the answer, what does it actually do?

Pick one (and tell me in the comments):

  • A) It always answers. Something is better than nothing.
  • B) It refuses only on hard-coded rules (blocked topics, missing permissions).
  • C) It abstains when the evidence is weak: says "I couldn't find this", shows what it searched, and offers a next step.
  • D) C, plus our evals score "I don't know" separately from wrong answers.

My lean: C on day one, D as soon as you have an eval set. Here's why.

The scoreboard problem

OpenAI's September 2025 post Why language models hallucinate argues that models hallucinate partly because standard training and evaluation reward guessing over acknowledging uncertainty.

Their analogy is a multiple-choice test: if you don't know, a wild guess might be right, while leaving it blank guarantees zero. When models are graded only on accuracy, guessing wins.

They split answers into three buckets: accurate, errors, and abstentions. And they show it with SimpleQA numbers from the GPT-5 system card:

Metric gpt-5-thinking-mini OpenAI o4-mini
Abstention rate 52% 1%
Accuracy rate 22% 24%
Error rate 26% 75%

o4-mini looks slightly better on accuracy (24% vs 22%). But its error rate, its rate of confidently wrong answers, is 75% vs 26%.

If your only metric is "% correct", you'd ship the model that is wrong three times as often.

Why this matters for your feature, not just for leaderboards

Your internal evals are a scoreboard too. If your RAG or support bot eval only asks "was the answer right?", every "I don't know" counts as a failure, and every prompt tweak that makes the model guess more will look like an improvement.

The post's suggested fix: penalize confident errors more than uncertainty, and give partial credit for appropriate expressions of uncertainty. OpenAI's Model Spec says the same thing in product terms: it's better to indicate uncertainty or ask for clarification than to give confident information that may be incorrect.

What option C looks like in practice

  • Give the model permission to abstain. Say explicitly in the instructions that "I couldn't find this in the provided documents" is a valid answer.
  • Abstain on evidence, not vibes. If retrieval returns nothing relevant, don't hand the model an empty context and hope.
  • Make abstention useful. Show what was searched, suggest a rephrase, or hand off to a human.
  • Log abstentions. A cluster of "I don't know" on the same topic is a content gap you can fix.

What option D adds

Split your eval results into three numbers instead of one: correct, wrong, abstained. Then decide what a wrong answer costs in your product compared to an abstention. For a medical or billing bot, a wrong answer is much worse than "I don't know". For a brainstorming tool, maybe not.

Once you track errors separately, you can see whether a change made the model smarter, or just bolder.

Your turn

A, B, C or D? And if you track abstentions: what abstention rate do you consider healthy?

References