An agent finishes a run and writes: "I sent the proposal to dana@acme.com and refunded the $45." The tool log for the same run shows the email API returned HTTP 202 (accepted, not sent) and the refund call timed out. The reply is not a hallucination in the usual sense. The agent did call the tools. It just described the outcome in stronger words than the evidence supports, and the user has no way to tell.

This post walks through a narrow, deterministic check for that gap: pull the action claims out of the final message, match each one to the tool calls from the same run, grade the evidence, and flag any claim whose wording outruns its grade. There's a small MIT-licensed Python implementation at the end that you can run on your own logs.

Why this happens

A few patterns produce most of these mismatches:

  • Accepted is not done. Many APIs return 202 or "status": "queued" when they take a request. Email providers, payment processors and job queues all do this. A model that sees a non-error response tends to report success.
  • Errors get smoothed over. A timeout or an ambiguous response comes back, the model retries or moves on, and the summary at the end says the task is done.
  • Batches get rounded up. Three of five sends succeed, and the reply says "sent all 5".
  • Umbrella phrases. "All set" or "everything is taken care of" covers every action in the run, including the one that failed.
  • Retries without idempotency keys. The opposite failure: the action happened twice.

Prompting the model to "only report what actually happened" helps a little, but the model is grading its own work, and it is reading the same ambiguous tool responses that misled it in the first place. The check needs to sit outside the model.

Claims vs. receipts

The approach is to treat the final message as a set of claims and the tool log as a set of receipts, then diff them:

  1. Extract claims. Find completed-action verbs in the reply ("sent", "booked", "refunded", "deleted") and skip the ones that are future, conditional or negated ("I will send", "I haven't booked"). Pull out what each claim is about: email addresses, order numbers, dollar amounts, counts like "all 5".
  2. Bind each claim to receipts. Match on action type, then on target. A claim that names dana@acme.com only binds to a send whose target contains that address.
  3. Grade the evidence and compare it with what the wording requires.

Evidence grades

Grade What the tool log shows Wording it supports
S0 No call for this claim "I did not ..."
S1 Call made, then error, timeout or unknown result "I tried to ..., not confirmed"
S2 Accepted or queued (HTTP 202, pending), or "done" with no object id "I queued / requested ..."
S3 Done, with an object id "I sent / booked / paid ..."
S4 Verified afterwards (delivered webhook, read-back) "... and it was delivered / confirmed"
F Provider reported failure (4xx/5xx, bounce) "I tried to ..., but it failed"

A plain past-tense claim needs S3. "Delivered", "confirmed" and "verified" need S4. Requiring an object id for S3 is deliberate: a provider that actually created something can usually tell you what it created.

Here's the grading function from the repo. It's ordinary code with no model involved:

def grade_call(call: dict) -> str: """Grade one tool-log entry. An explicit "grade" field wins.""" g = str(call.get("grade", "")).upper() if g in RANK: return g status = str(call.get("status", "")).lower().strip() http = call.get("http_status") has_id = bool(call.get("id") or call.get("object_id")) if status in FAILED or (isinstance(http, int) and http >= 400): return "F" if call.get("error") or status in ATTEMPTED: return "S1" if status in VERIFIED or call.get("verified") is True: return "S4" if status in ACCEPTED or http == 202: return "S2" if status in DONE or (isinstance(http, int) and 200 <= http < 300): return "S3" if has_id else "S2" return "S1"

Beyond single claims

Grading one claim at a time misses some cases, so the checker also looks for:

  • Wrong target. The claim names an address or id that no call of that action has.
  • Partial batch. "Sent all 5" against 3 distinct targets at S3. Retries to the same target count once.
  • Money mismatch. The amount in the reply differs from the amount in the log.
  • Umbrella claims. "All set" is flagged if any call in the run is below S3, or if there were no calls at all.
  • Duplicates. The same tool, action and target logged twice without a shared idempotency key.

Unsupported payment or delete claims are treated as stop-ship (exit code 2), because those are the ones that are expensive to get wrong.

Running it 

python3 -m saydo_diff --message examples/reply.txt --log examples/tool_log.json 
[GAP] sent S2 wording needs S3, best evidence is S2 [ok ] booked S3 supported at S3 [GAP] refunded S1 wording needs S3, best evidence is S1 [GAP] sent S3 partial batch: 3 of 5 reached S3 [GAP] (umbrella) S2 umbrella claim, but these calls are below S3: sendgrid.send (S2), stripe.refund (S1) [DUP] invoices.send send invoice 1003 logged 2 times without one idempotency key (entries [5, 6]) 4 gap(s) in 5 claim(s), gap rate 80% STOP-SHIP: a payment or delete claim is unsupported

That output is from the sample files in the repo, not from a production system. Add --json for a machine-readable report, which makes it easy to run in CI over saved transcripts and track the gap rate per tool over time.

Trade-offs

This is a narrow tool, and the limits are worth stating plainly:

  • Recall is limited. Claim extraction uses a fixed English verb list. A phrasing it doesn't know passes silently, so a clean report means "no gaps found", not "the reply is honest". The alternative, a model-based claim extractor, would catch more phrasings but brings back the non-determinism this approach tries to avoid. A middle ground is to let a model downgrade claims but never upgrade them.
  • The log is trusted as written. If the agent can write to the log, it can write its own evidence. For a real guarantee, the tool layer has to write the receipts and sign them with a key the model never sees.
  • Literal matching errs toward flagging. "Dana" in the reply doesn't match d.smith@acme.com in the log. That produces false positives, which is the cheaper failure here than false negatives.
  • Reporting isn't fixing. A diff after the fact tells you the gap rate. It doesn't stop the false sentence from reaching the user. Doing that means running the check in line, before the reply is sent, and deciding what to do with each failed claim: rewrite the clause, hold the reply or annotate it.
  • S3 needs an object id. Some tools don't return one. You can either change the tool to return one or set "grade": "S3" explicitly in your log, which moves the trust decision into your own code where it's visible.

Where to go from here

The free checker is a reasonable first step: point it at a week of saved transcripts and tool logs, and look at which tools have the highest gap rate. In many setups the fix is in the tool, not the prompt. Make it return an object id, map the provider's statuses to grades, or add an idempotency key.

If you want the in-line version, the Say-Do Gate is the paid companion kit. It adds HMAC-signed receipts, clause-level rewrites of the false part of a reply, holds for unconfirmed payments and deletes, late-confirmation status lines, a weekly Say-Do Scoreboard, and adapters for OpenAI and Anthropic tool calls, n8n, Zapier and Make. The trade-off is setup: you wrap your tools so they write receipts, rather than pointing a checker at a log you already have.