Bland published a page for Tofu-1, its new speech-to-speech model, and one section on it has stuck with me. They call it the TV Test. The agent asks, "Just to confirm, you want to cancel your policy?" The caller says nothing because she's still looking for her policy number. A TV in the room says "Yes!" A transcript-based agent can take that yes as her answer and cancel the policy.
Bland marks this as an illustrative scenario, not a recorded model output, so I'm not treating it as a benchmark result. But the failure it describes is real for anyone running a cascade (STT, then LLM, then TTS). The transcript says "yes", and nothing in the text tells you who said it.
I build Quick Start AI, a product for inbound and outbound AI calling, campaigns and lead qualification, so I think about this a lot. This post is about the parts you can control today without a new model: how audio cleanup helps, where it stops helping, and how to design confirmations for risky actions so one stray word can't trigger them.
Why the transcript loses the speaker
In a cascade, the STT turns audio into words, and the words are what the LLM sees. Bland's page puts it well: each step discards part of the call, like who was speaking, who they were speaking to, and whether they had finished.
A TV is the hard case because what it plays is clear speech, often louder than the caller. A noise filter is trained to keep speech and remove everything else, so the TV's "Yes!" passes straight through. On a phone line it gets worse because you usually get one mixed mono channel. Twilio Media Streams, for example, sends audio/x-mulaw at 8000 Hz with one channel. There's no separate track for the TV.
Background voice cancellation helps, with limits
There are models built for this. Krisp's Background Voice Cancellation (BVC) is designed to remove other people's voices around the user, plus noise and reverb, without voice enrollment. LiveKit ships it for agents through its Python and Node.js plugins, with a standard BVC() model and a BVCTelephony() variant for phone calls. In Node it looks like this:
import { BackgroundVoiceCancellation } from '@livekit/noise-cancellation-node'; import { AudioStream } from '@livekit/rtc-node'; const stream = new AudioStream(track, { noiseCancellation: BackgroundVoiceCancellation(), }); I'd turn this on for any agent that takes calls in homes, cars or offices. But I wouldn't let it carry the safety of a destructive action by itself, for three reasons.
First, read the fine print. Krisp's own docs say its BVC model works at 32 kHz, works well above 8 kHz, and isn't compatible with narrowband audio at 8 kHz or lower. That's exactly what a PSTN call looks like, which is why the telephony variant exists, and you should test that variant on your own call audio before trusting it. Second, any model makes mistakes on some share of calls, and a cancelled policy is expensive to undo. Third, Bland makes a point I agree with: volume can't settle the question. A loud TV can be irrelevant, and a quiet correction from the caller can be the most important thing on the call.
So treat audio cleanup as a layer that lowers how often the bad input shows up. The confirmation design is what decides what happens when it does show up.
Split your tools by risk
Not every tool call needs the same care. Looking up an appointment slot is cheap to get wrong. Cancelling a policy, sending a payment, deleting an account, or marking a lead as do-not-call is not. I like to tag each tool with a risk level in code, so the rule doesn't live only in the prompt:
type Risk = 'read' | 'reversible' | 'destructive'; const tools = { getPolicy: { risk: 'read' as Risk }, bookCallback: { risk: 'reversible' as Risk }, cancelPolicy: { risk: 'destructive' as Risk }, }; Reads run right away. Reversible actions run, and the agent tells the caller what it did. Destructive actions go through a confirmation gate that the LLM can't skip.
A confirmation a TV can't easily pass
The weak pattern is a yes/no question. "Do you want to cancel?" can be answered by anyone in the room with one word. A better pattern asks the caller to say something only they would say in that moment, tied to the specific action.
Compare these two prompts:
- "Do you want to cancel your policy?"
- "I'm about to cancel the home policy ending in 4471. If that's right, please say the last four digits back to me."
The second one is longer, but it does several jobs. It names the exact thing being changed, so a confused caller can catch a mistake. It also asks for an answer a TV is very unlikely to say at that moment. A bare "yes" from anyone doesn't count.
On phone calls there's another option: keypad input. Twilio's bidirectional Media Streams send a dtmf event when someone presses a key, and the event says which digit was pressed. "Press 1 to confirm the cancellation" is clumsy for small talk, but for one high-stakes step it's a clear signal that someone holding the phone made a choice. Keypad tones can still show up in the audio from other sources, so I'd treat DTMF as strong evidence, not proof.
Checks the gate can run
Once a confirmation answer arrives, the gate can look at more than the words. Here's a sketch of what it checks:
interface PendingAction { id: string; tool: 'cancelPolicy'; args: { policyId: string; last4: string }; askedAt: number; // ms, when the agent finished asking expiresAt: number; } interface HeardAnswer { text: string; words: { word: string; confidence: number; speaker?: number }[]; startedAt: number; // ms, when this speech began callerSpeaker?: number; } function checkConfirmation(p: PendingAction, a: HeardAnswer): 'ok' | 'reask' | 'expired' { if (Date.now() > p.expiresAt) return 'expired'; // Must contain the specific digits, not just "yes". const digits = a.text.replace(/\D/g, ''); if (!digits.endsWith(p.args.last4)) return 'reask'; // Speech that started before the question ended is not an answer to it. if (a.startedAt < p.askedAt) return 'reask'; // Low ASR confidence on the key words means we ask again. if (a.words.some((w) => /\d/.test(w.word) && w.confidence < 0.8)) return 'reask'; // If diarization says a different speaker answered, ask again. if (a.callerSpeaker !== undefined && a.words.some((w) => w.speaker !== undefined && w.speaker !== a.callerSpeaker)) { return 'reask'; } return 'ok'; } A few notes on this. The digit check assumes your STT writes spoken numbers as digits; Deepgram's smart_format option does that kind of formatting, and other providers have their own settings. The 0.8 threshold is a placeholder, not a tuned number; you'd pick yours from your own call data. The speaker check depends on your STT. Deepgram, for example, puts a speaker integer on each word when diarization is on, but in live streaming it doesn't return the speaker_confidence field that pre-recorded results have. Diarization on a short "4471" can also be shaky. That's why the speaker check here only makes the gate stricter. It never approves anything on its own.
The most important rule isn't in the function. The cancelPolicy tool on the server should refuse to run unless there's a pending action with a matching id that passed this check. If the confirmation only lives in the prompt, a model that hears "yes" can still call the tool directly. Put the gate in the code path that does the work.
What a good re-ask sounds like
When the gate says reask, the agent shouldn't repeat the same sentence louder. It should say why it's asking again, briefly, and keep the specifics:
"Sorry, I didn't catch that clearly. To cancel the policy ending in 4471, please say those four digits, or press 1 on your keypad."
If the gate fails twice, stop trying to finish the action on this call. Offer a callback, a human handoff, or a confirmation link by SMS, depending on what your product supports. Bland's page has a line I like here: ignoring the television must never become a reason to guess at what the caller said. Guessing is the one thing the gate exists to prevent.
Add the TV Test to your evals
You don't need a new model to test this. Take a recorded or synthetic call where the agent asks a confirmation question, then mix in a second voice saying "yes" while the caller is silent. Run it through your pipeline and check one thing: did the destructive tool run? Then try variations, like the second voice speaking over the agent's question, the caller saying the digits quietly while the TV is loud, and the caller saying "wait, no" right after the TV's yes.
These cases are cheap to build, and they test the part that matters. A model that hears better makes them pass more often. A gate in your tool layer makes sure a wrong pass doesn't cost a caller their policy.
