When a voice agent makes an outbound call, the first thing it has to work out is who, or what, answered. It might be a person saying "Hello?", a voicemail greeting, a company phone tree, or hold music. Outbound calling is part of what I work on in Quick Start, and this first moment shapes the rest of the call more than people expect. If the agent guesses wrong, it either pitches to a voicemail box for thirty seconds or sits quietly while a real person says "hello?" twice and hangs up.
Meta's Muse recently started placing calls to US businesses for its users, and those calls run straight into the same problem from the other side: phone trees, long holds, and recorded "please stay on the line" messages that sound a lot like a person. So I want to walk through how answering machine detection actually works, where it goes wrong, and how I'd structure an agent around it.
How answering machine detection decides
Most telephony providers offer answering machine detection (AMD), and it's less magical than the name suggests. Twilio's docs describe their version plainly: it isolates speech in the greeting and measures the periods of speech and silence, then uses that pattern to decide who answered.
The idea is that people and machines answer differently. A person usually says something short like "Hello?" and then stops to wait for you. A voicemail greeting is a longer stretch of speech, like "Hi, you've reached Sara, I can't take your call right now, please leave a message after the tone."
Twilio exposes the timing rules as parameters, and the defaults tell you a lot about the design:
| Parameter | Default | What it means |
|---|---|---|
MachineDetectionSpeechThreshold | 2400 ms | Speech shorter than this leans human, longer leans machine |
MachineDetectionSpeechEndThreshold | 1200 ms | Silence after speech that counts as "they finished talking" |
MachineDetectionSilenceTimeout | 5000 ms | Initial silence after which the result is unknown |
MachineDetectionTimeout | 30 s | Total time AMD may take before returning unknown |
The result arrives in an AnsweredBy field. With MachineDetection=Enable you get human, machine_start, fax or unknown. With DetectMessageEnd, Twilio keeps listening on a machine until the greeting ends, so you get values like machine_end_beep or machine_end_silence, which is what you want if you plan to leave a message after the tone.
Where the timing rules break
Because the decision is based on timing, the hard cases are the ones where people and machines swap their usual patterns.
The first is the long human greeting. Someone answering a business line might say "Good afternoon, Northside Dental, this is Maria, how can I help you?" That's easily longer than 2.4 seconds of continuous speech, so it can be marked as a machine. Twilio calls this a False Machine, and raising the speech threshold reduces it, at the cost of taking longer to detect real machines.
The second is the short voicemail greeting. Twilio's docs note that short voicemails often have 1 to 2 seconds of audio, then 1.2 to 2.4 seconds of silence, then more audio before the beep. To the detector, "Hi, it's Sara" followed by a pause looks exactly like a person waiting for you to speak. That's a False Human. Their suggested fix is raising the speech end threshold to around 2500 ms, so the pause counts as a gap inside the greeting. The trade-off is that every human detection now takes about that much longer too.
So you can't tune your way out of both errors at once. You pick which mistake costs you less for your use case, and you handle the other one in the agent itself.
Synchronous vs async detection
This is the part that matters most for how the call feels.
With synchronous AMD, which is Twilio's default, your call instructions run only after the detection result is ready. That's simple to code, but it means the person who picked up hears nothing while the detector is still deciding. They say "Hello?", wait, say it again, and some of them hang up before your agent has said a word.
With AsyncAmd=true, the call connects and your instructions run right away, while detection keeps going in the background. The result is posted to your AsyncAmdStatusCallback URL when it's ready. That lets the agent start a short, neutral opening while the answer is still unknown, and then switch behavior once the callback lands.
There's one detail in Twilio's docs that's easy to miss if your agent streams audio over Media Streams. While async AMD is analyzing a call, it uses one of the four forked audio streams allowed per call, and that limit is shared with Media Streams, SIPREC and real-time transcription. If you start several of those at once, you can hit the limit, so Twilio suggests starting them after the async AMD callback arrives.
Treat detection as a hint, not a verdict
I prefer to treat the AMD result as one input to a small state machine, rather than as the final answer. The agent also hears the audio itself, so it can correct a wrong guess. A rough version looks like this:
type CallState = | "connecting" // answered, AMD still running | "human" | "voicemail" | "phone_tree" | "on_hold"; function onAmdResult(answeredBy: string, state: CallState): CallState { if (state !== "connecting") return state; // the agent already learned more if (answeredBy === "human") return "human"; if (answeredBy.startsWith("machine")) return "voicemail"; return "connecting"; // "unknown": keep listening instead of guessing } The important line is the first check. If the transcript already showed the other side answering a question, a late machine_start callback shouldn't make the agent drop into voicemail mode in the middle of a real conversation. The reverse also helps: if the agent said hello and the other side kept talking right over it with "please leave a message after the tone", that's voicemail, whatever the detector said.
The unknown result deserves its own handling. It usually means a long silence or a strange greeting, so the safe move is a short prompt like "Hi, is this a good time?" and then listening, rather than launching into the full script.
Phone trees and hold are a different problem
AMD is built to separate people from voicemail. It wasn't designed for an agent that has to get through "press 1 for billing" and then wait on hold, which is what a Muse-style assistant does. In that case the agent needs a few extra behaviors.
On a phone tree, the agent should listen to the whole menu before choosing, then answer with DTMF tones rather than speech when the menu asks for a keypress. Twilio supports this with <Play digits="..."> in TwiML, or the SendDigits parameter when the call is created if you already know the path.
On hold, the agent should stay silent. Hold music and repeated recordings like "your call is important to us" will reach the speech-to-text step, and an agent that treats every transcript as a turn will start replying to the music. I'd keep the agent in an on_hold state where transcripts are only checked for signs of a live person, such as a new voice asking "how can I help you?" or a direct question that expects an answer.
The trickiest moment is the jump from recording to person. Many hold loops repeat a recorded voice every so often, and it can say "Hi, thanks for waiting" in a friendly human tone. One check I'd use is repetition: if the same sentence came up earlier in the call, it's almost certainly the recording.
How I'd test it
Timing rules and state machines are only as good as the audio you test them against, so I'd build a small set of real greetings from calls you're allowed to record. That set should include short and long voicemails, business greetings, people who answer with a long pause, phone trees, and hold loops. Then score each case on two things: whether the agent chose the right state, and how long the person waited in silence before the agent spoke.
That second number is the one users actually feel. A detector that's right every time but leaves people saying "hello?" into silence still loses the call.


