A caller rings a kitchen shop about a cast iron skillet that arrived cracked, hoping to get their money back. The voice agent politely replies, "Let me process that refund for you now." While the payment provider processes the transaction in the background, everyday life intervenes: the caller murmurs a quiet "mm-hmm," the evening news buzzes on a nearby television, or the caller suddenly remembers something and shouts, "Wait, wrong card!"
Real-time voice agents inhabit a messy world of background noise and spontaneous interruptions. When I set out to test these scenarios, I wanted to answer two practical questions: Which of these everyday sounds might trick an agent into issuing a duplicate refund? And second, what causes an agent to process a refund successfully, yet drop into complete silence without ever telling the caller it worked?
To find out, I built an automated refund desk on PolyAI's Dialog-RSN-1 API (pmckeetx/refund-voice-agent) and ran a scripted caller through 75 measured calls. Passive backchannels turned out to be completely harmless: a soft "mm-hmm" spoken over the agent never changed the outcome across 20 test runs. The real breakdown happened when the caller asked a natural follow-up question over the holding line: "Sorry, did that go through?"
Here is what that interaction looked like on the starting client implementation on 10 November 2026. This naive client suppresses follow-up agent responses once the caller speaks, and it mistakenly keys refunds on ephemeral tool calls rather than the underlying order (run as caller.py --before in the repository, producing R-4417-1):
(caller says refund.wav) Caller: Hi, I need a refund on order four thousand four hundred seventeen. The cast iron skillet arrived cracked. (issue_refund 4417 to 4242: refunded) (caller says did-it-go-through.wav) (barge-in: stopped the reply after 1.7 s) Agent: Let me process that refund- [cut off] Caller: Sorry, did that go through? Agent: It's still processing. I'll let you know as soon as it goes through. refund_id order amount card status told calls R-4417-1 4417 32.00 4242 pending no 1 The money was sent, but the call fell silent. Across 10 out of 10 test runs, the agent promised to confirm the refund as soon as it finished, but never spoke another word. In the real world, this customer wakes up tomorrow, assumes the request was lost, and calls back to demand a refund they have already received.
The bug is subtle. The language model answered the caller's immediate question before the payment gateway returned. When the tool result finally arrived, the client saw that the user had spoken in the interim, decided not to interrupt with an unprompted reply, and abandoned the confirmation entirely.
Solving this requires a client-side ledger that tracks both the state of the transaction and whether the caller has actually heard the confirmation. By keying refunds directly to the order ID, duplicate tool calls resolve to the existing transaction, and card switches update the pending record instead of adding a second charge. Whenever the gateway finishes and no subsequent response has started, the client explicitly prompts the model for confirmation, and automated tests verify that the ledger only marks a refund told when the customer has heard the outcome.
What you'll build
In this guide, you will build a terminal-based voice agent for a fictional store, Copperline Kitchen Supply. This agent enforces strict financial idempotency: it pays exactly one refund per order regardless of how many times the model triggers the tool. If the caller blurts out "wait, wrong card" while the transaction is still pending, the agent diverts the pending refund to the new card rather than creating a duplicate payout.
The agent also ensures every transaction is accompanied by an audible confirmation, even if the caller interrupts with a question mid-process or talks directly over the agent's first attempt to speak. The project includes a scripted caller that simulates acoustic environments and phone calls in real time, allowing you to observe edge cases without touching a physical microphone, along with a test suite that reconciles your SQLite ledger against the audio heard by the caller.
How it works
When a caller requests a refund on Dialog-RSN-1, the model emits an issue_refund call. The client receives the response.function_call_arguments.done event before the audible holding line begins playing; across 72 test runs, the spoken holding audio lagged the event by a median of 0.12 seconds. The client kicks off the background refund immediately upon receiving that event.
Even if the caller barges in and cancels the spoken holding phrase, the tool execution itself has already been dispatched. In 10 test runs where the caller interrupted the agent mid-sentence, the tool call remained safely recorded in the server's conversation history every time.
This means transaction integrity rests entirely on the client, where three failure points emerge. The model might trigger the refund tool more than once, which the ledger must recognize as the same refund. The caller might speak while the gateway is processing, so the client has to decide when to ask for a confirmation. And the confirmation itself might be cut off by an interruption, leaving a completed refund the caller never heard.
Under the hood, barge_in.py manages audio interruptions: it halts outbound audio as soon as the server detects incoming speech, truncates conversation history to what was actually played out loud, and logs cut-off replies. We will leave that module as-is to focus on the heart of the system: the ledger and turn coordinator.
Prerequisites
To follow along, you will need Python 3.13 and uv, which locks down dependencies to match this environment. You will also need an active Dialog-RSN-1 API key, which you can generate under the API Keys tab at studio.poly.ai.
Calls run $0.02 per connected minute plus tokens, new accounts receive launch credits, and sign-up tiers support two concurrent sessions (limits and pricing, verified 10 November 2026). Each automated test call in this walkthrough finishes in under a minute.
The scripted testing suite operates entirely without a physical microphone or speaker setup. However, if you would like to test the live voice agent interactively using agent.py, keep a microphone and a pair of headphones ready.
Set up the project
Start by cloning the repository and generating your local environment configuration:
git clone https://github.com/pmckeetx/refund-voice-agent.git cd refund-voice-agent cp .env.example .env Open .env in your editor and supply your API key:
# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab. POLY_API_KEY= # Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime. # DIALOG_REALTIME_URL= Leave DIALOG_REALTIME_URL commented out unless your key is for a different PolyAI host. Next, install project dependencies and execute the offline test suite, which runs without requiring network access or API credentials:
uv sync uv run pytest ... tests/test_agent.py ...... [ 20%] tests/test_audio.py .... [ 33%] tests/test_ledger.py ........ [ 60%] tests/test_live.py sss [ 70%] tests/test_refunds.py ......... [100%] ======================== 27 passed, 3 skipped in 22.07s ======================== The three skipped tests represent live calls. When enabled, they run scripted audio against the real-time API using your credentials, taking roughly 90 seconds:
POLYAI_LIVE=1 uv run pytest -m live -v ... tests/test_live.py::test_one_refund_and_the_caller_heard_it[mm-hmm] PASSED [ 33%] tests/test_live.py::test_one_refund_and_the_caller_heard_it[did-it-go-through] PASSED [ 66%] tests/test_live.py::test_one_refund_and_the_caller_heard_it[missed-confirmation] PASSED [100%] ================= 3 passed, 27 deselected in 84.89s (0:01:24) ================== Setting POLYAI_LIVE=1 activates live integration, while -m live limits test collection to those scenarios. If you encounter an HTTP 401 error, verify that an existing POLY_API_KEY in your shell environment is not overriding your .env value.
Step 1: Give every refund a key
Our ledger consists of a single SQLite table, and it uses order_id as its primary key, making it structurally impossible to insert multiple refunds against the same order.
Here is the schema defined in ledger.py:
SCHEMA = """ CREATE TABLE IF NOT EXISTS refunds ( order_id TEXT PRIMARY KEY, -- the key: what the refund is for, not who asked refund_id TEXT NOT NULL UNIQUE, amount_cents INTEGER NOT NULL, card_last4 TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'pending', -- pending until the payment provider settles it session_id TEXT NOT NULL, -- the call that asked first call_ids TEXT NOT NULL, -- every tool call that asked for it, as a JSON list accepted INTEGER NOT NULL DEFAULT 0, -- has the payment provider accepted it, to this card? told INTEGER NOT NULL DEFAULT 0 -- has the caller heard a reply that names its card? ) """ It is tempting to key transactions by the tool call's call_id because each call generates a unique identifier. However, that design is precisely why duplicate payouts happen: if the model decides to invoke the tool a second time, it generates a fresh call_id, and a naive insert creates a duplicate record. A refund is an action on an order, meaning the order must serve as the primary key. We still record session_id and all associated call_ids as an audit trail, while status, accepted, and told track provider state and user comprehension.
Warning: Don't key a side effect on
call_id. It identifies the request, not the thing being done, so a repeat of the same request looks new.
Writing to this ledger is handled by a single transaction method, Ledger.refund():
def refund(self, order_id: str, card_last4: str, session_id: str, call_id: str) -> tuple[str, dict]: """Record a refund for an order. Returns what happened, and the row as it now stands.""" amount = ORDERS[order_id]["amount_cents"] self.db.execute("BEGIN IMMEDIATE") # the check and the write happen under one lock try: added = self.db.execute( "INSERT INTO refunds (order_id, refund_id, amount_cents, card_last4, session_id, call_ids) " "VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT (order_id) DO NOTHING", (order_id, f"R-{order_id}", amount, card_last4, session_id, json.dumps([call_id])), ).rowcount if added: outcome = "refunded" else: row = self.row(order_id) self.db.execute("UPDATE refunds SET call_ids = ? WHERE order_id = ?", (json.dumps(json.loads(row["call_ids"]) + [call_id]), order_id)) if row["card_last4"] == card_last4: outcome = "already_refunded" elif row["status"] == "pending": self.db.execute("UPDATE refunds SET card_last4 = ?, accepted = 0, told = 0 WHERE order_id = ?", (card_last4, order_id)) outcome = "card_changed" else: outcome = "settled" # the money has gone: moving it is a new refund and a reversal, not an update self.db.execute("COMMIT") except BaseException: self.db.execute("ROLLBACK") raise return outcome, self.row(order_id) The write uses SQLite's ON CONFLICT (order_id) DO NOTHING, so the database itself decides whether this is the first refund for the order. Evaluating rowcount reveals whether a new row was added or an existing record caught the conflict. Relying on an in-memory dictionary or simple checks invites race conditions the moment two tasks execute concurrently. Wrapping the block in BEGIN IMMEDIATE guarantees that the conflict check, read, and subsequent update occur under the same write lock, and a test in tests/test_ledger.py races two connections at one order to prove it.
When an order row already exists, the method determines the appropriate business response. A repeat request on the same card is marked already_refunded. A switch to a different card while the status is still pending updates the target card in place and resets both accepted and told, because both the provider and the customer need to hear about the new card. If the refund has already cleared, the method returns settled, as modifying settled funds requires an explicit reversal and new transaction. Notice also that the refund amount is sourced directly from internal store records (ORDERS), never from model parameters.
Step 2: Answer a repeat call from the ledger
Because the language model only understands tool outcomes through returned values, our tool response must state the ledger's ground truth directly.
The tool definition is declared in agent.py:
TOOLS = [{ "type": "function", "name": "issue_refund", "description": "Refund an order in full to one of the caller's cards.", "parameters": { "type": "object", "properties": { "order_id": {"type": "string", "description": "The order number, like 4417."}, "card_last4": {"type": "string", "description": "The last four digits of the card to refund to."}, "say_while_working": { "type": "string", "poly_spoken": True, # the server speaks this argument while the tool runs "description": "A sentence to say while the refund goes through, like 'Let me process that refund for you now.'", }, }, "required": ["order_id", "card_last4", "say_while_working"], }, }] The say_while_working parameter is tagged with "poly_spoken": True, instructing the server to speak this phrase aloud while the tool runs in the background. We make this field required on purpose. As highlighted in the Dialog-RSN-1 overview, models tend to omit optional spoken arguments, which leaves callers waiting in awkward silence while the external payment gateway completes its work.
The execution handler validates the inputs, consults the ledger, and formats the response:
async def issue_refund(self, args: dict, call_id: str) -> dict: order_id, card = str(args.get("order_id", "")).strip(), str(args.get("card_last4", "")).strip() if order_id not in ORDERS: return {"status": "error", "note": f"There's no order {order_id} for this caller. Ask them to check the number."} if card not in CARDS: return {"status": "error", "note": f"The caller has no card ending {card}. Ask which card they want."} outcome, row = self.ledger.refund(order_id, card, self.session_id, call_id) self.log("", f"(issue_refund {order_id} to {card}: {outcome})") if not row["accepted"]: # new, a new card, or a repeat of one the provider never accepted await submit_refund(row) self.ledger.mark_accepted(order_id) return {"status": outcome, "order_id": order_id, "refund_id": row["refund_id"], "note": self.note_for(outcome, row)} Any transaction that has not yet been accepted (whether it is brand new, transferred to a replacement card, or retried after the provider failed the first time) is forwarded to the payment gateway through submit_refund(), mocked here as a four-second delay. Subsequent calls for an already accepted refund return immediately. In production, each request includes provider_key(row) as an idempotency key so the payment provider can reject duplicates on its end.
The model learns what to say next via note_for():
def note_for(self, outcome: str, row: dict) -> str: """What the model is told, worded from the ledger's row, so its next line says what actually happened.""" if outcome == "refunded": return f"Refunded {describe(row)}. Tell the caller it's done." if outcome == "already_refunded": when = "earlier in this call" if row["session_id"] == self.session_id else "on an earlier call" return f"This refund was already issued {when}: {describe(row)}. Tell the caller it's done; don't refund again." if outcome == "card_changed": return f"Still one refund, now {describe(row)} instead. Tell the caller." return f"This refund already settled, {describe(row)}, and can't be moved. Tell the caller a person will call back." The helper function describe(row) converts the database entry into natural phrases like "32 dollars to the Visa ending 4242," anchoring the agent's statements directly to the database state. If an order was previously refunded, the note indicates whether the event occurred in the current session or a previous call, providing clear context when a customer calls back.
Across 10 test calls where callers asked "Sorry, did that go through?", the model never triggered issue_refund a second time. However, when the caller interrupted with "Wait, wrong card. Can you put it on my Mastercard instead?", the model reliably invoked issue_refund with the new details in 5 out of 5 runs. While a naive client would have charged the customer twice, our ledger flagged the transition as card_changed, kept a single database row, and prompted the agent to confirm: "That's done. I've put the 32 dollars back to your Mastercard ending 8812."
Step 3: Confirm once the result is in
Now let us revisit the silence bug from our opening example.
In the initial implementation, the client skipped sending response.create after receiving a tool result if the caller had spoken in the meantime. The reasoning seemed sound: avoid talking over a caller who has already moved on to a new question. While that works for read-only database queries, it is disastrous for mutations. When the caller asked "did that go through?", the model answered "It's still processing." When the gateway finally finished, the client saw that the user had spoken, suppressed the follow-up response, and never closed the conversational loop.
To fix this, the client must trigger a reply whenever no new turn has started since the tool result entered history. The event handler tracks response counters, records which responses contained tool calls, and logs mapping identifiers:
async def handle(self, event: dict): match event["type"]: case "response.created": self.responses += 1 ... case "response.function_call_arguments.done": self.with_calls.add(self.responses) self.spawn(self.run_tool(event)) # the call is final: start the refund before the holding line plays await super().handle(event) The tool executes inside a dedicated task started as soon as argument streaming completes. When finished, run_tool() appends the outcome to the history and coordinates the response:
async def run_tool(self, call: dict): try: result = await self.issue_refund(json.loads(call["arguments"]), call["call_id"]) except Exception as error: # every call gets an output, or the model waits for it forever self.log("", f"(issue_refund failed: {error})") result = {"status": "error", "note": "The refund system failed, so the refund isn't confirmed. Don't call " "issue_refund again. Tell the caller the team will email them once it's sorted."} since = self.responses await self.send({ "type": "conversation.item.create", "item": {"type": "function_call_output", "call_id": call["call_id"], "output": json.dumps(result)}, }) if result.get("refund_id"): self.owed.setdefault(result["order_id"], since) # from the first result the caller hasn't heard self.reminded.pop(result["order_id"], None) await self.ask_for_reply(since) async def ask_for_reply(self, since: int): """Ask the model to speak about the result, unless a reply has already started since it went into history.""" while self.responding: # one response at a time: let the holding line, or the answer to the caller, finish await asyncio.sleep(0.05) if self.responses == since: await self.send({"type": "response.create"}) # sent while the caller talks, it's held for their turn A tool output event alone does not trigger speech. Appending function_call_output merely updates the conversation log; an explicit response.create event is required to prompt an agent response.
The since variable captures the response count at the exact moment the output was recorded, and owed preserves this marker until the caller hears the result. The ask_for_reply() method waits for any active turn to conclude (such as the holding line or the answer to an interruption), avoiding conversation_already_has_active_response errors. If no newer turn has begun, the client issues response.create. If the caller is speaking when that event arrives, the API reference says the server holds it and answers their turn with a single reply, which is what happened in all 5 mm-hmm-gap runs on 10 November.
Running the original problem scenario through the updated client demonstrates the fix:
... Caller: Sorry, did that go through? Agent: It's still processing. I'll let you know as soon as it goes through. Agent: Yes, it's done. I've refunded 32 dollars back to your Visa ending 4242. refund_id order amount card status told calls R-4417 4417 32.00 4242 pending yes 1 The caller heard the confirmation in 5 out of 5 runs, compared to 0 out of 10 on the older client.
Handling exceptions inside run_tool() protects this flow. Every tool invocation receives an explicit payload so the model is never left waiting indefinitely. In a run on 10 November with the payment provider down, the caller was promptly told: "The refund system failed, so it's not confirmed yet. Our team will email you once it's sorted." The row remained stored with accepted and told set to zero, an actionable record for support staff, and if the caller asks again, step 2's check submits it again.
Step 4: Tell the caller what the ledger says
Just because the server sent an audio stream does not mean the user actually heard it. An interruption can cut a sentence off halfway through.
To bridge this gap, the client sets the ledger's told flag only when a reply the caller actually heard names the refunded card:
async def heard(self, item_id: str, text: str, whole: bool): await super().heard(item_id, text, whole) for order_id, since in list(self.owed.items()): n = self.response_of.get(item_id, 0) if n <= since or n in self.with_calls: continue # a holding line, or a reply that started before the result was in history row = self.ledger.row(order_id) if mentions(text, row["card_last4"]): self.ledger.mark_told(order_id) del self.owed[order_id] elif self.reminded.get(order_id, 0) < 2: # twice at most, so a model that won't say it can't loop self.spawn(self.remind(row)) The heard() hook fires after every spoken reply, receiving the exact text delivered to the caller: either the full sentence or the truncated fragment captured when an interruption occurred. Holding lines and earlier responses are filtered out, and mentions() confirms the specific last four card digits made it across the wire.
If an interruption truncates the confirmation, remind() puts the ledger's word into history and asks for a reply:
async def remind(self, row: dict): """The refund is real, but the caller didn't hear it. Put the ledger's word in history and ask for a reply.""" self.reminded[row["order_id"]] = self.reminded.get(row["order_id"], 0) + 1 note = (f"Refund {row['refund_id']} is done: {describe(row)}. The caller hasn't heard that yet. In your next " "reply, tell them it's done, with the amount and the card's last four digits.") self.log("", Source：https://dev.to/pmckeetx/stop-your-voice-agent-going-quiet-after-it-refunds-a-caller-1c1e Related posts
Building a Real-Time GPS Proximity Radar & Mobile Chat PWA with $0/Month Map API Costs
Show and tell: I open-sourced Ambient Memory, a local-first context layer for Cursor and Claude Desktop
AI Daily Digest November 13, 2026: Gemini 4 Hydrogen Goes Default, Google's Gemini Glasses Open Pre-orders, Codex Security Goes Open Source
Start and end a voice agent call in Python without clipping a word
I Forked an Open-Source Mac Utility. Here's What I Changed, What I Kept, and When You Should Use the Original
