A real-time voice companion can behave correctly while its test suite reports failure.
The usual culprit is not that “the LLM knows better.” It is that the test encoded an assumption the product never promised: an exact sentence, a fixed callback order, or playback beginning within an arbitrary sleep interval.
That creates an uncomfortable engineering tension. You do not want to dismiss failures as harmless AI variability, but you also do not want brittle tests blocking valid behavior.
A better approach is to test the conversation protocol deterministically and evaluate the conversation content separately. For a Tencent RTC conversational AI application, that means verifying facts such as:
- User speech invalidates the previous turn.
- An interrupted response cannot resume later.
- Stale ASR, LLM, or playback callbacks cannot take control of the session.
- A timeout produces visible recovery instead of indefinite silence.
- Every asynchronous result remains correlated with its originating turn and request.
We will build that test oracle in TypeScript, reproduce the dangerous races, and then map it onto the live voice pipeline.
Start with the disputed failure
Consider this test:
expect(answer).toBe("Your meeting begins at 3 PM."); The model returns:
The meeting starts at 3 PM. The assertion failed, but that alone does not establish a product defect. The test mixed three different questions:
- Protocol: Did the response belong to the current user turn?
- Policy: Was the assistant permitted to answer?
- Content: Was the answer sufficiently correct and useful?
Only the last question involves linguistic variability. The first two should remain deterministic.
The human decision is therefore not “trust the model or trust the test.” It is: which product contract was this test supposed to enforce?
Draw the asynchronous boundary
A production voice companion is not one AI call. Keep these responsibilities separate:
Microphone / RTC media ↓ Speech recognition ↓ Application turn coordinator ↓ LLM ↓ Speech synthesis ↓ RTC playback The application coordinator owns the current turn. Neither a late model response nor a delayed playback callback gets to decide that it is still relevant.
Tencent RTC documents Conversational AI as a real-time voice interaction scenario that can connect users with multiple LLM providers. Its LLM configuration guidance also covers OpenAI-compatible models and request identifiers used for routing and observability:
The event and effect names below belong to our application. They are not Tencent RTC SDK method names.
Create the reproducible project
mkdir voice-protocol-oracle cd voice-protocol-oracle npm init -y npm install --save-dev typescript vitest tsx @types/node npm pkg set type=module npm pkg set scripts.test="vitest run" mkdir src Create src/session.ts:
export type Phase = | "idle" | "userSpeaking" | "transcribing" | "thinking" | "preparingAudio" | "speaking" | "recovery" | "offline"; export type State = { phase: Phase; turn: number; requestId?: string; }; export type Event = | { type: "USER_STARTED" } | { type: "USER_ENDED" } | { type: "TRANSCRIPT_READY"; turn: number; requestId: string; text: string; } | { type: "TRANSCRIPT_FAILED"; turn: number } | { type: "MODEL_DONE"; turn: number; requestId: string; text: string; } | { type: "MODEL_TIMEOUT"; turn: number; requestId: string } | { type: "PLAYBACK_STARTED"; turn: number; requestId: string } | { type: "PLAYBACK_ENDED"; turn: number; requestId: string } | { type: "PLAYBACK_FAILED"; turn: number; requestId: string } | { type: "DISCONNECTED" }; export type Effect = | { type: "REQUEST_TRANSCRIPT"; turn: number } | { type: "REQUEST_MODEL"; turn: number; requestId: string; text: string; } | { type: "SPEAK"; turn: number; requestId: string; text: string; } | { type: "CANCEL_AUDIO"; turn: number; requestId: string } | { type: "NOTICE"; message: string } | { type: "LOG_STALE"; source: string; turn: number; requestId?: string }; export const initialState: State = { phase: "idle", turn: 0, }; function isCurrent( state: State, turn: number, requestId?: string, ): boolean { return ( state.turn === turn && (requestId === undefined || state.requestId === requestId) ); } function stale( state: State, source: string, turn: number, requestId?: string, ): [State, Effect[]] { return [ state, [{ type: "LOG_STALE", source, turn, requestId }], ]; } export function reduce( state: State, event: Event, ): [State, Effect[]] { switch (event.type) { case "USER_STARTED": { if (state.phase === "userSpeaking") return [state, []]; const effects: Effect[] = []; if ( (state.phase === "preparingAudio" || state.phase === "speaking") && state.requestId ) { effects.push({ type: "CANCEL_AUDIO", turn: state.turn, requestId: state.requestId, }); } return [ { phase: "userSpeaking", turn: state.turn + 1, }, effects, ]; } case "USER_ENDED": { if (state.phase !== "userSpeaking") return [state, []]; return [ { ...state, phase: "transcribing" }, [{ type: "REQUEST_TRANSCRIPT", turn: state.turn }], ]; } case "TRANSCRIPT_READY": { if ( state.phase !== "transcribing" || !isCurrent(state, event.turn) ) { return stale(state, "transcript", event.turn, event.requestId); } if (!event.text.trim()) { return [ { ...state, phase: "recovery" }, [{ type: "NOTICE", message: "I did not catch that. Please try again." }], ]; } return [ { ...state, phase: "thinking", requestId: event.requestId, }, [ { type: "REQUEST_MODEL", turn: event.turn, requestId: event.requestId, text: event.text, }, ], ]; } case "TRANSCRIPT_FAILED": { if (state.phase !== "transcribing" || !isCurrent(state, event.turn)) { return stale(state, "transcript-error", event.turn); } return [ { ...state, phase: "recovery" }, [{ type: "NOTICE", message: "Speech recognition is unavailable. Try again." }], ]; } case "MODEL_DONE": { if ( state.phase !== "thinking" || !isCurrent(state, event.turn, event.requestId) ) { return stale(state, "model", event.turn, event.requestId); } if (!event.text.trim()) { return [ { ...state, phase: "recovery" }, [{ type: "NOTICE", message: "No answer was produced. Please retry." }], ]; } return [ { ...state, phase: "preparingAudio" }, [ { type: "SPEAK", turn: event.turn, requestId: event.requestId, text: event.text, }, ], ]; } case "MODEL_TIMEOUT": { if ( state.phase !== "thinking" || !isCurrent(state, event.turn, event.requestId) ) { return stale(state, "model-timeout", event.turn, event.requestId); } return [ { ...state, phase: "recovery" }, [{ type: "NOTICE", message: "The response took too long. Please try again." }], ]; } case "PLAYBACK_STARTED": { if ( state.phase === "preparingAudio" && isCurrent(state, event.turn, event.requestId) ) { return [{ ...state, phase: "speaking" }, []]; } // Playback may have crossed an interruption or cancellation boundary. // Stop it instead of accepting its late callback. return [ state, [ { type: "CANCEL_AUDIO", turn: event.turn, requestId: event.requestId, }, { type: "LOG_STALE", source: "playback-start", turn: event.turn, requestId: event.requestId, }, ], ]; } case "PLAYBACK_ENDED": { if ( state.phase !== "speaking" || !isCurrent(state, event.turn, event.requestId) ) { return stale(state, "playback-end", event.turn, event.requestId); } return [{ phase: "idle", turn: state.turn }, []]; } case "PLAYBACK_FAILED": { if ( !["preparingAudio", "speaking"].includes(state.phase) || !isCurrent(state, event.turn, event.requestId) ) { return stale(state, "playback-error", event.turn, event.requestId); } return [ { ...state, phase: "recovery" }, [{ type: "NOTICE", message: "Audio playback failed. A text answer can be shown." }], ]; } case "DISCONNECTED": { const effects: Effect[] = []; if ( (state.phase === "preparingAudio" || state.phase === "speaking") && state.requestId ) { effects.push({ type: "CANCEL_AUDIO", turn: state.turn, requestId: state.requestId, }); } effects.push({ type: "NOTICE", message: "Connection lost. Reconnect to continue." }); return [{ ...state, phase: "offline" }, effects]; } } } There are two important design choices here.
First, turn changes as soon as new user speech begins. That immediately invalidates work from the previous turn.
Second, a playback callback is not automatically trusted. If audio starts after its turn was interrupted, the coordinator issues another cancellation rather than moving back into speaking.
Make the races executable
Create src/session.test.ts:
import { describe, expect, it } from "vitest"; import { type Effect, type Event, initialState, reduce, } from "./session.js"; function scenario() { let state = initialState; return { step(event: Event): Effect[] { const [next, effects] = reduce(state, event); state = next; return effects; }, state() { return state; }, }; } describe("voice turn protocol", () => { it("rejects an LLM result from an interrupted turn", () => { const s = scenario(); s.step({ type: "USER_STARTED" }); s.step({ type: "USER_ENDED" }); s.step({ type: "TRANSCRIPT_READY", turn: 1, requestId: "req-1", text: "Tell me a story", }); // The user begins a replacement turn before req-1 completes. s.step({ type: "USER_STARTED" }); s.step({ type: "USER_ENDED" }); s.step({ type: "TRANSCRIPT_READY", turn: 2, requestId: "req-2", text: "Actually, stop", }); const effects = s.step({ type: "MODEL_DONE", turn: 1, requestId: "req-1", text: "Once upon a time...", }); expect(effects.some((effect) => effect.type === "SPEAK")).toBe(false); expect(s.state()).toMatchObject({ phase: "thinking", turn: 2, requestId: "req-2", }); }); it("cancels playback that starts after a barge-in", () => { const s = scenario(); s.step({ type: "USER_STARTED" }); s.step({ type: "USER_ENDED" }); s.stepSource：https://dev.to/susiewang/stop-snapshot-testing-voice-conversations-verify-the-turn-protocol-instead-109j Related posts
Shutting the Door on the XCPE Botnet (Threat Intelligence)
Canva Brings Free All-in-One Affinity App to iPad
GPT-Rosalind-6 System Card Addendum
Mediroute: An open-source AI platform exposing global drug price disparities
An intuition for how neural nets can understand text
