Building a .NET Dev Agent, Part 2: Give It a Voice
The first time my dev agent answered out loud, it took sixty-eight seconds to tell me that four tests had failed.
It was correct, and it was thorough: four test names read out in full, AddedLines_reports_new_file_numbers_across_hunks included, each with its line number and its expected value, and a closing paragraph offering to look at the parser. On screen that answer is a glance. Through headphones it is a minute of standing still, waiting to hear the one sentence that mattered, which came last: every failure was off by one.
Wiring speech-to-text and text-to-speech onto an agent takes an afternoon. This post is about what happens after that afternoon, when you discover that an answer written to be read is the wrong shape to be heard, and that the transcript your agent receives is wrong roughly every other word. Both problems turn out to be design problems, not audio problems, and everything below is measured on the agent from Part 1.
Where Part 1 left the agent
Building a .NET Dev Agent, Part 1: Teach It Your Actual Workflow built three typed tools on top of a Claude tool_use loop: git_status, dotnet_build and dotnet_test, each one filtering its command's output down to what a model needs. Those three tools come over unchanged, with one exception: Part 1's RunAsync helper had a bug that only shows up on a machine with two .NET SDKs, and building this post found it. Part 1 now carries the fix; the details are near the end of this post.
Here is the whole loop:
- Hold a key, talk, release.
- A local Whisper turns the audio into text.
- The agent runs, with Part 1's tools plus one new one.
- A local Piper voice reads the reply back, and anything that should not be read goes to the screen.
Everything that touches audio runs locally, from open-source models, with no account needed for anything but Claude itself.
Two containers
Speech-to-text is whisper.cpp's whisper-server, which takes a WAV file over HTTP and returns JSON. Text-to-speech is Piper, whose HTTP server takes JSON and returns a WAV file. Both run on CPU.
services: whisper: # v1.9.5. The image is tagged by commit; ":main" moves every day. image: ghcr.io/ggml-org/whisper.cpp:main-d1be6fde11ac6e0407606b4e42fe72d34add8037 # The image's entrypoint is `bash -c`, which runs only its FIRST argument. A plain # string gets split into words, so every flag after "whisper-server" is silently # dropped and it starts on the image's bundled model, bound to 127.0.0.1. # A one-element list keeps the whole command line as that first argument. command: ["whisper-server --host 0.0.0.0 --port 8080 --model /models/ggml-${WHISPER_MODEL:-tiny.en}.bin --threads 4"] volumes: - ./models:/models:ro ports: - "127.0.0.1:8080:8080" piper: build: ./piper ports: - "127.0.0.1:5000:5000" Piper has no official image with its HTTP server, so it gets a five-line piper/Dockerfile. The voice is downloaded at build time, so the container starts without the network:
FROM python:3.12-slim RUN pip install --no-cache-dir "piper-tts[http]==1.8.0" WORKDIR /voices # Bake the voice into the image so the container starts offline. RUN python -m piper.download_voices en_US-lessac-medium --data-dir /voices EXPOSE 5000 CMD ["python", "-m", "piper.http_server", "--host", "0.0.0.0", "--port", "5000", "--data-dir", "/voices", "-m", "en_US-lessac-medium"] The Whisper model is a file you download once into models/:
curl -L -o models/ggml-tiny.en.bin https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin docker compose up -d The comment on command: is there because I lost an hour to it. whisper.cpp's image uses bash -c as its entrypoint, and bash -c only runs its first argument. Written as a plain string, Compose splits the command into words, so bash -c receives whisper-server as its script and every flag after it as an ignored positional parameter: the server starts on the model bundled inside the image, bound to 127.0.0.1, where nothing outside the container can reach it. Written as a folded multi-line string it fails more quietly. The line break ends the command, so the server listens where you asked but loads the bundled model. My first run asked for large-v3-turbo, got base.en, and I spent a while blaming my accent. The one-element list keeps the whole command line together.
Pin the image by commit, too. :main is rebuilt daily, and the tag above is the one built from the v1.9.5 release.
Capture: push-to-talk, at the format Whisper wants
The .NET side needs one native dependency for audio. PortAudioSharp2 ships PortAudio's binaries for Windows, Linux and macOS inside the package, so there is nothing to install:
dotnet new console -n VoiceDevAgent dotnet add package Anthropic --version 12.57.0 dotnet add package PortAudioSharp2 --version 1.0.6 Recording happens at 16 kHz, mono, 16-bit, which is exactly what Whisper consumes, so nothing gets resampled on either side. PortAudio delivers audio on its own thread through a callback, and that callback has one job: copy the samples somewhere and return, without ever waiting on the rest of the program. A Channel<T> is the right somewhere, because writing to an unbounded one never blocks.
using System.Buffers.Binary; using System.Runtime.InteropServices; using System.Threading.Channels; using PortAudioSharp; using PaStream = PortAudioSharp.Stream; /// <summary>Push-to-talk capture: 16 kHz, mono, 16-bit — the format Whisper wants, so nothing resamples.</summary> sealed class Microphone { public const int SampleRate = 16_000; public static async Task<byte[]> RecordAsync(Func<CancellationToken, Task> waitForStop, CancellationToken ct) { // The callback runs on PortAudio's audio thread. It must never wait on anything the // rest of the program does, so it only copies samples into an unbounded channel, // whose writes never block. Channel<short[]> chunks = Channel.CreateUnbounded<short[]>(new() { SingleReader = true, SingleWriter = true }); PaStream.Callback callback = (IntPtr input, IntPtr output, uint frameCount, ref StreamCallbackTimeInfo timeInfo, StreamCallbackFlags flags, IntPtr userData) => { short[] chunk = new short[frameCount]; Marshal.Copy(input, chunk, 0, (int)frameCount); chunks.Writer.TryWrite(chunk); return StreamCallbackResult.Continue; }; int device = PortAudio.DefaultInputDevice; if (device == PortAudio.NoDevice) throw new InvalidOperationException("No input device found."); StreamParameters parameters = new() { device = device, channelCount = 1, sampleFormat = SampleFormat.Int16, suggestedLatency = PortAudio.GetDeviceInfo(device).defaultLowInputLatency, hostApiSpecificStreamInfo = IntPtr.Zero, }; using (PaStream stream = new(parameters, null, SampleRate, 0, StreamFlags.ClipOff, callback, null!)) { stream.Start(); try { await waitForStop(ct); } finally { stream.Stop(); // blocks until the last callback has returned chunks.Writer.Complete(); } } List<short> samples = []; await foreach (short[] chunk in chunks.Reader.ReadAllAsync(CancellationToken.None)) samples.AddRange(chunk); return Wav.Encode([.. samples], SampleRate); } } /// <summary>Plays 16-bit mono PCM, and stops the moment the token is cancelled.</summary> static class Speaker { static readonly short[] Silence = new short[4096]; public static async Task PlayAsync(byte[] wav, CancellationToken ct) { (short[] samples, int sampleRate) = Wav.Decode(wav); int position = 0; TaskCompletionSource finished = new(TaskCreationOptions.RunContinuationsAsynchronously); PaStream.Callback callback = (IntPtr input, IntPtr output, uint frameCount, ref StreamCallbackTimeInfo timeInfo, StreamCallbackFlags flags, IntPtr userData) => { int remaining = samples.Length - position; int count = Math.Min(remaining, (int)frameCount); if (count > 0) Marshal.Copy(samples, position, output, count); // Pad the final buffer with silence instead of replaying stale memory, // without allocating on the audio thread. for (int padded = count; padded < frameCount; ) { int n = Math.Min(Silence.Length, (int)frameCount - padded); Marshal.Copy(Silence, 0, output + padded * sizeof(short), n); padded += n; } position += count; return position >= samples.Length ? StreamCallbackResult.Complete : StreamCallbackResult.Continue; }; int device = PortAudio.DefaultOutputDevice; if (device == PortAudio.NoDevice) throw new InvalidOperationException("No output device found."); StreamParameters parameters = new() { device = device, channelCount = 1, sampleFormat = SampleFormat.Int16, suggestedLatency = PortAudio.GetDeviceInfo(device).defaultHighOutputLatency, hostApiSpecificStreamInfo = IntPtr.Zero, }; using PaStream stream = new(null, parameters, sampleRate, 0, StreamFlags.ClipOff, callback, null!); stream.SetFinishedCallback(_ => finished.TrySetResult()); stream.Start(); try { await finished.Task.WaitAsync(ct); } catch (OperationCanceledException) { // Barge-in: drop the buffered audio instead of draining it. The finished // callback still fires after this; completing an already-abandoned task is harmless. stream.Abort(); } } } static class Wav { public static byte[] Encode(short[] samples, int sampleRate) { byte[] wav = new byte[44 + samples.Length * 2]; Span<byte> h = wav; "RIFF"u8.CopyTo(h); BinaryPrimitives.WriteInt32LittleEndian(h[4..], wav.Length - 8); "WAVEfmt "u8.CopyTo(h[8..]); BinaryPrimitives.WriteInt32LittleEndian(h[16..], 16); // fmt chunk size BinaryPrimitives.WriteInt16LittleEndian(h[20..], 1); // PCM BinaryPrimitives.WriteInt16LittleEndian(h[22..], 1); // mono BinaryPrimitives.WriteInt32LittleEndian(h[24..], sampleRate); BinaryPrimitives.WriteInt32LittleEndian(h[28..], sampleRate * 2); // byte rate BinaryPrimitives.WriteInt16LittleEndian(h[32..], 2); // block align BinaryPrimitives.WriteInt16LittleEndian(h[34..], 16); // bits per sample "data"u8.CopyTo(h[36..]); BinaryPrimitives.WriteInt32LittleEndian(h[40..], samples.Length * 2); MemoryMarshal.AsBytes(samples.AsSpan()).CopyTo(h[44..]); return wav; } /// <summary>Walks the chunks instead of assuming a 44-byte header: Piper's is, other writers' aren't.</summary> public static (short[] Samples, int SampleRate) Decode(byte[] wav) { ReadOnlySpan<byte> s = wav; if (!s[..4].SequenceEqual("RIFF"u8) || !s[8..12].SequenceEqual("WAVE"u8)) throw new InvalidDataException("Not a RIFF/WAVE file."); int sampleRate = 0, offset = 12; while (offset + 8 <= s.Length) { ReadOnlySpan<byte> id = s.Slice(offset, 4); // Streaming writers put 0xFFFFFFFF here ("unknown"), which reads as -1: take the rest. uint declared = BinaryPrimitives.ReadUInt32LittleEndian(s[(offset + 4)..]); int size = (int)Math.Min(declared, (uint)(s.Length - offset - 8)); ReadOnlySpan<byte> body = s.Slice(offset + 8, size); if (id.SequenceEqual("fmt "u8)) { if (body.Length < 16) throw new InvalidDataException("fmt chunk too short."); short channels = BinaryPrimitives.ReadInt16LittleEndian(body[2..]); short bits = BinaryPrimitives.ReadInt16LittleEndian(body[14..]); if (channels != 1 || bits != 16) throw new InvalidDataException($"Expected 16-bit mono, got {bits}-bit x{channels}."); sampleRate = BinaryPrimitives.ReadInt32LittleEndian(body[4..]); } else if (id.SequenceEqual("data"u8)) { if (sampleRate == 0) throw new InvalidDataException("data chunk before fmt chunk."); return (MemoryMarshal.Cast<byte, short>(body).ToArray(), sampleRate); } offset += 8 + size + (size & 1); // chunks are word-aligned } throw new InvalidDataException("No data chunk."); } public static TimeSpan Duration(byte[] wav) { (short[] samples, int rate) = Decode(wav); return TimeSpan.FromSeconds((double)samples.Length / rate); } } Three details in there earn a sentence. stream.Stop() blocks until the last callback has returned, so the channel can be completed after it without losing the final buffer. The capture callback still allocates a small array per buffer. .NET makes no real-time guarantees either way; if you need them, record into a preallocated ring buffer instead. Playback pads its last buffer from a preallocated block of silence. And there is no GC.KeepAlive on the callbacks because PortAudioSharp2 pins each delegate with a GCHandle for as long as the stream lives. If you bind PortAudio yourself, that pinning is your job, or the next callback jumps into freed memory.
Speaker.PlayAsync takes a cancellation token for barge-in: when you have heard enough, Enter stops the audio with Abort(), which drops what is buffered instead of draining it.
Why push-to-talk, and not always listening
An always-on microphone looks friendlier and is worse in three measurable ways for this tool:
- It hears the agent. On laptop speakers, the reply goes straight back into the microphone and arrives as a new request.
- Whisper is unreliable at the edges of speech. On my eight recorded sentences,
tiny.enreturned[BLANK_AUDIO]for two that were perfectly audible,base.enturned one of them, a question aboutCS1061, into "Peace. See you in the next video. Bye." To be fair to Whisper, it was trained on a lot of YouTube. An always-on microphone feeds it far more edges than a key press does. - You lose the edges. With push-to-talk, the press says now I'm talking to you, and the release says I'm done. With always-on you need voice-activity detection to guess both, and every wrong guess is a turn.
So the harness filters Whisper's silence markers before the model ever sees them, which you will see in VoiceAgent.IsBlank below. An accidental key press costs nothing and runs nothing.
Speech-to-text: wrong roughly every other word
The HTTP clients are short:
using System.Net.Http.Headers; using System.Net.Http.Json; using System.Text.Json.Serialization; /// <summary>whisper.cpp's whisper-server: POST /inference, multipart, returns {"text": "..."}.</summary> sealed class WhisperClient(HttpClient http) { public async Task<string> TranscribeAsync(byte[] wav, string? vocabulary, string? language, CancellationToken ct) { using MultipartFormDataContent form = new(); ByteArrayContent file = new(wav); file.Headers.ContentType = new MediaTypeHeaderValue("audio/wav"); form.Add(file, "file", "speech.wav"); form.Add(new StringContent("json"), "response_format"); form.Add(new StringContent("0.0"), "temperature"); if (vocabulary is not null) form.Add(new StringContent(vocabulary), "prompt"); if (language is not null) form.Add(new StringContent(language), "language"); using HttpResponseMessage response = await http.PostAsync("/inference", form, ct); response.EnsureSuccessStatusCode(); WhisperResult result = await response.Content.ReadFromJsonAsync<WhisperResult>(ct) ?? throw new InvalidDataException("Empty response from whisper-server."); return result.Text.Trim(); } sealed record WhisperResult([property: JsonPropertyName("text")] string Text); } /// <summary>Piper's HTTP server: POST /synthesize with {"text": "..."}, returns a WAV.</summary> sealed class PiperClient(HttpClient http) { Source：https://dev.to/jgomezdev/building-a-net-dev-agent-part-2-give-it-a-voice-1ef3Related posts
03/04/2026: Why Date Formats Put Calendar Events in the Wrong Week
Export WooCommerce Products to CSV in 2026: Real Data From 4 Stores, No Plugin
AI models now find, exploit and patch bugs: what Gemini 4 Carbon means for defenders
Goldman Walls Off AI for Its Own Dealers
Which UK Bookshop Is Supplying Sally Rooney Novels After Faber Stopped Distribution?
