How I test local TTS on a Mac before it reads a book

A local TTS model can read a paragraph well and still ruin a chapter. A name gets dropped, a number changes, or the voice shifts after a break. Sample clips will not show that. I gate a model with one fixed 10,000-word manuscript, ten 1,000-word sections, and an audio audit before I let it narrate anything long.

I am publishing the method, not a leaderboard, because results depend on checkpoint, quantization, and memory pressure. A ranking from my Mac would tell you little about yours. This protocol gives you a reproducible pass or fail.

Freeze the manuscript

Take a finished 10,000-word chapter or short story and stop editing it. Include the awkward parts: character names, place names, dates, decimals, roman numerals, abbreviations, and initialisms. If you edit between runs, a mismatch could come from the text or the engine.

Split the file into ten UTF-8 text files of 1,000 words each. Keep the same section order for every model. Store a manifest with the input path and output path for each section. That manifest becomes the test fixture.

A 10,000-word manuscript is roughly 60,000 characters, or about 60 to 80 minutes of speech at 125 to 165 words per minute. That is long enough to expose drift and short enough to repeat.

Use one session, one setup

Generate all ten sections with the same voice, speed, model, and output format. Do not mix M4A and WAV in one comparison, since the formats use different encoders. If a tool has a batch mode, use it, but keep one file per section so you can replace failures individually.

Before the run, quit other memory-heavy applications. Note the macOS version, the Mac model, the engine build, the checkpoint, and any quantization setting. Memory pressure matters here. A model that behaves at the start of a session can behave differently after the machine starts swapping.

Murmur is a native macOS app for local text to speech. It bundles Kokoro, an 82-million-parameter open-weight model under Apache 2.0, and the Kokoro model card has the checkpoint details. I keep the longer setup notes in this writeup of the 10,000-word endurance test. The procedure below still applies to any engine that exports WAV or M4A.

This run needs Pro, since the free plan caps generations at 1,500 characters; Murmur also requires Apple Silicon and macOS 15 or newer.

Audit what the audio says

Transcribe each exported section with a local ASR model. Normalize the source and the transcript the same way: lowercase, strip punctuation, collapse whitespace, then tokenize. Number handling is the fiddly part. "1984" may come back as "nineteen eighty-four" or "one nine eight four." Pick a normalization rule, write it down, and apply it to both sides. Do not let the ASR decide the winner.

Align the token streams. Python's difflib.SequenceMatcher on token lists will show inserts and deletes, though a dedicated alignment library is better for long files. Source tokens with no audio match are omissions. Transcript tokens with no source match are candidate insertions.

Do not count every ASR difference as a failure. Play the audio around each flagged location and decide whether the engine omitted a word, invented a word, or the ASR misheard a name. Correct transcript errors in a copy, not in the raw transcript. Keep both so someone else can inspect your call.

The manual pass is the slow part, and it is what turns a word count into a decision.

Repair only the failed clips

Regenerate a section when it misses script coverage or adds words. Change one variable at a time: voice, speed, model, or checkpoint. If a clip passes, leave it alone. Replacing everything after a failure makes it impossible to know which change fixed or broke the run.

Keep a repair log: section, failure type, setting changed, result. It shows where the engine is fragile better than one score.

Listen for voice drift

Make a calibration passage, about 50 to 100 words, separate from the manuscript. Generate it at the start of the session, halfway through, and at the end. Use the same voice and speed every time. Listen to the three files back to back.

You are listening for changes in timbre, pace, and pronunciation that do not come from the text. A model can sound fine in isolation and still sound like a different take later in the run. If the calibration passage drifts, the manuscript has drift too, even when every clip passes the word check.

Test recovery after a restart

Do not stop at the last clip. Quit the app in the middle of the run, reopen it, and confirm that the project, generated sections, and approved clips are still there. Generate one section after the reopen. Then export the project and open the exported file in a separate player or editor. Check the start and end of each section for truncation.

Project state lives on the machine. A clean generation log means little if a restart loses an approved take or leaves a half-written file.

Pass thresholds

These are gates, not scores. The run stops at the first failed gate. Fix the failure, rerun that check, then continue.

Compute words per minute per section from duration and word count. This snippet uses ffprobe and the UTF-8 section files:

from pathlib import Path import re, subprocess def wpm(text_path, audio_path): text = Path(text_path).read_text(encoding="utf-8") words = len(re.findall(r"\b[\w'-]+\b", text)) seconds = float(subprocess.check_output([ "ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "default=noprint_wrappers=1:nokey=1", audio_path ])) return words / (seconds / 60) for i in range(1, 11): print(i, round(wpm(f"scripts/section-{i:02}.txt", f"audio/section-{i:02}.wav"), 1)) 

Run it after export. Dialogue is faster than description, but the fixed manuscript should make that repeatable across runs. A jump that appears once is instability, not style.

What this does not measure

No protocol this size can judge prosody, emotion, or casting. It measures whether an engine can finish a book without losing text, adding text, drifting, or losing approved work. That is the floor, not the ceiling.

Murmur's limits are real. It has no cloud voices or team admin UI, and the full feature set is Mac-only. Local models need disk space and memory, and cloning needs 16 GB or more. Its cloning quality is below ElevenLabs. If you need best-in-class cloning, expressive voices, a huge multilingual catalogue, browser access on any operating system, dubbing, or mature API and team features, ElevenLabs is stronger. It is also the right call when someone else should run the infrastructure.

Murmur is built for scripts that must stay on the Mac: after the models are downloaded it works offline. It is $49 one-time with a 7-day refund, and the free plan needs no account.

Disclosure: I build Murmur. If you want the app used in this test, it is at Murmur. Copy the method even if you use another engine. A reproducible gate beats a demo clip.