My TTS read a whole novel. The model was the most reliable part of the pipeline

In the first post of this series I mentioned, in a single table, that a synthesized voice had read an entire novel in Crimean Tatar: Merdiven ("Staircase") by Şamil Alâdin, sixteen chapters, 1 hour 58 minutes of audio. In the second one I promised the story behind that table. Here it is.

The short version, if you build anything like this: the model held up. The problems weren't in the model but in the text and audio preparation around it, which I'd written myself and could fix without retraining: splitting text into sentences, gluing short ones together, trimming breaths, pulling the book out of its file, checking the result. And a good share of those bugs simply cannot appear in a demo. They need a whole chapter, or a whole book, before they show up.

Thirty seconds of context

Crimean Tatar has a sound, къ (a /q/), that ready-made models from neighbouring languages flatten into a plain k, so qara ("black") comes out as kara. I picked a multilingual base that had heard a lot of Kazakh, Uyghur and Bashkir, and I write our къ, гъ, нъ as the Bashkir letters ҡ, ғ, ң, which the model already knows. From there it reads our text as if it were Bashkir. That whole story is in the first post.

The voice is a separate input. The model I fine-tuned learns the language, not the timbre. The timbre comes from a short reference clip, a few seconds of someone's recording with its exact transcript. Merdiven is read in the voice of Sevil, and every sentence of the book was conditioned on one and the same seven-second clip. Keep that in mind, it comes back later.

Fourteen sentences can't tell you how a voice behaves for ten minutes

Before the book, I compared model versions on fourteen probe sentences, stuffed on purpose with the hardest sounds we have: къ, гъ, нъ, дж. A fine regression set. It tells you nothing about whether the timbre drifts by the end of a chapter, what the model does with dialogue, names, numbers and one-word replies, or how it breathes. In a fifteen-second clip you don't hear any of that. In a book you hear all of it.

So the next step was one full chapter: chapter one of Merdiven, 1,608 words, about twelve minutes. The point was not to admire it. The point was to find what breaks.

First I made sure the text wasn't in the training material: zero matching runs of six consecutive words against nearly 1.4 million words. Otherwise the model wouldn't be reading, it would be remembering. Later the whole book went through the same check, also with zero matches.

Why this book? I love Şamil Alâdin's writing, and I wanted the first book a machine read in our language to be one of his. It also suited the model: even narrative prose. Humorous stories or children's books, where the voice has to play, were beyond it back then.

The first number was already a lesson. On the probe sentences, about 14% of characters came out wrong when I read the synthesis back with our own speech recogniser and compared it to the text. On the chapter's ordinary prose, under 2%. The model hadn't improved overnight. The probe set is hostile by design and ordinary text isn't. I no longer quote the probe number as if it described how the model actually reads. It's a tripwire, not an estimate.

Fourteen sentences couldn't tell two fixes apart

In the first post I told how I heard ниже (Russian for "lower") where the book said нидже ("how many"): my letter-mapping rule was also collapsing дж into ж, so the model was literally handed a Russian word. This is what happened next.

I compared five ways of spelling that sound on the input. On the fourteen probe sentences the two best were indistinguishable. Both fixed the word. So I re-rendered the whole chapter, 195 sentences, with each of them. One, the Tatar letter җ, fixed нидже and quietly damaged everything else: character errors across the chapter went up by 27%. Plain дж fixed the word and broke nothing.

Fourteen sentences said the two candidates were equally good. A chapter showed that one of them hurts. A fix that has only been checked on the example that was failing hasn't really been checked. Run it over enough ordinary text to see what else it touches.

What broke in chapter one

The verdict on chapter one surprised me. The voice held: the timbre barely changed from the first minute to the last, and the pace stayed even. The defects weren't in the model but in how text and audio were prepared around it. I'd written that part myself, which made it good news: none of it needed retraining.

Gluing short sentences dropped whole clauses

The model learned from clips no shorter than about three seconds, so I reasoned that a two-word line shouldn't go to it alone and glued short sentences to their neighbours. The logic felt airtight. I never tested it. That was a mistake.

On glued chunks the model started throwing out entire sentences:

"I was wrong. He isn't a teacher after all" turned into just "He isn't a teacher after all". The glued chunks were a quarter of the chapter's text and produced 43% of its errors.

Only then did I test the assumption itself and render the chapter with no gluing at all. Short sentences really are worse, roughly two and a half times the error rate, although in absolute terms that's about two percentage points. Errors keep falling until around seventy characters and barely move after that. And that penalty turned out smaller than the price of gluing, which was losing whole pieces of sentences. Gluing went off.

Chapter one rendered without gluing. Everything under 43 characters used to be glued to a neighbour, but errors keep falling until about seventy.

Before you build a workaround for something the model "can't handle", measure how badly it actually handles it. Here the workaround cost more than the problem it was meant to avoid.

The breath trimmer ate first words

Sometimes the synthesis opens a sentence with an inhale, like a human narrator would. I wrote a trimmer: everything before the first voiced sound counts as breath and gets cut. But plenty of first words start with a voiceless consonant. Çañ qaqıldı turned into Qaqıldı, Tışta, yeşergen… into yeşergen…. In 111 of 147 clips the cut landed inside the first word. The chapter scored better with no trimming at all than with my "fix".

The new trimmer looks at loudness instead of voicing. A consonant is quiet, but it's still far louder than the silence in front of it. With the new one, the chapter scores better than with no trimming.

Benchmark every post-processing step against doing nothing. Mine lost, and I only found out because I ran that comparison.

The 89% that was the trimmer grading itself

The first check reported that 89% of sentences opened with an inhale. Scary number. Then it turned out the trimmer was reporting on itself: it counted as breath everything before the first voiced sound, which included those same voiceless consonants. An honest measurement found a breath at the start of one sentence in five.

Most of the real breaths were somewhere else entirely: between words, exactly where the trimmer never looked. 76 of 195 sentences had them. I learned to turn those down to the level of the background noise without moving a single timestamp, and 68 of the 76 sentences transcribe exactly as they did before.

A component that defines what it detects will always agree with itself. And it only finds things where it looks.

Pace and pauses

The model reads faster than a human narrator: 17.4 characters per second against 14.5. A chapter I expected to run fifteen minutes came out at twelve. For an audiobook that sounds a little hurried.

The pauses between sentences I didn't make up. I measured them on ten minutes of a human narrator: the median pause between sentences was 288 milliseconds, and only three pauses in those ten minutes were longer than 0.6 seconds. So my pipeline puts 300 ms between sentences, a bit more when the speaker changes, and a bit more again between paragraphs.

After all of this, chapter one went from 2.56% character errors to 1.84%. Same model, same weights. What changed was what I fed it and what I did to the audio afterwards.

Sixteen chapters: bugs that only exist at book length

Then came the other fifteen chapters. I didn't re-render chapter one. It came out of the same pipeline, and after every fix below its extracted text was byte-for-byte identical to before. Which also means chapter one contained none of what came next, so a one-chapter test had no way to find it.

• The three asterisks that separate scenes, * * *, went to the model as text, and it dutifully read them out.
• The last paragraph of the novel is the author's dateline, "1940 s.". My extraction took it for ordinary text, so the novel ended with a year read aloud.
• Five sentences were torn in half wherever a paragraph broke off on an ellipsis.

None of that is the model. It's extraction, the least glamorous code in the whole pipeline. Run it over the entire text before you trust it on a sample.

A guard refused to build chapter seven, and it was right

The packaging step has guards built in. In chapter seven one of them refused to build the chapter: the words of one sentence, joined back together, did not reproduce the original sentence. The cause was a double space in the book file.

You can't hear a double space, and you can't see it on screen. But a reader that renders the text word by word would have lost it without a sound. It was the second time on this project that a program refusing to run caught something no amount of listening ever would.

That incident also changed how the run behaves. If a guard fires in one chapter, the remaining chapters keep going. The failed chapter still doesn't get built, the rest just don't wait for it. A book that stops at chapter seven at night is a lost night.

The checker is a model too

I checked every chapter separately, and one looked bad: chapter four, 8.3% character errors, four times the book's median. I went to listen, and the chapter was fine. The whole error came from one two-word line, "– dep tüşündi.", on which the checking recogniser got stuck in a loop and wrote it out twenty-five times in a row. Without that line the chapter is at about 2%, like the rest. The same line tripped the check in chapter two as well, it just stood out less there. The machine that read the book hadn't failed. The machine that checked it had.

That's also why one average for the whole book would have been the wrong number. A single looping line wrecks the mean and says nothing about the reading. Per chapter the picture is flat: median 2.15%, best chapter 1.5%, worst 2.7%. Quality doesn't drift towards the end of the book or on long chapters. The longest, chapter two, runs nineteen and a half minutes and reads at 2.5%.

Hatched: what checker glitches on short lines added, not the reading itself. Nearly all of it is one line, "– dep tüşündi.", which the recogniser looped on in chapter four and again in chapter two.

If your evaluation runs on a model, it has failure modes of its own. Report per unit, keep the checker's failures visibly apart from the system's, and listen to an outlier before you believe it.

On cost, as I wrote in the first post: two hours of audio take about three hours of my home computer. Synthesis itself is less than half of that. More goes into tying every word to a timestamp and checking everything. It's worth it.

A reader that highlights the current word

I didn't want just an audio file. I wanted a reader: the text on screen, the voice reading it, the current word highlighted, and a click on any word jumps playback there.

For that you need the time of every word. The trick the whole project started from helped again: align audio to text you already know. Only now the known text is exactly what the model was given as input. Each sentence is aligned on its own, against its own text, and then shifted into the chapter's timeline. In all sixteen chapters 100% of the words have a timestamp. The only things left without one are dashes and symbols that aren't pronounced.

The doubtful timestamps aren't scattered at random. Most of them sit on the first word after a dialogue dash, which is exactly where the recognition check stumbles too. Two independent instruments point at the same spots. So a low word score is now my cue to listen to that sentence, not to delete the timestamp.

Two more guards come from earlier mistakes. The timeline that had drifted 44 seconds by the end of a chapter is in the first post. Since then the build compares the end of the last sentence with the length of the audio file and refuses to run if they disagree by more than half a second. The other one is from the нидже story. I made the first build of the reader on the old narration, from before the дж fix, and the only symptom was the voice saying ниже. Now every chapter stores a fingerprint of its audio file, so a stale narration shows up in the data and not only to someone listening carefully.

Put together, the packaging step looks roughly like this:

# a chapter is written only if these hold assert " ".join(sentence.words) == sentence.text # words rebuild the sentence exactly assert abs(last_sentence_end - audio_seconds) <= 0.5 # the timeline matches the audio  # and the chapter records which audio it was built from chapter["audio_fingerprint"] = content_hash(audio_file) # stale narration is visible in data 

None of these is clever. Two of them are only there because something had already got past me once.

The reader is live on ana-yurt.com, one page for both reading and listening. You can open Merdiven and listen to the whole thing: ana-yurt.com/kutuphane/3444/read.

The reference clip is an input too

Since the timbre comes from the reference, picking a voice means picking a few seconds of audio. That clip turned out to matter more than I expected.

I first tried my own voice on single sentences and heard long, out-of-place pauses in the middle of them. My first thought was the model. It was the reference: 0.77 seconds of silence before the words and 1.77 seconds after. The model takes more than timbre from the reference. It also takes the tempo, how much time to spend on each letter, and it was spending that extra time as pauses inside phrases. I trimmed the silence at the edges, and the median total pause inside a sentence fell from 930 to 85 milliseconds. In a blind comparison the voice with the trimmed reference couldn't be told apart from the old one: 7 to 6. The defect went away and the voice stayed.

Then the voice started to sound dull to me. This time it was the microphone: a lavalier clipped to the chest sits off to the side of the mouth and loses exactly the frequencies that were missing, and a flat setting didn't help. No filter can paint that back in. What the mic didn't record isn't in the recording. I recorded the references again.

I also tried to help the male voice through training, by adding more male speech to the training material. In a blind comparison the new version lost to the previous one, 3 to 11, and one of the female voices slid lower in pitch, from 208 to 199 Hz. I only added the pitch measurement after my ear had spoken, and both pointed at the same voice. That version didn't stay.

If you build voice cloning into a pipeline, treat the reference clip like any other input data and check it. A bit of silence at its edges turned into pauses in the middle of every sentence.

Once the pauses and the microphone were sorted out, I decided to try my own voice on a whole book. So the second book in our library, Yunus Qandım's Küreş meydanını ot basmaz, 28 chapters, 6 hours 29 minutes of audio, was read by the machine in my voice. You can listen to it as well: ana-yurt.com/kutuphane/3516/read. It was read by the current version of the model. Merdiven was read by an earlier one.

This book wasn't a random pick either. Its author, Yunus Qandım, is my uncle, and I really wanted his book to be heard in my voice.

Usually, when you hear your own voice in a recording, it sounds foreign and a bit strange. This time it didn't. When I heard the first sentence, I was over the moon: the quality was very good.

What still doesn't work

Stress. This one went in a circle, and I care about it more than the rest. I noticed the model sometimes reads the name Riza as Rıza, with a back vowel. A check confirmed it happens in about half the sentences, and the same goes for Aziz. There was a fix: spell the vowel on the input as the Cyrillic «ий» instead of «и». The back vowel disappeared completely and the errors in the name went down. The instruments were happy.

I listened blind and rejected it. My note on the blind page said: "if I hear И, the stress is wrong." The fix brought back the right vowel and dragged the stress to the first syllable, while ours sits on the last. Both versions were wrong, each in its own way.

I wasn't too upset. These are first models, and I accept mistakes like this from them. I decided to come back to it later.

Then I started measuring the stress itself. In our language the last syllable of a word is usually stretched a little, and human narrators stretch some words a lot and others hardly at all. On average the model stretches the last syllable as much as people do. But it reproduces only about six tenths of that range: it under-stretches exactly the words narrators stretch most and over-stretches the flat ones.

So I went through the suspects one at a time. The reference clip changes only the overall tempo. The training recordings have the stretch where it belongs, so there was something to learn from. The inference settings change nothing. And the base model, before my fine-tuning, has the same flatness. The next suspect is how the model decides the length of the whole phrase up front. If the total duration is fixed in advance, the last syllable can only grow at its neighbours' expense. For now that's a hypothesis, and testing it means working on the model itself, not on the text that goes into it.

Breaths. Breath cleaning never made anything worse, but short breaths, around a tenth of a second, partly remain. My voice has the most: in one measurement breaths took up 15% of the audio, against about 1% for Sevil. Trimming the silence in the reference cut that by more than half, but not to zero.

Pauses on emotional lines. I had the impression that the model pauses longer where the text is more emotional. I tested that on sentences picked so that calm and emotional ones didn't differ in length or number of commas, and found no link. Pauses follow sentence length and commas, and emotional sentences in prose just tend to be longer. But I could only measure the total length of the pauses, not where they fall. A pause in the wrong place in an exclamation grates more than the same pause at a comma, so for me the question is still open.

Dialogue. The model sees the dash before a reply, but it reads a change of speaker as a new sentence, not a new voice.

Numbers. I turn them into words beforehand, but recognition can't check how they were read: it writes down what it hears as digits, and "1924" comes back as "19124". For now only an ear checks the numbers.

The whole list, in one place

The voice turned out to be the most reliable part of the whole chain. Nearly everything that broke was in the handling of text and audio around it: gluing, trimming, letters, timestamps, a double space. Some of it only instruments found, and no ear ever would have. Some of it only an ear found, and the instruments couldn't even score it: нидже, the stress in Riza, breaths between words. And the whole book found things that neither fourteen sentences nor one chapter would ever have shown.

When I finished listening to the book, I felt proud. The work hadn't been for nothing, and the result was excellent. But that's no reason to relax. You can still hear places in the book that need work, which means there's plenty of it ahead.

Third post in a series. Next I'll go deeper into how I chose a base for the voice that can already say our къ, and why neighbouring Turkish doesn't work for that. After that, as promised, cloud GPUs.

Audio samples and project page: https://ai.ana-yurt.dev/.

Want to help? What the project needs most is native speakers. Listen to a chapter and note where the stress lands in the wrong place, where a name is misread, where a pause tears a phrase apart. You don't need to program, just a good ear for your own language. In this book the most important finds were made by ear. Write to support@ana-yurt.com.

If you'd rather support the work financially, you can do it here: ko-fi.com/anayurt. That support turns into new recordings of living speech, and those need to be made now, while there are still people to learn from.