Everyone who builds an AI dubbing pipeline hits the same wall in week two. The ASR works, the translation reads fine, the TTS sounds like the speaker. Then you lay the synthesized German over the original video and it is still talking 1.3 seconds after the speaker’s mouth closed, and the next line starts on top of it.
This post is about that wall. Not the voice model, not the translation model, but the step between them that nobody writes about: taking a translated segment whose natural spoken duration is wrong and making it land where the original did. At AI Video Translator this step went through four rewrites before it stopped generating support tickets, and the final version is less clever than the first one.
https://ai-video-translator.com/
Why the durations are wrong in the first place
Translated text is a different length from the source. By character count, English to German typically expands 10 to 35 percent, Spanish 15 to 25, French 15 to 20, while Japanese and Chinese contract by a third or more. Speech rate then modulates this: Spanish and Japanese speakers produce around 7.8 syllables per second, English about 6.2, German about 6.0. So Spanish expansion partly cancels out, Japanese contraction almost fully cancels out, and German gets hit twice: more material and a slower rate.
The practical result: a 4.0 s English segment comes back from TTS as roughly 4.0 to 4.4 s in Japanese, 4.3 to 4.8 s in Spanish, and 4.8 to 5.6 s in German. Nothing about that is a defect. It is what the languages do.
The naive fix and why it fails
The first thing everyone does:
python
ratio = target_duration / synthesized_duration
audio = time_stretch(audio, ratio) # phase vocoder / WSOLA
Stretch every segment to its original slot. It works up to about a 10 percent compression. Between 10 and 15 percent listeners notice something is off even if they cannot say what. Beyond 15 percent the dub sounds like legal disclaimers at the end of a radio ad. German needs 20 to 40 percent on a bad line, so the naive fix produces unusable German while looking fine on Japanese test clips. If your test set is Japanese, you will ship this and find out from users.
What the final version does
The current approach is a budget-and-fallback chain. Each segment carries a timing budget derived from the original, and four strategies are tried in order of how much they damage the result.
- Compute the real budget, not the segment duration
The original segment runs from t_start to t_end, but the speaker does not necessarily start talking again at t_end. There is usually silence after it: a breath, a cut, a pause before the next sentence. That silence is free time, as long as it belongs to the same speaker.
python
def budget(seg, next_seg, speaker_turn_end):
hard_end = min(next_seg.start, speaker_turn_end)
slack = hard_end - seg.end
return seg.duration + min(slack, MAX_SPILL) # MAX_SPILL ~ 0.6 s
Two details that mattered. First, cap the spill; a line that drifts a full second late feels wrong even if nothing collides with it, because the viewer sees the mouth stop. Second, the speaker-turn boundary is a hard wall. With one speaker you can borrow any pause. With two, the pause after speaker A’s line may be the lead-in to speaker B’s line, and borrowing it makes the two synthesized voices overlap where the originals never did.
- Mild time stretch, bounded python MAX_STRETCH = 1.12 needed = synth_dur / budget if needed <= MAX_STRETCH: return time_stretch(audio, 1 / needed)
A 12 percent ceiling is conservative. It is also the point past which our own listening tests stopped agreeing on whether the voice was “fine”, so it is where the automatic path stops.
- Re-translate against a length target
This is the strategy with the most headroom and the one most pipelines skip because it means the translation step and the timing step have to talk to each other.
When a segment exceeds the stretch ceiling, we call the translation model again with the same context plus a constraint: the target rendering should be at most N syllables (or characters, for languages where syllable counting is unreliable) and should preserve meaning over phrasing. N is derived from the budget and the language’s speech rate:
python
syllable_rate = {"de": 5.97, "en": 6.19, "es": 7.82, "fr": 7.18, "ja": 7.84}
max_syllables = int(budget * syllable_rate[lang] * 0.95)
The model is good at this. “We’re now going to take a look at how this works in practice” becomes “Now let’s see how this works.” The meaning survives and the line fits. We allow at most two re-translation attempts with tightening budgets; after that we stop, because the third attempt starts dropping information.
One subtlety: re-translating a single line in isolation can break consistency with the lines around it (a term rendered one way in the long version and another way in the short one). The glossary layer that pins fixed terms across the whole document also applies here, and the re-translation call gets the same surrounding-context window the first pass got.
- Flag it for a human
If the line still does not fit after re-translation plus mild stretch, it gets marked in the review UI with the overrun amount, and the user can edit the text and re-voice just that line. The billing follows: re-voicing one line costs seconds of new speech, not a full render, because if fixing a line costs the same as the whole video nobody fixes lines.
The order is the point. Stretch is cheap and bounded. Re-translation is free in audio terms and has the most range. Human editing is the backstop. The first version of this pipeline did them in the wrong order (stretch as hard as needed, then give up) and the fourth version is the one above.
Measuring it
The metric that tracks user complaints best is not average overrun but the 95th percentile of synth_dur / budget per video, after all automatic strategies. A video with a p95 under 1.0 almost never gets a timing complaint. A video with p95 above 1.15 almost always does, even if the mean looks fine, because the viewer remembers the one line that spilled over the scene cut.
Per language on our content mix, after the full chain: Japanese and Mandarin need no re-translation on most segments; Spanish and French need it on roughly one segment in ten; German on roughly one in four. German remains the language where the human step gets used, and it is the language I would test first if I were evaluating anyone else’s dubbing tool.
Things I’d tell myself at the start
Build the budget from the next speech onset, not the segment end. Most of the free time is in the pauses.
Treat speaker boundaries as walls from day one, even if you only support one speaker today.
Length-constrained re-translation is the big lever. Wire the MT step so it can be called with a budget.
Don’t tune on Japanese. It is the easy case and it will lie to you.
Expose overruns to the user per line. They will fix the two lines that matter far faster than you can tune the model to.
The user-facing version of this, with the W3C and Lyon speech-rate numbers behind it, is here:
