A clip from our launch film. Same script, same voice. Left: what hand-timed captions tend to look like. Right: captions built from the voice's own timing data.
If you generate narration with ElevenLabs and caption it in Remotion, you already have what you need for exact word timing. The with-timestamps endpoint returns the audio and a start and end time for every character, in the same response.
This post walks through the pipeline we use for narrated videos:
- Get audio and character timings in one request
- Group characters into words and sentences
- Render a word-by-word caption in Remotion, where each sentence clears before the next one starts
- Snap cuts to the beat of the music
- Run an automated QA pass with ffmpeg: frozen frames and loudness
The snippets are short teaching versions. Each one was run or type-checked as shown (Node 18+, Remotion 4.0, ffmpeg on your PATH).
Why captions drift
Hand-timed captions are placed by ear. Every cue carries a small error, and a word-by-word build needs a cue per word instead of one per sentence. Then the script changes, the voice is regenerated, and every cue after the edit is in the wrong place.
Speech-to-text after the fact (Whisper and friends) is much better, but it's still an estimate made from the audio. Word boundaries around pauses and breaths are guessed, and the transcript doesn't always match your script word for word. Numbers, product names and brand spellings are the usual suspects. You end up mapping the transcript back onto the script and fixing the stragglers by hand.
With TTS you can skip both steps. The service that produced the audio tells you where each character is, and the text is your script exactly as you sent it.
1. Audio and timings in one request
// tts.mjs (Node 18+): one request, audio + character timings back import fs from "node:fs"; const VOICE_ID = "JBFqnCBsd6RMkjVDRZzb"; // any voice from your ElevenLabs library const text = "Captions should land on the word. Not near it."; const res = await fetch( `https://api.elevenlabs.io/v1/text-to-speech/${VOICE_ID}/with-timestamps?output_format=mp3_44100_128`, { method: "POST", headers: { "xi-api-key": process.env.ELEVENLABS_API_KEY, "Content-Type": "application/json", }, body: JSON.stringify({ text, model_id: "eleven_multilingual_v2" }), }, ); if (!res.ok) throw new Error(`${res.status}: ${await res.text()}`); const { audio_base64, alignment } = await res.json(); fs.writeFileSync("voice.mp3", Buffer.from(audio_base64, "base64")); fs.writeFileSync("alignment.json", JSON.stringify(alignment, null, 2)); alignment looks like this (first nine characters of the real response):
{ "characters": ["C", "a", "p", "t", "i", "o", "n", "s", " "], "character_start_times_seconds": [0, 0.116, 0.232, 0.29, 0.348, 0.383, 0.441, 0.499, 0.546], "character_end_times_seconds": [0.116, 0.232, 0.29, 0.348, 0.383, 0.441, 0.499, 0.546, 0.58] } The response also includes normalized_alignment, which follows the text after normalization (numbers spelled out and so on). Use alignment if you want the captions to show the script as you wrote it.
A note on models. In our tests, eleven_multilingual_v2 alignment matched the audio closely. The newer v3-generation models returned timings that ran about 0.7 to 1 second early. If your captions look consistently early, try generating with eleven_multilingual_v2 before you start adding offsets by hand.
2. Characters to words to sentences
Split on whitespace, take the start time of the first character and the end time of the last one:
// words.mjs: ElevenLabs character alignment -> words -> sentences export function toWords(al, offset = 0) { const words = []; let text = "", start = 0, end = 0; const flush = () => { if (text) words.push({ text, start: start + offset, end: end + offset }); text = ""; }; al.characters.forEach((ch, i) => { if (/\s/.test(ch)) return flush(); if (!text) start = al.character_start_times_seconds[i]; end = al.character_end_times_seconds[i]; text += ch; }); flush(); return words; } export function toSentences(words) { const out = [[]]; for (const w of words) { out[out.length - 1].push(w); if (/[.!?]["')\]]*$/.test(w.text)) out.push([]); } return out.filter((s) => s.length > 0); } Run on the response above:
{ text: 'Captions', start: 0, end: 0.546 } { text: 'should', start: 0.58, end: 0.766 } { text: 'land', start: 0.813, end: 0.998 } ... { text: 'it.', start: 2.543, end: 3.019 } [ 'Captions should land on the word.', 'Not near it.' ] offset is for when the voice doesn't start at 0 in your composition. If the narration starts at 2 s, pass 2 and every word time is already in composition time.
Punctuation stays attached to its word, which is what you want on screen. Splitting on whitespace works for any language that separates words with spaces, Korean included.
3. The word-build caption in Remotion
An example render from the kit: words appear as they're spoken, and each sentence clears before the next one starts.
The rules behind it:
- One sentence on screen at a time. The live sentence is the last one whose first word has started.
- The old sentence leaves before the new one arrives. It fades out during the last 0.1 s before the next sentence's first word, so two sentences never share the screen.
- New words settle in quickly. They enter dim and at 85% scale and reach full size in 0.1 s, scaling from the bottom-left so the line doesn't wobble.
- Words show one frame early. On the exact frame the word starts, it felt a touch late to us. One frame ahead reads as on time.
// WordBuildCaption.tsx import React from "react"; import { AbsoluteFill, interpolate, useCurrentFrame, useVideoConfig } from "remotion"; type Word = { text: string; start: number; end: number }; const CLEAR = 0.1; // seconds the old sentence takes to leave const ENTER = 0.1; // seconds a new word takes to settle export const WordBuildCaption: React.FC<{ sentences: Word[][] }> = ({ sentences }) => { const frame = useCurrentFrame(); const { fps } = useVideoConfig(); const t = frame / fps + 1 / fps; // show each word one frame early // the live sentence is the last one whose first word has started let idx = -1; sentences.forEach((s, i) => { if (s[0].start <= t) idx = i; }); if (idx < 0) return null; const words = sentences[idx]; const next = sentences[idx + 1]?.[0].start; // fade out during the last CLEAR seconds before the next sentence, so two never overlap const opacity = next === undefined ? 1 : interpolate(t, [next - CLEAR, next], [1, 0], { extrapolateLeft: "clamp", extrapolateRight: "clamp", }); return ( // anchored at the top: a wrapped line grows downward and nothing above it moves <AbsoluteFill style={{ padding: 96, paddingTop: 700 }}> <div style={{ fontSize: 84, fontWeight: 700, lineHeight: 1.15, opacity }}> {words .filter((w) => w.start <= t) .map((w, i) => { const p = interpolate(t, [w.start, w.start + ENTER], [0, 1], { extrapolateRight: "clamp", }); return ( <span key={i} style={{ display: "inline-block", marginRight: "0.25em", opacity: 0.35 + 0.65 * p, transform: `scale(${0.85 + 0.15 * p})`, transformOrigin: "0% 80%", }} > {w.text} </span> ); })} </div> </AbsoluteFill> ); }; And in the composition:
// Video.tsx import React from "react"; import { AbsoluteFill, Html5Audio, staticFile } from "remotion"; import { WordBuildCaption } from "./WordBuildCaption"; import sentences from "./sentences.json"; // output of toSentences(toWords(alignment)) export const Video: React.FC = () => ( <AbsoluteFill style={{ background: "#0d0d10", color: "white", fontFamily: "Inter, sans-serif" }}> <Html5Audio src={staticFile("voice.mp3")} /> <WordBuildCaption sentences={sentences} /> </AbsoluteFill> ); Html5Audio is the current name in Remotion 4. On older 4.0 versions it's <Audio>, with the same props.
Two details make this hold up. Words that haven't started aren't rendered, transform doesn't affect layout, and the block is anchored at the top, so earlier words stay put when a new one arrives or a line wraps. And because every time comes from the alignment, regenerating the voice after a script edit re-times the captions with it, with no cues to move by hand.
A production version needs more than this: a fixed line width so wrapping is predictable, waiting for fonts before measuring, and a cap on lines per sentence. The timing logic stays the same.
4. Snap cuts to the beat, leave the voice alone
With music under the narration, scene cuts that land on a beat feel deliberate. The rule we follow: snap the visuals to the beat and leave the voice where it is. Cuts, hits and sound effects move. Speech doesn't, because nudging words onto a grid makes them sound unnatural.
Any beat tracker that gives you beat times in seconds will do (librosa's beat_track, for example). Ours is a small Node script that reads raw PCM from ffmpeg, builds an onset envelope, estimates tempo by autocorrelation, then picks the grid phase that lands on the most onset energy, with the bass band weighted so kicks win over hi-hats.
Once you have the beats, snapping is small:
// snap.mjs: move a cut onto the nearest beat, but only if one is close export function snapToBeat(t, beats, maxShift = 0.12) { let best = t, dist = Infinity; for (const b of beats) { const d = Math.abs(b - t); if (d < dist) { dist = d; best = b; } } return dist <= maxShift ? best : t; } const fps = 60; const cuts = sentences.slice(1).map((s) => { const wanted = s[0].start - 0.15; // cut a little before the sentence starts const at = snapToBeat(wanted, beats); // pull it onto a beat if one is within 120 ms return Math.round(at * fps); // Remotion works in frames }); Run on the example from section 3 with a 120 BPM track (a beat every 0.5 s), two of the five cuts moved and three stayed where the voice put them:
1.278 s -> 1.278 s kept (no beat within 120 ms) 2.242 s -> 2.242 s kept 4.587 s -> 4.540 s snapped 6.514 s -> 6.536 s snapped 8.859 s -> 8.859 s kept The maxShift cap matters. Without it, a cut can get dragged a quarter second away from the sentence it belongs to, and the edit feels off in a different way.
5. QA before you post: frozen frames and loudness
Two problems that viewers notice and a render log won't tell you about: a stretch where nothing moves (a scene that ran out, a missing asset, a hold you forgot about), and audio that's much quieter or louder than the videos around it in the feed.
Both are one ffmpeg filter each:
// qa.mjs: node qa.mjs out/video.mp4 import { spawnSync } from "node:child_process"; const file = process.argv[2]; const ff = (args) => spawnSync("ffmpeg", ["-hide_banner", "-i", file, ...args, "-f", "null", "-"], { encoding: "utf8", maxBuffer: 1 << 28 }).stderr; // 1) frozen stretches of 0.5 s or more const fz = ff(["-map", "0:v:0", "-vf", "freezedetect=n=0.001:d=0.5"]); const freezes = [...fz.matchAll(/freeze_start: ([\d.]+)[\s\S]*?freeze_duration: ([\d.]+)/g)] .map((m) => ({ at: +m[1], len: +m[2] })); // 2) integrated loudness (EBU R128) const ebu = ff(["-map", "0:a:0", "-af", "ebur128=peak=true"]); const summary = ebu.slice(ebu.lastIndexOf("Summary:")); const lufs = Number(/I:\s+(-?[\d.]+) LUFS/.exec(summary)?.[1]); console.log("freezes:", freezes.length ? freezes : "none"); console.log(`loudness: ${lufs} LUFS`, Math.abs(lufs + 14) <= 1 ? "OK" : "FIX (target -14)"); On a test file with a 1-second freeze spliced in and the audio turned down:
freezes: [ { at: 2.983333, len: 1.016667 } ] loudness: -23.6 LUFS FIX (target -14) n=0.001 is the noise tolerance (-60 dB, the filter's default) and d=0.5 is the minimum length. A rendered video that isn't moving repeats identical frames, so a low tolerance still catches real freezes, while slow camera pushes and caption ticks count as motion. If your video ends on an end card that holds on purpose, ignore a freeze that runs into the last second or so.
To fix loudness, use two-pass loudnorm. The first pass measures, the second applies the measured values as one linear gain, and the video stream is copied untouched:
// loudnorm.mjs: node loudnorm.mjs in.mp4 out.mp4 (two-pass, -14 LUFS) import { spawnSync } from "node:child_process"; const [input, output] = process.argv.slice(2); const ff = (args) => spawnSync("ffmpeg", ["-hide_banner", ...args], { encoding: "utf8" }).stderr; const target = "I=-14:TP=-1.5:LRA=11"; // pass 1: measure const log = ff(["-i", input, "-af", `loudnorm=${target}:print_format=json`, "-f", "null", "-"]); const m = JSON.parse(log.slice(log.lastIndexOf("{"), log.lastIndexOf("}") + 1)); // pass 2: apply the measured values as one linear gain const af = `loudnorm=${target}:measured_I=${m.input_i}:measured_TP=${m.input_tp}` + `:measured_LRA=${m.input_lra}:measured_thresh=${m.input_thresh}:offset=${m.target_offset}:linear=true`; ff(["-y", "...
