Add speaker labels to video captions with WhisperX and WebVTT voice spans

TL;DR

We'll turn a two-person video into a WebVTT caption file where each cue says who is speaking. Pipeline: ffmpeg extracts audio, whisperx --diarize transcribes and labels speakers, a small Python script groups words into <v Speaker> cues.

Plain speech-to-text gives you a wall of words. For interviews, podcasts and meetings, readers need to know who said what. We'll use WhisperX (Whisper with voice activity detection, forced alignment and pyannote-based diarization) and the standard WebVTT voice span so any player can show the names. #python #video

You'll need python 3.10+, ffmpeg, a Hugging Face account, and ideally a CUDA GPU. CPU works with --compute_type int8 --device cpu, just slower.

1. 🔑 Get access to the diarization model

WhisperX's diarization uses pyannote's speaker-diarization-community-1 pipeline, which is gated on Hugging Face:

1. Create a read token at hf.co/settings/tokens.
2. Open the pyannote/speaker-diarization-community-1 model page and accept the user conditions.

export HF_TOKEN=hf_xxx # never commit this 

⚠️ Note: if you skip the user-conditions step, the pipeline fails to load with an authorization error even with a valid token. Accept the conditions first.

2. 🎧 Extract audio from the video

# extract_audio.sh ffmpeg -y -i interview.mp4 -vn -ac 1 -ar 16000 audio.wav 

Mono, 16 kHz is what speech models work with internally, and a WAV file avoids decoder surprises.

3. 🗣️ Transcribe and diarize

pip install whisperx whisperx audio.wav --model large-v2 --diarize --hf_token "$HF_TOKEN" \ --output_format json --output_dir out 

On CPU:

whisperx audio.wav --model large-v2 --diarize --hf_token "$HF_TOKEN" \ --compute_type int8 --device cpu --output_format json --output_dir out 

The README lists the CPU flags and the diarize flag. WhisperX does four things in order: voice activity detection, Whisper transcription, wav2vec2 forced alignment (word-level timestamps), then pyannote diarization with speakers assigned to words.

💡 Tip: WhisperX's README claims 70x realtime for large-v2 batched transcription. That's their best case. Time it on your own hardware before planning capacity.

4. 🔍 Look at the JSON

Each segment has a speaker, and each aligned word carries one too (shortened example):

{ "segments": [ { "start": 0.5, "end": 4.2, "text": "Welcome back to the show.", "speaker": "SPEAKER_00", "words": [ {"word": "Welcome", "start": 0.5, "end": 1.0, "speaker": "SPEAKER_00"}, {"word": "42", "speaker": "SPEAKER_01"} ] } ] } 

Notice the last word: numbers and symbols often come back with no start/end. Our converter has to skip those or it will crash.

⚠️ Note: the exact JSON fields can shift between WhisperX versions. Open one real output file and confirm words[].speaker exists before trusting the script below.

5. 🧩 Group words into WebVTT cues

The rules:

• a new cue starts when the speaker changes,
• or after a pause longer than 1.5 s,
• or when a cue would run longer than 7 s (so captions stay readable).

#!/usr/bin/env python3 """Convert WhisperX JSON (with --diarize) into a WebVTT file using <v Speaker> voice spans.""" from __future__ import annotations import json import sys MAX_CUE_SECONDS = 7.0 # split long monologues so captions stay readable GAP_SECONDS = 1.5 # a pause this long starts a new cue even for the same speaker  def ts(seconds: float) -> str: ms = round(seconds * 1000) h, ms = divmod(ms, 3_600_000) m, ms = divmod(ms, 60_000) s, ms = divmod(ms, 1000) return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}" def words_with_speakers(data: dict) -> list[dict]: out = [] last_speaker = "UNKNOWN" for seg in data["segments"]: for w in seg.get("words", []): if "start" not in w: # numbers and symbols often get no timestamp  continue last_speaker = w.get("speaker", seg.get("speaker", last_speaker)) out.append({"word": w["word"], "start": w["start"], "end": w["end"], "speaker": last_speaker}) return out def build_cues(words: list[dict]) -> list[dict]: cues, cur = [], None for w in words: new = ( cur is None or w["speaker"] != cur["speaker"] or w["start"] - cur["end"] > GAP_SECONDS or w["end"] - cur["start"] > MAX_CUE_SECONDS ) if new: cur = {"speaker": w["speaker"], "start": w["start"], "end": w["end"], "text": [w["word"]]} cues.append(cur) else: cur["text"].append(w["word"]) cur["end"] = w["end"] return cues def to_vtt(cues: list[dict], names: dict[str, str]) -> str: lines = ["WEBVTT", ""] for i, c in enumerate(cues, 1): name = names.get(c["speaker"], c["speaker"]) lines += [str(i), f"{ts(c['start'])} --> {ts(c['end'])}", f"<v {name}>{' '.join(c['text'])}", ""] return "\n".join(lines) if __name__ == "__main__": data = json.load(open(sys.argv[1])) names = json.loads(sys.argv[2]) if len(sys.argv) > 2 else {} print(to_vtt(build_cues(words_with_speakers(data)), names)) 

Run it, mapping the generic labels to names you know:

python3 diar_to_vtt.py out/audio.json '{"SPEAKER_00":"Host","SPEAKER_01":"Guest"}' > captions.vtt 

On a small hand-written sample in the same shape as WhisperX output, the result is:

WEBVTT 1 00:00:00.500 --> 00:00:02.100 <v Host>Welcome back to the show. 2 00:00:02.600 --> 00:00:04.100 <v Guest>Thanks for having me. 

The <v Name> syntax is the WebVTT voice span from the W3C spec. The closing </v> can be omitted when the span covers the whole cue, which is the case here.

6. ▶️ Attach it to a player

<!-- player.html --> <video controls src="interview.mp4"> <track kind="captions" srclang="en" label="English" src="captions.vtt" default /> </video> 

HLS players take the same file as a subtitles rendition if you package it. Style the speaker with the ::cue(v[voice="Host"]) selector.

/* captions.css */ video::cue(v[voice="Host"]) { color: #ffd166; } video::cue(v[voice="Guest"]) { color: #8ecae6; } 

7. 🚧 Limits to know before shipping

What's next

• Score your own accuracy: hand-label ten minutes of your content and compute diarization error rate against it.
• Index the speaker field so search can answer "what did the guest say about X".
• If your content has fixed hosts, look at speaker-embedding matching to name them automatically.

Wrapping up: three commands and one small script give you captions that identify speakers, in a format every player already understands.