Offline live subtitles with Whisper and M2M100: how it works and what it costs
Cloud captioning is convenient until you look at the bill, the latency, or the privacy policy. For streamers translating Russian speech into English in real time, a local pipeline can be faster, cheaper, and fully private. Here is how ours works.
The pipeline
Audio → VAD (voice activity detection) → Whisper (speech recognition) → local translation → WebSocket → OBS overlay.
- Capture — microphone or system audio (WASAPI loopback on Windows).
- VAD — detects phrase boundaries so we don't feed silence or game noise into the model.
- Whisper — transcribes speech; for RU→EN it can translate directly with
task=translate. - Translation model — for other language pairs we use local models: M2M100 (MIT) or the fast Helsinki opus-mt-ru-en (CC-BY-4.0).
- Output — a WebSocket stream rendered as a browser source in OBS, with configurable font, colors and line history.
- Filters — hallucination and repetition filters keep captions readable.
Latency
On a modern NVIDIA GPU we see roughly 0.3–0.7 s per phrase, including translation. On CPU it still works, just slower (1–2 s and more depending on the model). For live captions, predictable latency matters more than "maximum accuracy": medium on a GPU is the sweet spot for most streams.
Why we removed NLLB
NLLB is a great model, but its license (CC BY-NC 4.0) forbids commercial use. Shipping it inside a paid product is not an option, so in Stream Subs 1.8 we removed NLLB and kept:
- M2M100-418M (MIT) — local translation for 15 languages, ~0.5 s per phrase;
- Helsinki opus-mt-ru-en (CC-BY-4.0) — very fast RU→EN (120–230 ms);
- Whisper translate — one model, simplest path;
- optional LLM via API — better quality when the user explicitly opts in (text leaves the machine, so it is off by default).
Cloud vs local: the honest comparison
| Cloud captions | Local (Whisper + M2M100) | |
|---|---|---|
| Latency | depends on your uplink | 0.3–0.7 s on GPU |
| Cost | subscription, minute limits | one-time purchase, no limits |
| Privacy | audio goes to someone's server | everything stays on your machine |
| Works offline | no | yes |
| Hardware | none | NVIDIA GPU recommended |
For a streamer doing 10+ hours a month, a cloud subscription usually costs more in the first year than a one-time license — and the audio never leaves their PC.
What we shipped
Stream Subs 1.8 adds a CPU build for machines without a discrete GPU, session statistics (lines, translations, average latency), presets for stream/webinar/recording scenarios, and a choice of translation engine. It is open source (MIT): github.com/TenLux-dev.
Product page: tenlux.tech/stream-subs
What we learned
- VAD quality matters more than model size. Bad segmentation ruins captions faster than a small model.
- Filters are mandatory. Whisper will happily invent phrases on silence.
- Licenses decide architecture. One model with a non-commercial license can force a rewrite.
- CPU mode is not a gimmick. Many streamers have no discrete GPU; supporting them doubled the audience.
If you are building something similar, or you stream and want to try it, feedback is welcome — we are a small studio, and real use cases shape the roadmap more than feature lists.
