One Minute of Audio, Three Runs: 13 Segments, Then 9, One of Them 21.94 Seconds Long

I run a visa agency in Bali. The half of my week that has nothing to do with visas goes into Apify Actors, and one of them is a transcriber: give it an audio or video URL, it runs Whisper large v3 and hands back a transcript, timed segments, SRT or WebVTT subtitles, and optional retrieval chunks that keep their timecodes.

The transcript is the part everyone checks. It is also the part that is nearly stable. What is not stable is the structure around it, and almost every pipeline I have seen built on a transcription API quietly assumes the opposite.

Three runs of the same file

One sample: a 60.01-second m4a of someone reading the opening of Pride and Prejudice three times over. Same Actor build (IdudfuIRo8hXFkRgH), same input — quality: fast (Whisper large v3 turbo), language: en. Run on 13 October, 14 October, and again on 14 October twelve hours later.

The words are fine. A diff of the two transcripts that differ at all is two tokens out of 215:

- neighborhood. This truth is so well fixed + neighborhood, this truth is so well fixed  - property of some one or other + property of someone or other 

A comma that became a full stop, and a space that appeared inside "someone". If you are measuring word error rate, this file is stable to 0.5%.

Now look at the structure. Between those same two runs the segment count moved 44%, and the longest unit of output went from five seconds to twenty-two.

And here is my favourite detail, which has nothing to do with run-to-run variance: the recording loops the same sentence three times, so one run contains the same audio three times over. The 14 October run transcribed it as "someone or other" at 15.30 s and as "some one or other" at 34.50 s and 53.74 s. Identical audio, one run, two spellings. Do not build a deduplicator that compares transcript strings.

Why a 21.94-second segment exists

Whisper works in windows of up to 30 seconds and returns segments, not words. How it carves a window into segments depends on where it thinks the sentences end — and that decision is downstream of sampling, so it moves between runs even when the text does not. Nine segments and thirteen segments are both legitimate readings of the same sixty seconds.

The 9-segment run put the final third of the file into one object:

{ "start": 38.06, "duration": 21.94, "text": "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. However little known…" } 

414 characters with a single start time. Pass segments straight into an SRT writer — the obvious implementation, and the one I shipped first — and a viewer gets a paragraph filling the screen for twenty-two seconds. I caught it on a jfk.flac test last September: 108 characters held for 11 seconds, valid SRT, unwatchable.

So subtitles are not built from segments here. They are rebuilt from them:

LINE_MAX = 42 # the Netflix/BBC reading norm CUE_MAX = 2 * LINE_MAX CUE_SECONDS_MAX = 7.0 count = max(1, math.ceil(len(text) / CUE_MAX), math.ceil(duration / CUE_SECONDS_MAX)) 

Each segment is split into at least that many cues, preferring breaks after a comma or a sentence end, then timed by dividing the segment's duration in proportion to characters. That proportional split is an approximation and I want to be honest about it: the API gives segment timings only, no word timestamps, so there is nothing better available without a forced-alignment pass.

Those 9 segments came out as 21 cues, longest 4.51 seconds:

16 00:00:38,060 --> 00:00:42,405 It is a truth universally acknowledged, that a single man in possession of a good 17 00:00:42,405 --> 00:00:46,750 fortune, must be in want of a wife. However little known the feelings or views of 

One measured line came out 45 characters rather than 42: the wrapper needs a break point that leaves both halves under the limit, and a long unbroken clause has none, so it falls back to the nearest space. Three characters over beats hyphenating.

The segments also do not cover the file

Add up the durations: 57.12 s, 58.12 s, 58.30 s against a 60.01-second file. Between two and three seconds of every run live in the gaps between segments. On a reading with real pauses that is correct behaviour.

Sometimes it is not. Whisper can skip to its next 30-second window and silently drop the speech in between, and the status still says ok — a 134-second reading came back with roughly 32 seconds missing in one of my golden checks. The transcript reads fluently. Nothing in the output says a third of it is gone.

The Actor now scans for holes of 6 seconds or more, cuts each one out of the prepared audio, and recognises it again with the preceding text as a prompt. Recovered segments are only kept if the service itself is confident it heard speech:

if float(item.get('no_speech_prob') or 0) > 0.6 or float(item.get('avg_logprob') or 0) < -1.0: continue 

That filter is not optional. Whisper hallucinates over silence, and its favourite invention is a polite "Thank you." Re-check a genuine pause without the confidence gate and you fill real silences with phantom gratitude, which is worse than the gap.

What this breaks downstream

Retrieval chunks inherit the instability. Chunks are assembled from segments, breaking on sentence ends. The 9-segment run gave me three chunks starting at 0, 25.54 and 38.06 seconds, with 491, 255 and 414 characters. The 13-segment run of the same audio cannot give the same boundaries — it has different sentence ends to break on. So: never use chunk index or chunk start time as a stable key in a vector store. Re-embed whole files, keyed on the file, and delete the old vectors rather than diffing them.

Never diff two transcription runs. Any sane diff of these three runs reports changes. One is a real text change of two tokens; the rest is re-segmentation noise. If you need change detection on a recurring recording, compare normalised text — lowercased, punctuation stripped, whitespace collapsed — and never segment counts.

Speaker labels are per-run identities. They are computed inside the Actor from the audio (Silero VAD to cut phrases at pauses, 3D-Speaker CAM++ for one voice print per phrase, then clustering). Clustering happens per file, so Speaker 1 in Monday's run and Speaker 1 in Tuesday's run are not promised to be the same human. For cross-file identity you need enrolled voice prints, which is a different product.

One counter-example, because I only have two data points on it: the other URL in those runs was a 19.02-second YouTube clip, and it came back byte-identical on both days, with segment boundaries on whole seconds (0, 4, 12, 16). Short clean clips may well be stable. I would not build on the assumption.

Calling it

import requests rows = requests.post( "https://api.apify.com/v2/acts/lergassy~speech-to-text/run-sync-get-dataset-items", params={"token": TOKEN}, json={"urls": ["https://example.com/episode-12.mp3"], "language": "en", "subtitleFormat": "srt", "chunkForRag": True, "chunkSize": 600}, ).json() 

The Actor is Speech to Text. One row per file, no key to set up; the third run in this post cost $0.011 — one billed minute at $0.009 plus $0.002 for the file. Minutes are charged per started minute, so that 19-second YouTube clip billed a full minute, and failed files cost nothing. Direct file links pay no proxy fee; a YouTube, TikTok or Instagram page does, because its audio has to come through a residential exit.

When it is the wrong tool. Live or streaming audio — this is batch, one row per finished file. Word-level timestamps or karaoke-style alignment: segment timings are all the API exposes, and my cue splitter interpolates rather than aligns. Cross-recording speaker identity, per the point above. And if you already run Whisper on your own GPU, nothing here beats that on price; what you are buying is the download, the ffmpeg prep, the splitting of long files, the gap re-check and the retries.

The lesson I keep relearning: when a model returns structure as well as content, the content is the part people test and the structure is the part they build on. Check which of the two your pipeline actually depends on — and if the answer is structure, measure it twice on the same input before you ship.