I work on Koedori at W Lab. It is an iPhone app that records a lecture, a meeting or an interview and turns it into text on the phone itself, with no account, and the audio is never uploaded. The early builds ran Whisper through WhisperKit. The version on the App Store has no Whisper in it at all: it uses Apple's SpeechAnalyzer framework and its SpeechTranscriber module. This post is about why, and what building on SpeechTranscriber is like.
The rule that picks the engine
Koedori makes three promises: transcription is free and needs no account, audio never leaves the phone, and re-transcribing costs nothing, as often as you like. The second one is architecture, not a setting. There is no code path that uploads audio, and the app has no server of its own. So recognition has to run on the device, and it has to be cheap enough to run again on a whim.
Whisper on the phone met that on paper. The trouble was everything around the model.
What went wrong with Whisper on real phones
WhisperKit compiles a model for the Neural Engine the first time it loads it. On the iPhone 14 I measured, the first load of large-v3 turbo after an install took 197 to 205 seconds, every time. A reload within the same install took 4 seconds, and a 10-second clip then transcribed in 0.9 seconds. Fast once warm, minutes from cold. Every Xcode run is an install, so development paid that wait on every build, and a pass started mid-compile showed nothing for minutes, which meant a "preparing" step on screen.
Memory was the next surprise. Without WhisperKit's prewarm, one run logged that it couldn't reach the compiler's helper, fell back to the CPU and peaked at 799 MB. With prewarm: true the same load peaked at 193 MB.
Older hardware was worse. On some older chips the encoder heard nothing when it ran on the CPU, and one test device never finished optimizing Whisper small. And the app had to carry a model: Whisper small, bundled, is about 490 MB, plus code to download the others.
So the app got one engine everywhere: SpeechTranscriber. That cost two things. SpeechTranscriber is new, so older iOS versions were dropped rather than kept on Whisper. And it has about ten languages (English, Japanese, Chinese, Korean, Spanish, French, German, Italian, Portuguese…) against Whisper's hundred. What left with Whisper: the bundled model, the model picker, our download code and the compile wait. iOS now fetches a language's model itself, once, through AssetInventory.
The file pass
Whatever the mode, a note is made by one pass over the finished recording: a SpeechAnalyzer with one SpeechTranscriber module.
SpeechTranscriber( locale: locale, transcriptionOptions: [], reportingOptions: reportingOptions, attributeOptions: [.audioTimeRange] ) .audioTimeRange is the option that matters. A result covers roughly 20 seconds of audio, far too coarse to tap. But every attributed run inside a result carries its own time range (for Japanese, one character), so a line cut from the middle of a result keeps a real start time. That is what lets a tap on any transcript line play the audio from that moment.
The pass hands the file to the analyzer whole (shortened from the app):
let analyzer = SpeechAnalyzer(modules: [module]) let results = module.results let collected = Task { for try await result in results where result.isFinal { /* keep it, report progress */ } } if let end = try await analyzer.analyzeSequence(from: file) { try await analyzer.finalizeAndFinish(through: end) } else { await analyzer.cancelAndFinishNow() } Three details. analyzeSequence(from:) converts the file itself (our test fixture is 22.05 kHz Float32, the engine wants 16 kHz Int16), so there is no reason to feed buffers by hand. Results are read on their own task while the file is read, or the progress bar jumps from 0 to 1. And when no audio reaches the analyzer you still have to finish it, or module.results never ends and that task waits forever.
Is the language on this phone?
Before offering to transcribe, the app asks three questions (condensed):
guard SpeechTranscriber.isAvailable else { return .unavailable } guard let locale = await SpeechTranscriber.supportedLocale( equivalentTo: Locale(identifier: id)) else { return .localeUnsupported } let wanted = locale.identifier(.bcp47) let installed = await SpeechTranscriber.installedLocales .contains { other in other.identifier(.bcp47) == wanted } The obvious API for the last question is AssetInventory.status(forModules:), and it misled me. On a Mac with the Japanese model installed and transcribing, it said .supported, not installed, until the locale had been reserved. Reserving is a side effect, and an app holds only five reserved locales at a time, so a readiness check has no business doing it. installedLocales answers the real question. Compare in BCP-47 too, rather than trusting ja_JP and ja-JP to match.
If the model is missing when a recording stops, the recording is still saved, marked "Waiting to transcribe", and retried whenever the app comes to the front.
Live text is shown, never kept
On the New recording screen, Transcribe is "After stopping" or "While recording". While recording, the audio tap writes the .m4a and also hands each buffer to a second analyzer, through a converter from the microphone's format (typically 48 kHz Float32) into what SpeechAnalyzer.bestAvailableAudioFormat asks for. One converter lives as long as the input format holds, so the resampler's state carries across buffers and nothing drifts at the seams. The tap is a realtime thread with a file to write, so it only copies each buffer into a queue. The engine takes a moment to load, and audio from that moment is held, not dropped.
Two measurements, made on a Mac with the same framework, shaped this mode. With .volatileResults alone, the first words showed 11.7 seconds after they were spoken. Adding .fastResults brought that under a second. But the streamed text was never the file pass's text, even from an identically configured module: over the same 10 minutes, about 1.5 places a minute (around 0.7% of characters) came out differently, at the same places every run, neither side more accurate.
So the live box is a display. The note always comes from the file pass after Stop, in every mode, and the file pass never uses .fastResults, which trades accuracy for speed.
Corrections are a text layer
Speech recognition gets names wrong: a lecturer's surname, a course term, a colleague. Koedori's Dictionary takes "heard as X, should be Y".
Each note stores the engine's text exactly as the engine wrote it, plus a snapshot of the dictionary it was read with. Corrections apply when the note is shown, searched or exported. A new correction makes the snapshot stale, and the difference is the note's count of pending fixes. Fixing them does no recognition at all: it stamps the current snapshot. That is why rewriting earlier notes with a new correction is instant, why new transcripts get the same fix, and why a note whose audio was deleted can still be fixed.
The API offers a second route, AnalysisContext.contextualStrings, and the app passes the dictionary's words there before any audio. I haven't seen it do anything. In a test on a Mac I registered 限界効用 ("marginal utility") and the engine still wrote 限界公用, in the transcript and in its alternatives. It isn't measured on a phone, so the app promises the text layer, not the hint.
Turning results into lines
The engine's own boundaries over-segment, which left a line like "confidence." alone under the sentence it finishes. Phrases under 1.5 seconds apart are merged first, then lines are capped at 200 characters, or 70 for Japanese, Chinese and Korean. That is longer than a subtitle cue on purpose: tapping a line to hear it again is what this screen is for.
Japanese has one more catch: the recognizer emits no 。, so sentences can't be found by punctuation. Where the app needs them, for a note's title, it also cuts after a polite ending (ます, です, ください) unless the next words continue the clause (…ますが, …ますので).
Testing an engine the simulator doesn't have
The simulator has the Speech framework but no engine: isAvailable is false and the language list is empty. Engine tests are gated with .enabled(if:) on a file-scope constant, and the real check runs on a Mac: a script compiles the production files and transcribes a fixture under sandbox-exec with the network denied. "Works offline" is a test, not a line in the listing.
The fixture is synthesized speech. It proves the pipeline runs and says nothing about accuracy, so I don't quote numbers from it.
What it doesn't do
Speaker labels aren't available on the device, so a transcript is lines with times, not names. It doesn't write summaries. It records with the screen locked (a recording over three hours is saved in parts, so nothing is lost), transcribes on the phone, and lets you search, fix and export as plain text, Markdown, SRT or WebVTT.
Koedori is on the App Store for iPhone as “Lecture Recorder: Transcribe”: https://w-lab.app/apps/koedori/

