How to Fix Jargon in Speech-to-Text Without Adding Words Nobody Said

Engineer's view. This is a search spellchecker with a custom dictionary. Add your product names and their misspellings get fixed, but nearby ordinary words get "corrected" too. A transcription prompt works the same way: a listed spelling wins wherever the audio is close to it. You get your jargon right and pay in confident wrong words.

TL;DR. Give a speech-to-text model a short list of your terms and it spells most of them right. It also writes those terms where someone said a word that only sounds alike, so test for both.

Explain Like I'm 5

Imagine a friend writing down everything you say at a noisy party. Before you start, you hand
them a card with the tricky names on it. Now they spell those names right.

But then someone says "Laura." Your friend glances at the card and writes the name from the card
instead.

The card helps your friend choose between words that sound alike. It cannot tell them which word
was actually said. So the card fixes some mistakes and makes a few new ones.

The Problem

You have shipped this bug before, in a system with no model in it.

Your search box had a spellchecker. Users typed your product names and got "no results," so you
added the names to its dictionary. The failed searches stopped. A week later, a support ticket
arrived. Someone searched for "laura" and got pages about your LoRA feature. The dictionary did
what you asked. It pulled every close word toward the words you listed.

A speech-to-text model is that spellchecker, running on sound.

You send it meeting audio, a support call or a video voiceover. It hears "Q drant" and writes
"Q-drant." It hears "K V cache" and writes "cavity." I read 15 AI terms to two OpenAI
transcription models with no help. whisper-1 spelled 14% of them right, and gpt-transcribe spelled
36% right. Nothing raised an error. The transcript just looked slightly wrong, and every search,
summary and caption built on it inherited the mistake. My own video captions hit this last week.
The voice said "halve" and the transcript said "have."

Both models take a hint: a text prompt, or a list of keywords. Add one and most of the jargon
comes out right. Some of it also comes out where nobody said it.

The fix: give the model a short list of the terms this recording is likely to contain, put the
important ones last, and test it on clips that say near-miss words, not only on clips that say the
terms.

The Fix: Keep the Term List Short, Ordered and Tested on Near Misses

What does the prompt actually change?

A transcription model has two halves. The encoder turns audio into features. The decoder
writes the text, one token (a word or a piece of one) at a time, like any language model. It
picks each token by how well it fits the sound and the text written before it.

The prompt goes in front of that text, as if it had been said just before the recording. Whisper's
open-source code marks it with a "previous text" token. It is not an instruction. OpenAI's guide
says Whisper copies a prompt's style and ignores its commands. So a list of spellings does one
thing: it makes those spellings more likely.

Why does it write words nobody said?

Because the boost never checks the audio, it wins any close call. With a 15-term list, whisper-1
spelled 90% of the terms right. On the same run, "Laura will call you back" became "LoRA will call
you back" in 6 of 6 clips. "Cash the check" became "cache the check" in 6 of 6. gpt-transcribe,
given the terms as keywords, wrote "Mixtral" for "mixed trial" in 5 of 6. Sentences that sounded
like no term never changed.

Doesn't a longer list cover more?

Not on Whisper. It keeps only the last 223 tokens of the prompt and drops the rest from the front.
My 75-term list was 277 tokens. With the 15 real terms first, the cut took 14 of them, and accuracy
fell to 19%. The same list with those terms last scored 92%.

What can a list never fix?

A list chooses between spellings the audio already supports. It cannot add an underscore nobody
spoke, so "max tokens" never came out as max_tokens. And it cannot separate two words that sound
the same. "Halve" stayed "have" on whisper-1 in every clip.

What This Means for You

When this matters. You transcribe audio that contains names, product terms or acronyms. That
covers meeting notes, call summaries, video captions, voice agents and search over recordings. It
matters most when code acts on the words: a search index, a ticket field, or a voice agent that
picks a tool from what it heard.

How it affects you. A term list is the cheapest accuracy win you have. In my test it moved
whisper-1 from 14% to 90% on jargon, for no extra cost per call. But it changes two numbers, and
most teams only watch one. A voice agent that hears "cache" for "cash" may call the wrong tool. A
long glossary on Whisper can quietly do nothing, because the terms you cared about were cut first.

What to do about it. Start with no setup. Take ten real recordings and search their
transcripts for your product names. Count the misspellings. That number tells you whether this is
worth an afternoon.

Next, send a short list with each request. Pick the terms this recording is likely to contain,
from its meeting title, its customer or its agenda, rather than your whole glossary. Put the
important terms last.

Then build a small decoy set: clips that say ordinary words close to your terms, such as
"Laura" for LoRA. Run it with and without the list. Ship the list only if term accuracy rises and
the decoys stay clean.

Finally, fix formatting in code after the transcript arrives, never in the prompt.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever calls the API. Send the terms in the field each model reads. whisper-1 takes one
prompt string. gpt-transcribe takes literal terms in keywords, one form field each:

curl https://api.openai.com/v1/audio/transcriptions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -F model=whisper-1 -F language=en -F file=@call.mp3 \ -F "prompt=Kubernetes, pgvector, SGLang, Qdrant, vLLM." curl https://api.openai.com/v1/audio/transcriptions \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -F model=gpt-transcribe -F 'languages[]=en' -F file=@call.mp3 \ -F 'keywords[]=Qdrant' -F 'keywords[]=pgvector' -F 'keywords[]=vLLM'

A keyword containing <, > or a line break makes the API reject the whole request. Strip those
before you send.

Job 2, whoever owns the glossary. Build the list per recording, inside the budget, with the most
important term last. Whisper drops from the front, so the order is the priority:

def build_prompt(terms, budget=223): """terms: most important first. Returns the prompt and the terms that did not fit.""" kept, used = [], 0 for t in terms: cost = est_tokens(t) + 1 # +1 for the comma  if used + cost > budget: break kept.append(t) used += cost return ", ".join(reversed(kept)) + ".", terms[len(kept):]

est_tokens deliberately overestimates. For an exact count, use the GPT-2 tokenizer, which is what
Whisper uses.

Job 3, whoever consumes the transcript. When a wrong term is costly, transcribe twice and diff.
A listed term that appears only in the prompted version is either a fix or an insertion. A person
or a second check decides which:

def suspect_terms(plain, prompted, terms): words = lambda s: set(re.findall(r"[\w-]+", s)) pieces = {p for t in terms for p in re.findall(r"[\w-]+", t)} return sorted((words(prompted) - words(plain)) & pieces)

On my sample, it flagged all five fixes and both insertions. It compares exact case, so "LoRa"
becoming "LoRA" counts. It compares words, so "cash" becoming "cache" is caught even though the
listed term is "KV cache."

Job 4, whoever owns the eval. Score two numbers on every list change: terms spelled right on
clips that say them, and decoy clips with any listed term in them. score() in
code_example.py does both, and it runs on real transcripts from my test.

How you know it worked.

Term accuracy on your term clips should rise sharply. On whisper-1 it went from 14% to 90%. Decoy
insertions should stay at or near zero. Mine were 13 of 72 with the short list, all on decoys
built to sound like a term. If accuracy does not move on Whisper, count your prompt's tokens
first. A list over 223 tokens has lost its front.

When a Term List Is the Wrong Tool

When your vocabulary is huge. A catalog of 5,000 product codes will not fit in 223 tokens, and a
long keyword list raises the insertion risk. Transcribe without the list, then correct the text
against a lookup table, or with a text model given the catalog. OpenAI's guide suggests the second.

When the words are ordinary. Listing common words gives the boost nothing to fix and plenty to
break.

When you need high accuracy on one domain. NVIDIA reports a medical benchmark where a better prompt
cut the word error rate from 30.8% to 23.8%, and fine-tuning cut it to 11.8%. A prompt is the
first step, not the last.

When you chain chunks. Passing the previous chunk's transcript as the prompt carries its mistakes
forward as well as its spellings.

My test has limits. It used three synthetic macOS voices, 15 terms and one run per clip. Real
speakers mumble, and accents move the close calls. Run your own decoy set before you trust my
numbers.

Three questions before you ship a list:

Which of your terms sound like an ordinary word?
Does your prompt fit in the model's budget, with the important terms last?
What happens downstream when a listed term appears that nobody said?

Glossary

decoder — the half of a transcription model that writes text, one token at a time, like a language model.
token — a word or piece of a word; the unit a model reads and writes, and the unit prompt budgets count.
keywords — gpt-transcribe's field for literal terms you expect to hear; hints, not required output.
decoy set — test clips that say ordinary words close to your terms, used to catch insertions.
insertion — a listed term written into a transcript where the speaker said a different word.