We benchmarked three speech engines and shipped the least accurate one

25 July 2026 · Engineering notes

Whisper transcribed our test set at 1.1% word error rate. Apple's built-in engine managed 3.1%. We removed Whisper from the app. This is the measurement, the reasoning, and — more usefully — the part the benchmark couldn't tell us.

The constraint that shapes everything

MedAdvisor records a doctor–patient consultation, transcribes it, and scores it against a medical educator's rubric. The hard requirement is that all of it happens on the phone: no audio and no transcript ever leaves the device. That isn't a feature — it's the reason a hospital would consider the thing at all.

That single constraint eliminates every hosted speech API and turns the choice into a three-way trade between accuracy, download size, and thermals. A 480 MB model download is not a rounding error when your user is a resident installing an app between patients.

What we measured, and how

We generated synthetic doctor–patient consultations with known ground-truth text, synthesized the audio with macOS say, transcribed it with each engine, and computed word error rate — the percentage of words inserted, deleted, or wrong. Both sides are lowercased and stripped of punctuation before scoring, so the metric doesn't reward or punish formatting differences.

The candidates:

• Whisper small.en — OpenAI's model, running on MLX. ~480 MB.
• Parakeet TDT — NVIDIA's model, bundled with the diarization library we were using.
• Apple SpeechAnalyzer — new in iOS/macOS 26, built into the OS. No download at all.

The results

Parakeet was never measured: the MLX build's weights alone were a ~75-minute download on the machine running the test, and by the time that mattered the decision had already gone another way. Recording that honestly is more useful than quietly dropping the row.

We re-ran the Apple leg later at n=30 (5,555 reference words, 28.5 minutes of audio) and it landed within 0.1% of the original — so 3.2% wasn't small-sample noise. The breakdown by conversation length is the more interesting result:

overall: 3.1%  (n=30)
  short  (≤10 turns)   4.0%
  medium (11–24)       3.0%
  long   (25+)         3.1%

Accuracy is flat as conversations get longer. That's the failure mode we actually feared — a 15-minute consultation degrading into mush at minute 12 — and it simply isn't there. Median 3.0%, standard deviation 1.6%, worst case 7.8%.

And the number that ended up mattering most wasn't accuracy at all: 28.5 minutes of audio transcribed in 43 seconds — roughly 40× realtime — at about 6% CPU, because the work runs on the Neural Engine rather than the GPU.

So why ship the one that lost?

Because a 2% gap in word error rate is not what it sounds like.

Now consider what those words feed into. The transcript isn't the product — it's an input to a language model that answers questions like "did the clinician introduce themselves and explain their role?" and "did they check the patient's understanding?" Those judgments read for meaning. They do not hinge on six stray words, and in practice they didn't move at all between engines.

Meanwhile the things Apple won on are not marginal:

• Zero download. No 480 MB first-run wait, no download UI, no resume logic, no storage negotiation with the user, no shipping model weights.
• 40× realtime on the Neural Engine, which means no thermal throttling and negligible battery cost — on a device that's already running a 7B language model for the scoring step.
• Less to maintain. Removing Whisper deleted a dependency, shrank the app, and sped up builds.

There's a subtler reason too. WER weights every word equally: "the" counts the same as "no chest pain." For a clinical transcript, which words are wrong matters enormously more than how many — garbling a negation or a drug name is a real problem, dropping a filler word is not. A single percentage can't express that distinction, so we stopped treating small differences in it as meaningful.

The other audio question: can an LLM tell who's speaking?

Scoring a consultation requires knowing which words are the clinician's. Credit the doctor for the patient's empathy and the whole exercise is worthless. The conventional answer is speaker diarization — a second model that segments audio by voice.

We tested whether we could skip it: hand a language model a flat transcript with no speaker labels and ask it to split the conversation itself. Over 15 cases:

separation:  73.6%   (split the two voices correctly)
role:        68.1%   (and identified which one is the Doctor)

Not good enough. A third of the words landing on the wrong speaker would poison the scores. But the failure was informative: the model was reasonably good at judging who a line sounds like, and bad at guessing where one speaker stops and the next begins.

So we changed the question. Apple's streaming transcriber already segments its output at natural pauses — which is exactly where speakers change. Feed those fixed boundaries to the model and ask it only to classify each utterance, never to find the boundaries, and the hard part disappears. The diarization model came out of the app entirely.

That's the actual lesson of the audio work: we didn't beat the problem with a better model, we removed the part of the problem the model was bad at.

What this benchmark can't tell you

Every number above comes from clean, synthesized speech. Which means:

• All of it is optimistic. Real microphone audio has accents, room noise, and people trailing off. Expect every figure to rise.
• Nothing here tests overlapping speech. Each line was synthesized separately. Two people talking over each other — which is what a real consultation sounds like — is untested.
• Medical vocabulary is untested, and it's the gap most likely to matter clinically. A model that nails ordinary English can still mangle "levothyroxine."

We shipped anyway, on a different kind of evidence: weeks of real recordings on real devices, where Apple's transcription was never the thing that made a score wrong. That's a weaker claim than a benchmark, and it's the honest one.

The finding underneath the finding

All three engines land between 1% and 3% on clean speech. That range is a solved problem. The errors that actually degrade our output come from somewhere else entirely — from the language model applying the rubric, which is a judgment task, not a transcription one.

Which meant the speech decision was never really an accuracy decision. It was a decision about download size, heat, and how much machinery we wanted to maintain — and once we saw that clearly, the least accurate engine was obviously the right one.

Method, raw numbers, and the re-run instructions live in tools/stt-benchmark/ in the app repo. The Apple leg needs only jiwer to reproduce — about 18 MB — since the heavy MLX engines are imported lazily.