Multilingual offline STT

[ihorkozar]

Hi! I’m considering  flutter_gemma_speech  for an on-device speech-to-text feature in a Flutter app.

The documentation lists the following STT support:
•  moonshine-tiny  — raw 16 kHz PCM
• Whisper  (tiny, English)  — log-mel
• Parakeet  (CTC)  — log-mel

Could you please clarify which exact Parakeet CTC model(s) are currently supported by the package?

In particular, I would like to know:
• Is the supported Parakeet CTC model multilingual, or English-only?
• Which languages are supported?
• Is there an official compatible LiteRT/TFLite model URL and tokenizer/vocabulary file you recommend using?

I understand that Moonshine Tiny and the currently listed Whisper Tiny profile are English-focused, so multilingual offline STT support is important for my use case.

Thank you!

[DenisovAV]

Hi @ihorkozar — I measured this rather than answering from the docs, and the answer is better than the docs suggest.

Parakeet, directly

One model is wired: nvidia/parakeet-ctc-0.6b, and it is English-only. There is no multilingual Parakeet in the package.

model      https://huggingface.co/litert-community/parakeet-ctc-0.6b/resolve/main/parakeet_ctc_0.6b_5s_f32.tflite
tokenizer  https://huggingface.co/nvidia/parakeet-ctc-0.6b/resolve/main/tokenizer.json

But multilingual STT already works — via Whisper

The docs say "Whisper (tiny, English)". That is wrong, and the reason is a single line rather than a missing capability.

The package ships litert-community/whisper-tiny / whisper-base with the openai/whisper-tiny tokenizer — no .en suffix, i.e. the multilingual weights (99 languages). What pins it to English is one hardcoded token in the decoder prompt:

decoderPromptTokens = const [
  SttTokenRef.name('<|startoftranscript|>'),
  SttTokenRef.name('<|en|>'),          // <- this
  SttTokenRef.name('<|transcribe|>'),
  SttTokenRef.name('<|notimestamps|>'),
]

I generated a German clip ("Das Wetter ist heute sehr schön und die Sonne scheint", 16 kHz mono) and ran it through whisper-tiny twice — same audio, same build, one token different:

<|en|>   " This weather is very beautiful and the sun is shining."
<|de|>   " Das Wetter ist heute sehr schön und die Sonne scheint."

The <|de|> run is verbatim, umlaut included. Note the first one too: with <|en|> the model did not garble the audio — it understood the German and translated it. So the multilingual capability is fully present and working; the token only decides the output language.

What this means for you

Offline multilingual STT is a hardcoded string away, not a model swap. The language will be exposed as a parameter in the next release, so you will be able to pick it at install/transcribe time instead of taking English. Whisper's language set is the standard 99.

If you need it before that lands, the same override is reachable by pointing SttModelProfile's whisper decoderPromptTokens at your language token — that is exactly what the measurement above did.

I will fix the "Whisper (tiny, English)" wording in the docs as part of the same change; it undersold what is already there.

[DenisovAV]

This shipped — flutter_gemma 1.8.0 + flutter_gemma_speech 0.5.0, both on pub.dev now.

dependencies:
  flutter_gemma: ^1.8.0
  flutter_gemma_speech: ^0.5.0

The language turned out to belong on the transcription rather than on the model, so you get both a default and a per-call override:

// Default for every transcription on this recognizer.
final stt = await FlutterGemma.getActiveStt(language: 'de');
final german = await stt.transcribe(germanPcm);

// One call in another language — same recognizer, nothing reloaded.
final french = await stt.transcribe(frenchPcm, language: 'fr');

Both are cheap by construction: the language is one token in the decoder's seed prompt, and that prompt is rebuilt per transcription. Switching never reloads the model and never invalidates a recognizer you are already holding, so a language picker in the UI costs nothing.

Codes are Whisper's own without the delimiters — 'en', 'de', 'uk', any of the 99 — and it still defaults to 'en'.

Two things worth knowing:

• An unusable value throws instead of being ignored. A malformed code ('de-DE', 'german') is rejected before the model is even loaded. A well-formed code the installed checkpoint does not have is rejected against that checkpoint's own tokenizer, with the valid set named in the error — so it cannot go stale against a different Whisper export.
• moonshine and Parakeet reject a language rather than silently transcribing in English. Neither has a language token in its decoder prompt, so asking for one is an ArgumentError, not a no-op.

The measurement from my previous comment is now a regression test, and the models.md / speech.md wording that said "Whisper (tiny, English)" is corrected — it undersold what was already there.

Thanks for the report; it was a one-token fix hiding behind a documentation error.

[ihorkozar]

@DenisovAV Thank you for the incredibly fast update and for implementing this so thoughtfully. It’s great that multilingual Whisper transcription is now supported both as a default setting and per transcription call, without requiring a model reload.