feat(stt): Whisper output language is per-transcription — core 1.8.0, speech 0.5.0 - #501
Merged
Conversation
…en|>
`getActiveStt(language: 'de')` now selects Whisper's output language. It
defaults to 'en', which is what the profile hardcoded before, so nothing moves
unless a caller asks.
The capability was already there and nobody could reach it. The shipped
checkpoints are `litert-community/whisper-tiny` / `whisper-base` with the
`openai/whisper-tiny` tokenizer — no `.en` suffix, i.e. the multilingual weights
— while `SttModelProfile.whisper`'s decoder prompt pinned `<|en|>`. Measured on
one German clip, same build, one token apart:
<|en|> " This weather is very beautiful and the sun is shining."
<|de|> " Das Wetter ist heute sehr schön und die Sonne scheint."
The first is the telling one: with `<|en|>` the model does not garble non-English
speech, it TRANSLATES it. Comprehension was never the limit; the token only ever
decided the output language.
Threading it took four signatures and, on the first attempt, silently did
nothing: `language` reached each shell's `createSttModel` and was dropped before
`RuntimeConfig`, so `getActiveStt(language: 'de')` still returned English. That
failure is indistinguishable from working — transcription succeeds, text comes
back, just always English — so it is verified through the public API rather than
by patching the profile, which is how the gap surfaced.
Both packages take a minor: this is new public API on core, and speech's floor
goes to ^1.8.0 because it READS `config.language` while only core 1.8.0 can SET
it. On an older core it would resolve, compile and transcribe in English with the
feature advertised and inert. Core publishes first.
models.md said "Whisper (tiny, English)", which undersold what already shipped.
Corrected, with the 99-language note and the caveat that asking for 'en' on
non-English audio returns a translation rather than an error. Parakeet CTC 0.6B
is genuinely English-only; moonshine and parakeet ignore the parameter.
The change is one optional named parameter threaded through five files (18 lines), and RuntimeConfig.language already existed on main — this release only connects a public field to a public call that were never wired together. SttModelProfile is not exported from the speech barrel, so its loss of const is invisible outside the package. A minor here would not match what this project has shipped as one: 1.7.0 was the whole Hugging Face install path, 1.6.0 a new file type plus engine, 1.5.0 genai_primitives and four groups of facade API. Speech's floor moves to ^1.7.5 rather than staying at ^1.7.4: it reads config.language, and only core 1.7.5 sets it — on 1.7.4 the parameter is dropped before RuntimeConfig and every transcript comes back in English, which is exactly the bug being fixed.
DenisovAV
changed the title
Review found getActiveStt(language:) was a no-op after the first call in a process: all three shells key the STT singleton on the active model's name, so the second call returned the recognizer built for the FIRST language and transcribed into it with no error. The manual macOS check missed it because it made exactly one call on a cold start. Rather than guard the cache, remove what the guard would protect. The decoder seed prompt is already copied on every transcription, so the language token is swapped per call instead of baked at load: SpeechRecognizer.language mutable default, retargeted on reuse transcribe(pcm, language:) per-call override SttCore._promptFor replaces one id in a copy of the seed Nothing is reloaded to change language, so there is no stale state to invalidate. SttModelProfile loses its language parameter and is const again; forType matches TtsModelProfile.forType and no longer silently discards a language for moonshine and parakeet, which now reject it. Validation is in two halves: a shape check at create time, so 'de-DE' fails before an isolate is spawned, and the exact code checked against the tokenizer's own index — harvested at load from the same map the resolver builds. That index is also what stops language: 'translate' resolving cleanly and rewriting the prompt's task slot. Tests: promptForLanguage extracted as a pure function (matching the file's other testable helpers) and covered, plus stt_language_singleton_test.dart modelled on the TTS one. Mutation-checked: dropping the retarget, guarding it on non-null, an off-by-one slot, a silent fallback on an unknown code, and removing language from RuntimeConfig each turn the suite red. Docs: RuntimeConfig.language no longer claims to be TTS-only and spells out that its two consumers use incompatible vocabularies; language documented on getActiveStt, createSttModel, transcribe, speech.md and the speech README; website/docs/speech.md pinned flutter_gemma_speech ^0.4.3 while core moved.
…gumentError's type across the isolate port The unit suite fakes everything below transcribe, so it cannot show that the model actually writes a different language. whisper_language_test.dart runs the real checkpoint. Measured on macOS, one clip, one recognizer: en She had Jedak Su in greasy wash for all year. de Sie hat ja dags zu den Grissie-Waschwarrohrer. default-de Sie hat ja dags zu den Grissie-Waschwarrohrer. The third line is the regression this PR exists for: getActiveStt(language:) on an ALREADY-BUILT recognizer, which previously returned the first language's recognizer and said nothing. identical() holds across it, so the retarget really is a field write and not a reload. That run also caught a defect the fakes could not: every worker error reached the caller as StateError, because an exception object is not sendable and the reply carried only e.toString(). So transcribe(language: 'zz') threw, correctly, but could not be caught as the ArgumentError it is. The reply now carries a badArgument flag and the type is reconstructed — general to the error path, not specific to the language.
Review round two found the fix incomplete in three places and wrong in a fourth, all on paths the first round had already named: - the in-flight-completer guard (mobile and desktop) and the legacy explicit-paths branch returned an existing recognizer without applying the requested language — #500 again, in a window wide enough to hold an isolate spawn and a 51k-entry tokenizer parse; - the retarget branch assigned the language with NO validation, so `getActiveStt(language: 'de')` on moonshine returned normally instead of throwing as documented, and then every later `transcribe()` failed — including calls passing no language. One call broke the recognizer for good. Validation now lives in `LiteRtSpeechRecognizer`'s `language` setter, which every write goes through, and `create()` applies the same rule before spawning the worker so a bad code costs no isolate. Two more from the same round: `badArgument` was `e is ArgumentError`, but `RangeError` and `IndexError` both EXTEND `ArgumentError`, so any bad index in the decode path reached the app as a bad `language`; and rebuilding the error from `toString()` dropped `name`/`invalidValue` and doubled the prefix. Both fixed, both verified against dart:core rather than assumed. The test fixture claimed `<|yue|>` at 50325. The real openai/whisper-tiny tokenizer has `<|yo|>` there and no `<|yue|>` at all; its only 3-letter code is `<|haw|>`. Downloading the file also showed the regex matches exactly 99 entries on its own, so `_nonLanguageCodes` excluded nothing and is gone. Version: 1.8.0, not 1.7.5. The patch call was made before this branch widened the abstract `SpeechRecognizer`, and speech 0.4.3 (`^1.5.6`) co-resolves with a new core — so `pub upgrade flutter_gemma` alone breaks the build in a package the user never touched. It is a loud compile error, not silent wrong output, so a minor plus a CHANGELOG line is the honest label rather than a major. Coverage: the desktop shell had none at all (`defaultFlutterGemmaInstance` returns mobile on the VM), so deleting its `RuntimeConfig` threading left the suite green. It has a file now — with a recorded KNOWN GAP: desktop never reuses its recognizer, because an install names the spec after the basename and the restore path after the full filename, so its cache guard always sees a changed model. That is pre-existing and separate from #500. Mutation-checked: dropping either guard's retarget, swallowing instead of throwing in the setter, and removing desktop's `RuntimeConfig` threading each turn the suite red. macOS re-verified end to end.
DenisovAV
changed the title
CI failed on four warnings my local analyzer did not report — the trap CLAUDE.md names: CI installs `channel: stable` (latest), a dev box is whatever was last pulled, and warnings are fatal there while `--no-fatal-infos` only spares infos. Three were `unnecessary_cast` on `(e as ArgumentError)` in the worker's catch. A `bool` flag leaves `e` an `Object`, so each field read needed a cast, and whether the analyzer calls that cast redundant differs by SDK. Binding a typed local instead removes the casts entirely, so the code no longer depends on which flow analysis is in front of it. The fourth was `SttModelType` left in a `show` clause after the validation that used it moved into the recognizer's setter. Local Flutter upgraded 3.47.2 -> 3.47.3 (Dart 3.13.2 -> 3.13.3) so this stops being discovered by a red CI. Confirmed aligned: local now reports 569 issues, CI reported 573 before the fix, and the difference is exactly these four warnings. The upgrade also churned pubspec.lock and the generated plugin registrants (a new transitive connectivity_plus). Reverted — that is a dependency change, not this release, and it belongs in its own PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the promise made on #500:
getActiveStt(language: 'de')now selects Whisper's output language. Defaults to'en', which is what the profile hardcoded, so nothing moves unless a caller asks.The capability was already shipped and unreachable
The package ships
litert-community/whisper-tiny/whisper-basewith theopenai/whisper-tinytokenizer — no.ensuffix, i.e. the multilingual weights. What pinned it to English was one line inSttModelProfile.whisper's decoder prompt.Measured on one German clip, same build, one token apart:
The first line is the informative one: with
<|en|>the model does not garble non-English speech — it translates it. Comprehension was never the limit; the token only ever decided the output language.The first attempt compiled and did nothing
Threading
languagetouches four signatures. On the first pass it reached each shell'screateSttModeland was dropped beforeRuntimeConfig, sogetActiveStt(language: 'de')still returned English.That failure mode is indistinguishable from success — transcription runs, text comes back, just always English. It surfaced only because the check goes through the public API rather than patching the profile, which is how the gap was found in the first place.
Versions
core 1.8.0 / speech 0.5.0. This started as a patch, and the patch call was
right for what it described: widening
createSttModelonFlutterGemmaPlugin,a
PlatformInterfacewith a private token that nothing outside the repo cansubclass. That is the same shape as
createTtsModelgaininglanguagein core1.5.2, which shipped as a patch.
The fix commits changed the situation. Making the language per-transcription
widened the abstract
SpeechRecognizer—transcribegainedlanguage:, andthe type gained a
languagefield — andflutter_gemma_speechextends thattype. Published speech 0.4.3 declares
flutter_gemma: ^1.5.6, so itco-resolves with the new core:
The build breaks in a package the user never touched. It is a loud compile
error rather than silent wrong output, and it only bites someone upgrading core
on its own or pinning speech exactly —
pub getand a fullpub upgradeareboth fine. So a minor plus an explicit CHANGELOG line is the honest label; a
major would only be about excluding old speech, which is not worth it for this.
flutter_gemma_speech's floor moves to^1.8.0: it readsconfig.languageandoverrides the widened members, and only core 1.8.0 has them.
Publish order: core 1.8.0 first, then speech 0.5.0.
Docs
models.mdsaid "Whisper (tiny, English)", which undersold what already shipped. Now states the 99-language support, thegetActiveStt(language:)call, and the caveat that asking for'en'on non-English audio returns a translation rather than an error.Parakeet CTC 0.6B is genuinely English-only — that part of the answer on #500 stands. Moonshine and Parakeet ignore the parameter.
Testing
getActiveStt(language: 'de')→ verbatim German, umlaut included