From baeb0100874427767270cf72feb3ee50f32249ab Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Thu, 10 Sep 2026 17:11:33 +0200 Subject: [PATCH 01/21] feat: ship agent skills for consumers of the package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Dart team's `skills` CLI (labs.dart.dev, 1.0.1) lets a package carry instruction files that a user's coding assistant installs with `dart run skills@ get`. Three skills, aimed at the mistakes that actually generate issues here: flutter-gemma-setup core registers no engine; installModel defaults fileType to `task`, so a `.litertlm` file is routed to MediaPipe unless declared flutter-gemma-inference maxTokens is the CONTEXT WINDOW, not the reply length (#318 — below 1024 it crashes `.litertlm`, it does not truncate); use maxOutputTokens to cap output; Message.isUser defaults to false and a missing `isUser: true` returns an empty string with no error flutter-gemma-speech-stt 16 kHz mono PCM in; the Whisper output language is a property of a transcription, not of the loaded model Our own `.claude/skills/` (release, build-native, review-pr, ci-status, upgrade-genkit) stay where they are — those are maintainer tools, and shipping instructions for rebuilding our dylibs to every consumer would be noise. The distribution half is the part that would have failed silently. `packages/flutter_gemma/.pubignore` carries a blanket `**/*.md` with only README and CHANGELOG exempted, so every SKILL.md was stripped from the archive. `dart run skills@ get` resolves the package on disk from package_config.json and reads `skills/` from there, so the skills would have reached nobody, with no error at any point. Measured before and after: `dart pub publish --dry-run | grep -c SKILL.md` was 0 for core and 1 for speech (which has no such rule); it is 2 and 1 now. Verified end to end, not by reading the spec: ran `dart run skills@ get --all --agent claude` from the example app and all three installed, then `skills@ remove --all` to clean up. Versions bumped because SKILL.md ships inside the archive — without a release the files exist only in the repo. Both READMEs now say the command exists, since a mechanism nobody knows about is not a mechanism. --- CLAUDE.md | 2 +- packages/flutter_gemma/.pubignore | 6 + packages/flutter_gemma/CHANGELOG.md | 3 + packages/flutter_gemma/README.md | 12 ++ packages/flutter_gemma/example/pubspec.lock | 4 +- .../flutter_gemma/ios/flutter_gemma.podspec | 2 +- .../flutter_gemma/macos/flutter_gemma.podspec | 2 +- packages/flutter_gemma/pubspec.yaml | 2 +- .../skills/flutter-gemma-inference/SKILL.md | 198 ++++++++++++++++++ .../skills/flutter-gemma-setup/SKILL.md | 146 +++++++++++++ packages/flutter_gemma_speech/CHANGELOG.md | 3 + packages/flutter_gemma_speech/README.md | 8 + .../flutter_gemma_speech/example/pubspec.lock | 4 +- packages/flutter_gemma_speech/pubspec.yaml | 2 +- .../skills/flutter-gemma-speech-stt/SKILL.md | 131 ++++++++++++ 15 files changed, 516 insertions(+), 9 deletions(-) create mode 100644 packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md create mode 100644 packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md diff --git a/CLAUDE.md b/CLAUDE.md index 1bf5cacd..ccff3291 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -150,7 +150,7 @@ Core has NO pigeon (dropped at the 1.0 cut; its value types are hand-written in - **LiteRT-LM**: native libs from `native-v0.16.0` GitHub Release (LiteRT-LM pin `924e79c9`, LiteRT pin `0ff28117`). Android tarball bundles the Qualcomm QNN dispatch stack and Windows tarball bundles Intel NPU dispatch (`LiteRtDispatch.dll` + OpenVino runtime + TBB) for `PreferredBackend.npu` (Qualcomm Snapdragon / Intel LunarLake/PantherLake) — both dispatch libs are **rebuilt from the pin every release**; carrying them forward is what silently broke NPU on both platforms (see the `build-native` skill). v0.16.0: fixes the Android OpenCL per-turn memory leak (LiteRT-LM #2699, #348/#402); v0.15.0 **broke the stream-callback ABI** (4-arg → 2-arg chunk object) with no compat path, handled by a runtime probe in `stream_proxy.c`. Windows discrete GPU works again — the crash was our own dead `litert_link_capi_so` Bazel define, not an upstream regression (#2957 retracted). - **sqlite-vec**: `flutter_gemma_rag_sqlite` fetches the per-platform `vec0` loadable from the `native-sqlite-vec-v` GitHub Release (`sqlite-vec-.tar.gz` + `checksums_sqlite_vec.txt`), SHA256-verified by its `hook/build.dart`. `` names the **upstream sqlite-vec release** the bytes were built from; a letter suffix (`0.1.9-a`) is only for RE-releasing changed bytes under an already-published number. The loadables are NOT committed — `native/sqlite_vec/prebuilt/` is a maintainer override produced by `build_local.sh`, gitignored and `.pubignore`d. - **large_file_handler**: `^0.5.0` (core dep; 0.5.0 declares all 6 platforms — needed for pana platform support + the dart2wasm-clean web graph) -- **Current Version**: core `flutter_gemma` `1.8.0`, `flutter_gemma_rag_sqlite` `1.3.1`, `flutter_gemma_rag_qdrant` `1.3.0`; `flutter_gemma_litertlm` `1.6.3`, `flutter_gemma_mediapipe` `1.0.5`, `flutter_gemma_embeddings` `2.1.1`, `flutter_gemma_speech` `0.5.0`; `flutter_gemma_agent` `0.2.5`, `flutter_gemma_builtin_ai` `0.2.1`, `flutter_gemma_onnx` `0.3.3`; `genkit_flutter_gemma` `0.6.1`, `genkit_hybrid` `0.2.1` +- **Current Version**: core `flutter_gemma` `1.8.1`, `flutter_gemma_rag_sqlite` `1.3.1`, `flutter_gemma_rag_qdrant` `1.3.0`; `flutter_gemma_litertlm` `1.6.3`, `flutter_gemma_mediapipe` `1.0.5`, `flutter_gemma_embeddings` `2.1.1`, `flutter_gemma_speech` `0.5.1`; `flutter_gemma_agent` `0.2.5`, `flutter_gemma_builtin_ai` `0.2.1`, `flutter_gemma_onnx` `0.3.3`; `genkit_flutter_gemma` `0.6.1`, `genkit_hybrid` `0.2.1` - **0.15.2**: embedding unified on LiteRT C API via Dart FFI on all native platforms (Android + iOS + Desktop). Drops `localagents-rag` JVM dep on Android and the separate TFLite C 0.12.7 tarball on Desktop; `TensorFlowLiteC` pod no longer needed on iOS. Single source of truth for `TaskType.prefix` in Dart, fixes cross-platform embedding drift (#264). ## Platform-Specific Setup diff --git a/packages/flutter_gemma/.pubignore b/packages/flutter_gemma/.pubignore index fa78459e..c231dcb8 100644 --- a/packages/flutter_gemma/.pubignore +++ b/packages/flutter_gemma/.pubignore @@ -8,6 +8,12 @@ test_reports/ **/*.md !README.md !CHANGELOG.md +# Agent skills MUST ship inside the archive — `dart run skills@ get` resolves +# the package on disk from package_config.json and reads `skills/` from there. +# Without this exception the blanket `**/*.md` above silently strips every +# SKILL.md and the skills reach nobody, with no error anywhere. +!skills/ +!skills/** tools/ # Build artifacts diff --git a/packages/flutter_gemma/CHANGELOG.md b/packages/flutter_gemma/CHANGELOG.md index 05a6c659..f575f58d 100644 --- a/packages/flutter_gemma/CHANGELOG.md +++ b/packages/flutter_gemma/CHANGELOG.md @@ -1,3 +1,6 @@ +## 1.8.1 +- Ship agent skills — `dart run skills@ get` teaches your AI assistant this package. + ## 1.8.0 - Whisper output language on `getActiveStt` and `transcribe` (#500). - **Breaking for custom `SpeechRecognizer` implementations**: `transcribe` gained `language:` and the type gained a `language` field. diff --git a/packages/flutter_gemma/README.md b/packages/flutter_gemma/README.md index 8f69127b..d3a42166 100644 --- a/packages/flutter_gemma/README.md +++ b/packages/flutter_gemma/README.md @@ -56,6 +56,18 @@ There is an example of using: - **🔐 Typed Download Errors:** Catch the public `DownloadException` sealed type (401/403/404/429/5xx) for gated HuggingFace models instead of substring-matching error strings - **💾 Web Persistent Caching:** Models persist across browser restarts — Cache API for models <2GB, OPFS streaming for large ones (>2GB, e.g. Gemma 4 E4B) — no re-download on reload (Web only) +## Teach your AI assistant this package + +`flutter_gemma` ships [agent skills](https://dart.dev/blog/skills-cli-1-0-bundle-and-distribute-ai-agent-skills-for-your-packages) — short instruction files your coding assistant reads so it uses this API correctly the first time: + +```bash +dart run skills@ get +``` + +That scans your dependencies, offers the skills each one bundles, and installs them where your agent looks (Claude Code, Codex, Cursor, Antigravity, Cline, Copilot, OpenCode are all supported). + +What they cover: registering an engine (core ships none), routing by the declared `ModelFileType` rather than the filename, and the two defaults that fail quietly — `maxTokens` is the context window and not the reply length, and `Message.isUser` defaults to `false`. + ## What's new in 1.6.4 - 📱 **iOS deployment floor lowered to 15.0** — core, built-in AI and embeddings build from iOS 15.0 (only `flutter_gemma_mediapipe` still needs 16.0). Every OS-26-only Foundation Models call is `#available`-guarded ([#441](https://github.com/DenisovAV/flutter_gemma/issues/441)). diff --git a/packages/flutter_gemma/example/pubspec.lock b/packages/flutter_gemma/example/pubspec.lock index f061b7d1..04372d6e 100644 --- a/packages/flutter_gemma/example/pubspec.lock +++ b/packages/flutter_gemma/example/pubspec.lock @@ -241,7 +241,7 @@ packages: path: ".." relative: true source: path - version: "1.8.0" + version: "1.8.1" flutter_gemma_agent: dependency: "direct main" description: @@ -304,7 +304,7 @@ packages: path: "../../flutter_gemma_speech" relative: true source: path - version: "0.5.0" + version: "0.5.1" flutter_inappwebview: dependency: transitive description: diff --git a/packages/flutter_gemma/ios/flutter_gemma.podspec b/packages/flutter_gemma/ios/flutter_gemma.podspec index 727e85b2..01dfa64d 100644 --- a/packages/flutter_gemma/ios/flutter_gemma.podspec +++ b/packages/flutter_gemma/ios/flutter_gemma.podspec @@ -4,7 +4,7 @@ # Pod::Spec.new do |s| s.name = 'flutter_gemma' - s.version = '1.8.0' + s.version = '1.8.1' s.summary = 'Flutter plugin for running Gemma and other LLMs locally on iOS.' s.description = <<-DESC Core runtime for running Gemma 4, Gemma3n, Gemma 3, FastVLM, Qwen3, diff --git a/packages/flutter_gemma/macos/flutter_gemma.podspec b/packages/flutter_gemma/macos/flutter_gemma.podspec index 77645712..a745e660 100644 --- a/packages/flutter_gemma/macos/flutter_gemma.podspec +++ b/packages/flutter_gemma/macos/flutter_gemma.podspec @@ -4,7 +4,7 @@ # Pod::Spec.new do |s| s.name = 'flutter_gemma' - s.version = '1.8.0' + s.version = '1.8.1' s.summary = 'Flutter Gemma - Run Gemma AI models locally on desktop' s.description = <<-DESC Flutter plugin for running Gemma AI models locally on macOS using LiteRT-LM. diff --git a/packages/flutter_gemma/pubspec.yaml b/packages/flutter_gemma/pubspec.yaml index 26590c7f..304ad196 100644 --- a/packages/flutter_gemma/pubspec.yaml +++ b/packages/flutter_gemma/pubspec.yaml @@ -1,6 +1,6 @@ name: flutter_gemma description: "Run Gemma and other LLMs on-device in Flutter (Android, iOS, Web, Desktop). Multimodal vision/audio, function calling, thinking mode, GPU, embeddings, RAG." -version: 1.8.0 +version: 1.8.1 resolution: workspace homepage: https://fluttergemma.dev repository: https://github.com/DenisovAV/flutter_gemma diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md new file mode 100644 index 00000000..9eacebc3 --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md @@ -0,0 +1,198 @@ +--- +name: flutter-gemma-inference +description: Use when generating text with flutter_gemma — sessions, chats, streaming, tools, or anything involving maxTokens. maxTokens is the context window and NOT the reply length; setting it low crashes .litertlm models, and Message.isUser defaults to false. +--- + +# Generating with flutter_gemma + +Three defaults in this API produce wrong behaviour rather than errors. They +account for most of the issues filed against the package. + +## maxTokens is the CONTEXT WINDOW, not the reply length + +`maxTokens` on `getActiveModel` / `createModel` is the whole KV-cache budget: +system prompt + history + the current message + everything generated. It is not +"how long the answer may be". + +Setting it small to get a short answer is the single most common mistake, and on +`.litertlm` it does not merely truncate — it crashes. Every supported +`.litertlm` model bakes `kv_cache_max_len = 1024`; below that the native +KV-cache resize underflows and tensor allocation fails at generation time with a +message that names an internal executor file and nothing else. + +```dart +// WRONG — meant "a 100-token reply", actually a 100-token context. +final model = await FlutterGemma.getActiveModel(maxTokens: 100); + +// RIGHT — roomy context, capped output. +final model = await FlutterGemma.getActiveModel(maxTokens: 1024); +final session = await model.createSession(maxOutputTokens: 100); +``` + +Measured on a Pixel 8a (CPU): 100 / 256 / 512 crash; 1024 and 4096 work. The +litertlm engine now clamps values below 1024 upward and logs a warning, but do +not rely on that — say what you mean. + +`maxOutputTokens` is `.litertlm` only. MediaPipe `.task` has no session-level +output cap and logs that the value was ignored. + +## Message.isUser defaults to false + +```dart +// WRONG — silently treated as an assistant message, so the model has +// nothing to answer and returns an empty string. +const Message(text: 'Hello') + +// RIGHT +const Message(text: 'Hello', isUser: true) +``` + +There is no error. The response is just empty. Always pass `isUser` +explicitly. + +## Always close sessions and models + +Both hold native resources — an isolate, a compiled model, GPU buffers. Leaking +them exhausts memory and, on some backends, wedges the next load. + +```dart +final session = await model.createSession(); +try { + // … +} finally { + await session.close(); +} +await model.close(); +``` + +## One-shot generation + +```dart +final session = await model.createSession( + temperature: 0.8, + randomSeed: 1, + topK: 40, + maxOutputTokens: 256, +); +try { + await session.addQueryChunk( + const Message(text: 'Explain isolates in one paragraph.', isUser: true), + ); + final answer = await session.getResponse(); +} finally { + await session.close(); +} +``` + +Streaming, same session shape: + +```dart +await for (final chunk in session.getResponseAsync()) { + stdout.write(chunk); +} +``` + +## Multi-turn conversation + +`InferenceChat` keeps the history and applies the model's chat template. + +```dart +final chat = await model.createChat( + tokenBuffer: 256, + maxOutputTokens: 512, +); +try { + await chat.addQueryChunk(const Message(text: 'Hi', isUser: true)); + final reply = await chat.generateChatResponse(); +} finally { + await chat.close(); +} +``` + +`generateChatResponse()` returns a `ModelResponse`, not a `String`. Switch on +it — with tools enabled it may be a function call: + +```dart +switch (response) { + case TextResponse(:final token): // plain text + case FunctionCallResponse(:final name, :final args): // one tool call + case ParallelFunctionCallResponse(:final calls): // several at once +} +``` + +## Function calling + +Two arguments beyond `tools` are load-bearing. Without `supportsFunctionCalls` +the tools are dropped with a warning; without `modelType` the correct call +format cannot be derived. + +```dart +final chat = await model.createChat( + tools: [ + const Tool( + name: 'change_color', + description: 'Change the UI background colour.', + parameters: { + 'type': 'object', + 'properties': { + 'color': {'type': 'string', 'description': 'A colour name.'}, + }, + 'required': ['color'], + }, + ), + ], + supportsFunctionCalls: true, + modelType: ModelType.gemma4, +); + +final response = await chat.generateChatResponse(); +if (response is FunctionCallResponse) { + final result = await runTool(response.name, response.args); + await chat.addQueryChunk( + Message.toolResponse(toolName: response.name, response: result), + ); + final followUp = await chat.generateChatResponse(); +} +``` + +Not every model supports tools — check the package README's support table before +enabling them. + +## Thinking models + +Qwen3 and DeepSeek R1 emit reasoning in `` tags. Pass `isThinking` so the +tags are handled rather than shown to the user: + +```dart +final chat = await model.createChat(isThinking: true, modelType: ModelType.qwen3); +``` + +With `isThinking: false` on Qwen3 the tags are stripped automatically. + +## Vision and audio + +Declare support at model creation, then attach bytes to a message: + +```dart +final model = await FlutterGemma.getActiveModel( + maxTokens: 4096, + supportImage: true, +); +await session.addQueryChunk( + Message(text: 'What is in this photo?', isUser: true, imageBytes: bytes), +); +``` + +Multimodal models need a much larger context than text-only ones — an image is +worth hundreds of tokens. Start at 4096. + +## Sessions are not free + +Model loading is expensive; session creation is cheap by comparison. Load the +model once and keep it, create and close a session per interaction. Do not call +`getActiveModel` per message. + +`getActiveModel` returns a process-wide singleton. Calling it again with +different runtime arguments rebuilds it and closes the previous instance — so a +handle you are still holding becomes unusable. Decide the runtime configuration +once. diff --git a/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md new file mode 100644 index 00000000..a7fa5eed --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md @@ -0,0 +1,146 @@ +--- +name: flutter-gemma-setup +description: Use when adding flutter_gemma to a Flutter app, calling FlutterGemma.initialize, or installing a model — core registers no inference engine by default and routes by the DECLARED file type, so both are easy to get silently wrong. +--- + +# Setting up flutter_gemma + +Two decisions here fail quietly rather than loudly. Get them right first. + +## 1. Core ships no engine — you must register one + +`flutter_gemma` is the contracts, the registry and the platform shells. It +contains no inference runtime. Adding only `flutter_gemma` to `pubspec.yaml` +compiles fine and throws on the first `getActiveModel()`. + +Pick the engine package for the model format you actually have: + +| Package | Handles | Platforms | +| --- | --- | --- | +| `flutter_gemma_litertlm` | `.litertlm` | Android, iOS, macOS, Windows, Linux, web | +| `flutter_gemma_mediapipe` | `.task`, `.bin` | Android, iOS, web | +| `flutter_gemma_builtin_ai` | the OS model (Gemini Nano, Apple Foundation Models) | Android 26+, iOS, macOS | +| `flutter_gemma_onnx` | ONNX / ORT-GenAI | macOS arm64, Linux x64, Windows x64, Android arm64, iOS arm64 | + +Register the providers before anything else runs: + +```dart +import 'package:flutter_gemma/flutter_gemma.dart'; +import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; + +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], +); +``` + +Every capability is opt-in the same way and defaults to an empty list: + +```dart +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], // text / vision / audio + embeddingBackends: [LiteRtEmbeddingBackend()], // flutter_gemma_embeddings + sttBackends: [LiteRtSttBackend()], // flutter_gemma_speech + ttsBackends: [LiteRtTtsBackend()], // flutter_gemma_speech + huggingFaceToken: 'hf_…', // only for gated repos +); +``` + +If a list is empty, the corresponding first call throws a `StateError` naming +the package to add. That message is the intended diagnostic — read it rather +than guessing. + +## 2. The engine is chosen by the DECLARED file type, never the filename + +`installModel` defaults to `ModelFileType.task`. A `.litertlm` file installed +without declaring its type is routed to MediaPipe, which cannot read it. + +```dart +// WRONG — the name says .litertlm, the declaration says .task, +// and the declaration is what routes it. +await FlutterGemma.installModel(modelType: ModelType.gemma4) + .fromNetwork('https://example.com/model.litertlm') + .install(); + +// RIGHT +await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, +).fromNetwork('https://example.com/model.litertlm').install(); +``` + +`modelType` is a separate axis and drives the chat template and capabilities +(`ModelType.gemma4`, `.gemma3`, `.qwen3`, `.deepSeek`, `.general`, …). Getting +it wrong does not fail loudly either — it produces a model that generates, but +with the wrong prompt format. + +## Installing from Hugging Face + +When the repo publishes a deployment manifest, one call resolves the variant, +the revision and the runtime defaults: + +```dart +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], + huggingFaceResolvers: [LitertlmManifestResolver()], +); + +final install = await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.litertlm, +).fromHuggingFace('litert-community/LFM2.5-230M').install(); + +// The manifest's own runtime defaults — pass them through rather than +// inventing values. +final model = await FlutterGemma.getActiveModel(defaults: install.runtime); +``` + +Without a registered resolver `fromHuggingFace` throws a clear error. `defaults` +carries what the model publisher tested; an explicit argument you pass alongside +it always wins. + +## Other model sources + +```dart +.fromNetwork(url, token: hfToken) // download, resumable +.fromAsset('assets/model.litertlm') // bundled in the app +.fromFile(File(path)) // already on disk +``` + +## Platform notes that bite + +- **Android + `.litertlm`** needs `minSdk 30`. The native library uses + API-30-only Bionic symbols; on 29 it fails at `dlopen`. +- **`flutter_gemma_builtin_ai`** needs `minSdk 26`, or the manifest merger + fails. +- **iOS** builds from 15.0, except with `flutter_gemma_mediapipe`, which + requires 16.0. +- **Web** is GPU-only and needs the CDN script tags plus the runtime JS files + copied into the app's `web/` directory. See the package README. +- **iOS Simulator** cannot run GPU inference — Metal there has a 256 MB + single-allocation cap and model weights exceed it. Use CPU or a real device. + +## Verifying setup + +```dart +await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); +await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, +).fromNetwork(url).install(); + +final model = await FlutterGemma.getActiveModel(maxTokens: 1024); +final session = await model.createSession(); +try { + await session.addQueryChunk( + const Message(text: 'Say hello.', isUser: true), + ); + print(await session.getResponse()); +} finally { + await session.close(); +} +await model.close(); +``` + +If that runs, the engine is registered, the file type routed correctly and the +model loaded. See `flutter-gemma-inference` for what to do next, and why +`maxTokens: 1024` rather than something smaller. diff --git a/packages/flutter_gemma_speech/CHANGELOG.md b/packages/flutter_gemma_speech/CHANGELOG.md index b32cdaf8..9e62a572 100644 --- a/packages/flutter_gemma_speech/CHANGELOG.md +++ b/packages/flutter_gemma_speech/CHANGELOG.md @@ -1,3 +1,6 @@ +## 0.5.1 +- Ship an agent skill for speech-to-text (`dart run skills@ get`). + ## 0.5.0 - Whisper: output language is a parameter, no longer hardcoded English (#500). diff --git a/packages/flutter_gemma_speech/README.md b/packages/flutter_gemma_speech/README.md index 4191a81d..bf398e5e 100644 --- a/packages/flutter_gemma_speech/README.md +++ b/packages/flutter_gemma_speech/README.md @@ -20,6 +20,14 @@ native bundle and exposes the LiteRt interpreter FFI (`LiteRtBindings`) used her Both backends are pure factories (`canHandle` always `true`) — the *model* is selected per-install via `SttModelType` / `TtsModelType`, not the backend. +## Teach your AI assistant this package + +```bash +dart run skills@ get +``` + +Installs a bundled skill covering STT model choice, the 16 kHz mono PCM input contract, and the Whisper output language — which is a property of a *transcription*, not of the loaded model. + ## Usage ```dart diff --git a/packages/flutter_gemma_speech/example/pubspec.lock b/packages/flutter_gemma_speech/example/pubspec.lock index 6bf95308..fd13df15 100644 --- a/packages/flutter_gemma_speech/example/pubspec.lock +++ b/packages/flutter_gemma_speech/example/pubspec.lock @@ -180,7 +180,7 @@ packages: path: "../../flutter_gemma" relative: true source: path - version: "1.8.0" + version: "1.8.1" flutter_gemma_agent: dependency: "direct main" description: @@ -208,7 +208,7 @@ packages: path: ".." relative: true source: path - version: "0.5.0" + version: "0.5.1" flutter_inappwebview: dependency: transitive description: diff --git a/packages/flutter_gemma_speech/pubspec.yaml b/packages/flutter_gemma_speech/pubspec.yaml index deb02eb4..960a91f7 100644 --- a/packages/flutter_gemma_speech/pubspec.yaml +++ b/packages/flutter_gemma_speech/pubspec.yaml @@ -1,6 +1,6 @@ name: flutter_gemma_speech description: "On-device speech (STT + TTS) for flutter_gemma via the LiteRT C API + dart:ffi. Opt-in Stt/TtsBackendProvider." -version: 0.5.0 +version: 0.5.1 homepage: https://fluttergemma.dev repository: https://github.com/DenisovAV/flutter_gemma/tree/main/packages/flutter_gemma_speech topics: [speech, stt, gemma, litert, on-device] diff --git a/packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md b/packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md new file mode 100644 index 00000000..05024620 --- /dev/null +++ b/packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md @@ -0,0 +1,131 @@ +--- +name: flutter-gemma-speech-stt +description: Use when transcribing audio with flutter_gemma_speech — installing an STT model, choosing between moonshine/Whisper/Parakeet, or setting the output language. Whisper is multilingual and the language is a per-transcription property, not a property of the loaded model. +--- + +# Speech-to-text with flutter_gemma_speech + +## Register the backend first + +STT is opt-in. Core registers nothing: + +```dart +import 'package:flutter_gemma/flutter_gemma.dart'; +import 'package:flutter_gemma_speech/flutter_gemma_speech.dart'; + +await FlutterGemma.initialize(sttBackends: [LiteRtSttBackend()]); +``` + +Native only — Android, iOS, macOS, Windows, Linux. The web arm is a stub that +throws `UnsupportedError`. + +## Pick the model deliberately + +| `SttModelType` | Input | Languages | Notes | +| --- | --- | --- | --- | +| `moonshine` | raw 16 kHz PCM | the language it hears | ~104 MB, 5 s window, fastest | +| `whisper` | log-mel | 99, selectable | tiny / base, 30 s window | +| `parakeet` | log-mel | English only | CTC 0.6B, 2.35 GB f32, desktop | + +Only Whisper has a selectable output language. The other two transcribe +whatever they hear and **reject** a language argument rather than ignoring it. + +## Install and transcribe + +```dart +await FlutterGemma.installStt() + .modelFromNetwork('https://huggingface.co/litert-community/whisper-tiny/resolve/main/whisper_tiny_30s_f32.tflite') + .tokenizerFromNetwork('https://huggingface.co/openai/whisper-tiny/resolve/main/tokenizer.json') + .ofType(SttModelType.whisper) + .install(); + +final recognizer = await FlutterGemma.getActiveStt(); +try { + // pcm: 16 kHz mono 16-bit little-endian PCM — the data chunk of a WAV, + // or frames from a recorder. NOT the WAV file itself. + final transcript = await recognizer.transcribe(pcm); +} finally { + await recognizer.close(); +} +``` + +An STT model needs **two** files, a model and a tokenizer, and they usually come +from different repos: the LiteRT conversion of the weights, and the original +publisher's `tokenizer.json`. + +## The output language is per transcription + +This is the part that is easy to model wrongly. The language is one token in +Whisper's decoder seed prompt, and that prompt is rebuilt on every +transcription. Changing it costs a map lookup — it never reloads the model and +never invalidates a recognizer you are holding. + +```dart +// A default for this recognizer. +final stt = await FlutterGemma.getActiveStt(language: 'de'); +final german = await stt.transcribe(germanPcm); + +// One call in another language — same recognizer, nothing reloaded. +final french = await stt.transcribe(frenchPcm, language: 'fr'); +``` + +`getActiveStt` returns a process-wide singleton, and calling it again with a new +`language` retargets that recognizer. You never need to `close()` just to change +language. + +Codes are Whisper's own, without the delimiters — `'en'`, `'de'`, `'uk'`, any of +the 99 — and the default is `'en'`. + +## Language decides the OUTPUT, not comprehension + +The shipped Whisper checkpoints are the multilingual ones (no `.en` suffix), so +the weights understand the audio either way. The token only decides what the +model writes. Measured on one German clip, same audio and build, one token +apart: + +``` +'en' -> " This weather is very beautiful and the sun is shining." +'de' -> " Das Wetter ist heute sehr schön und die Sonne scheint." +``` + +So asking for the wrong language does not garble the output — it translates, +fluently and without any error. If a user reports "it always answers in +English", the language was never applied; it is not a model failure. + +## Bad values throw, they are never ignored + +- A malformed code (`'de-DE'`, `'DE'`, `'german'`, `''`) is rejected before the + model is loaded. +- A well-formed code the installed checkpoint does not have (`'zz'`) is rejected + against that checkpoint's own tokenizer, with the valid set named in the + error. +- Any language on `moonshine` or `parakeet` throws `ArgumentError` — those + models have no language token to set. + +Catch `ArgumentError` around a user-supplied language. Do not fall back to a +default silently; the whole design here exists because a silently ignored +language is indistinguishable from success. + +## Requirements + +- Audio must be **16 kHz mono 16-bit little-endian PCM**. Resample first; there + is no conversion inside the package. +- Clips are padded or trimmed to the model's fixed window (moonshine 5 s, + Whisper 30 s). Longer audio needs chunking by the caller. +- Transcription runs in a background isolate, so it does not block the UI. +- `flutter_gemma_speech` requires a matching core — check its `flutter_gemma` + constraint. A core too old accepts `language:` and drops it. + +## Voice loop + +`VoiceSession` chains STT to an LLM to TTS for a push-to-talk turn. It inherits +the recognizer's current language, so set it before starting the session: + +```dart +final session = VoiceSession.fromChat( + recognizer: await FlutterGemma.getActiveStt(language: 'de'), + chat: chat, + synthesizer: await FlutterGemma.getActiveTts(), +); +await for (final event in session.runTurn(pcm)) { /* … */ } +``` From 8232d35e47bb2ade8f2f03c8255d2d50f2bea27e Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Thu, 10 Sep 2026 22:23:38 +0200 Subject: [PATCH 02/21] feat(skills): eight consumer skills, all in the core package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reworked after discussion. Three changes from the first attempt. Split by TASK, not by API area. A single request — "add on-device chat" — needs the engine registration, the fileType declaration, the maxTokens floor and the isUser default all at once. Separate `setup` and `inference` skills risked the agent loading one and writing half-correct code, so they are one skill now. All eight live in `flutter_gemma`, not in the packages they describe. Per-package placement is what the naming convention is built for, but it buys one release path per skill: a typo in the litertlm skill would mean publishing litertlm. One home means one publish, and the CLI is interactive, so a MediaPipe-only user is offered the litertlm skill and simply declines it. Coverage now spans the engines rather than stopping at core: flutter-gemma-inference registry, fileType routing, maxTokens vs maxOutputTokens, isUser, lifecycle flutter-gemma-tools function calling — supportsFunctionCalls AND modelType are both required flutter-gemma-litertlm kv_cache 1024 floor, minSdk 30, backends flutter-gemma-mediapipe iOS 16 floor, web script tags, no desktop flutter-gemma-onnx the model is a DIRECTORY, five host arches flutter-gemma-builtin-ai no file to install, availability probing flutter-gemma-speech STT + TTS + VoiceSession flutter-gemma-rag TaskType asymmetry, CPU-only embedding `flutter_gemma_speech` drops out of this release entirely — its skill moved to core, so its version and README are back to what main has. Verified by running the CLI from the example app, not by reading the spec: all eight install, then `skills@ remove --all` cleans up. Archive carries exactly eight SKILL.md. --- .../skills/flutter-gemma-builtin-ai/SKILL.md | 80 +++++++ .../skills/flutter-gemma-inference/SKILL.md | 217 ++++++++---------- .../skills/flutter-gemma-litertlm/SKILL.md | 87 +++++++ .../skills/flutter-gemma-mediapipe/SKILL.md | 87 +++++++ .../skills/flutter-gemma-onnx/SKILL.md | 72 ++++++ .../skills/flutter-gemma-rag/SKILL.md | 94 ++++++++ .../skills/flutter-gemma-setup/SKILL.md | 146 ------------ .../skills/flutter-gemma-speech}/SKILL.md | 75 +++++- .../skills/flutter-gemma-tools/SKILL.md | 112 +++++++++ packages/flutter_gemma_speech/CHANGELOG.md | 3 - packages/flutter_gemma_speech/pubspec.yaml | 2 +- 11 files changed, 693 insertions(+), 282 deletions(-) create mode 100644 packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md delete mode 100644 packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md rename packages/{flutter_gemma_speech/skills/flutter-gemma-speech-stt => flutter_gemma/skills/flutter-gemma-speech}/SKILL.md (62%) create mode 100644 packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md diff --git a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md new file mode 100644 index 00000000..704c130c --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md @@ -0,0 +1,80 @@ +--- +name: flutter-gemma-builtin-ai +description: Use when running the OS's own model through flutter_gemma_builtin_ai — Gemini Nano on Android or Apple Foundation Models on iOS/macOS. There is no file to download or install; the OS owns the weights, so availability must be probed at runtime and can legitimately be "not yet downloaded". +--- + +# The built-in OS model + +`flutter_gemma_builtin_ai` runs the model the operating system already ships: +Gemini Nano through ML Kit GenAI / AICore on Android, Apple Foundation Models on +iOS and macOS. + +```dart +await FlutterGemma.initialize(inferenceEngines: [BuiltInAiEngine()]); +``` + +## There is no model file + +This is the difference from every other engine. `ModelFileType.builtIn` means +the OS owns the weights — nothing to download, nothing to bundle, no storage +budget, and `installModel` is not part of the flow. Ready-made specs are +provided: + +```dart +final model = await FlutterGemma.getActiveModel( + spec: BuiltInAiModels.geminiNano, // or .appleFoundationModels +); +``` + +The trade is that availability is not yours to control. + +## Probe availability before using it + +```dart +final availability = await BuiltInAi.availability(); +``` + +`BuiltInAiAvailability` has seven states, and three of them are not failures: + +| State | Meaning | +| --- | --- | +| `available` | ready now | +| `downloadable` | supported, weights not fetched yet | +| `downloading` | fetch in progress | +| `unavailableDeviceUnsupported` | this hardware will never support it | +| `unavailableOsTooOld` | an OS upgrade would fix it | +| `unavailableDisabled` | turned off by the user or by policy | +| `unavailableOther` | something else | + +Treat `downloadable` and `downloading` as "not yet", not as "no". `ensureReady` +triggers and awaits the download: + +```dart +await BuiltInAi.ensureReady(); +``` + +That can take minutes on first use and needs network, so drive it from an +explicit user action with visible progress — never from app start. + +An unusable state throws `BuiltInAiUnavailableException`. Catch it and fall back +to a downloadable model through another engine; do not let it reach the user as +a crash. + +## Android needs minSdk 26 + +ML Kit GenAI / AICore will not merge below API 26, so the manifest merger fails +at build time with a `uses-sdk:minSdkVersion` conflict. Raise `minSdk` to 26 in +`android/app/build.gradle(.kts)` for any app that includes this package. + +## Platforms + +Android, iOS and macOS only — **no web, no Windows, no Linux**. The engine +declines elsewhere rather than throwing, so a registry with another engine +registered still works. + +## What you give up + +The OS model is small and its behaviour is set by the platform: no choice of +weights, no LoRA, no control over quantisation, and capabilities that differ by +OS version. Use it when "zero download, zero disk" matters more than +capability; use `.litertlm` when you need a specific model. diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md index 9eacebc3..11614485 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md @@ -1,177 +1,140 @@ --- name: flutter-gemma-inference -description: Use when generating text with flutter_gemma — sessions, chats, streaming, tools, or anything involving maxTokens. maxTokens is the context window and NOT the reply length; setting it low crashes .litertlm models, and Message.isUser defaults to false. +description: Use whenever writing flutter_gemma code — installing a model, calling FlutterGemma.initialize/getActiveModel/createSession, or generating text. Core registers no engine by default, maxTokens is the context window and NOT the reply length, and Message.isUser defaults to false; all three fail quietly. --- -# Generating with flutter_gemma +# Running a model with flutter_gemma -Three defaults in this API produce wrong behaviour rather than errors. They -account for most of the issues filed against the package. +This is the path from an empty app to a generated token. Four defaults on it +produce wrong behaviour rather than an error — start with those. -## maxTokens is the CONTEXT WINDOW, not the reply length +## 1. Core ships no engine — register one -`maxTokens` on `getActiveModel` / `createModel` is the whole KV-cache budget: -system prompt + history + the current message + everything generated. It is not -"how long the answer may be". +`flutter_gemma` is the contracts, the registry and the platform shells. It has +no inference runtime. Adding only `flutter_gemma` compiles fine and throws on +the first `getActiveModel()`. -Setting it small to get a short answer is the single most common mistake, and on -`.litertlm` it does not merely truncate — it crashes. Every supported -`.litertlm` model bakes `kv_cache_max_len = 1024`; below that the native -KV-cache resize underflows and tensor allocation fails at generation time with a -message that names an internal executor file and nothing else. +| Package | Handles | +| --- | --- | +| `flutter_gemma_litertlm` | `.litertlm` — the main path, all six platforms | +| `flutter_gemma_mediapipe` | `.task`, `.bin` — mobile + web | +| `flutter_gemma_builtin_ai` | the OS model, no file to install | +| `flutter_gemma_onnx` | ONNX / ORT-GenAI | ```dart -// WRONG — meant "a 100-token reply", actually a 100-token context. -final model = await FlutterGemma.getActiveModel(maxTokens: 100); +import 'package:flutter_gemma/flutter_gemma.dart'; +import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; -// RIGHT — roomy context, capped output. -final model = await FlutterGemma.getActiveModel(maxTokens: 1024); -final session = await model.createSession(maxOutputTokens: 100); +await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); ``` -Measured on a Pixel 8a (CPU): 100 / 256 / 512 crash; 1024 and 4096 work. The -litertlm engine now clamps values below 1024 upward and logs a warning, but do -not rely on that — say what you mean. +Every capability is opt-in the same way and defaults to an empty list — +`embeddingBackends`, `sttBackends`, `ttsBackends`, `huggingFaceResolvers`. If a +list is empty, the matching first call throws a `StateError` naming the package +to add. Read that message rather than guessing. -`maxOutputTokens` is `.litertlm` only. MediaPipe `.task` has no session-level -output cap and logs that the value was ignored. +## 2. The engine is chosen by the DECLARED file type, never the filename -## Message.isUser defaults to false +`installModel` defaults to `ModelFileType.task`. A `.litertlm` file installed +without declaring its type is routed to MediaPipe, which cannot read it. ```dart -// WRONG — silently treated as an assistant message, so the model has -// nothing to answer and returns an empty string. -const Message(text: 'Hello') +// WRONG — the name says .litertlm, the declaration says .task, +// and the declaration is what routes it. +await FlutterGemma.installModel(modelType: ModelType.gemma4) + .fromNetwork(url).install(); // RIGHT -const Message(text: 'Hello', isUser: true) +await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, +).fromNetwork(url).install(); ``` -There is no error. The response is just empty. Always pass `isUser` -explicitly. +`modelType` is a separate axis: it drives the chat template and the model's +capabilities (`gemma4`, `gemma3`, `qwen3`, `deepSeek`, `general`, …). Getting it +wrong also fails quietly — the model generates, with the wrong prompt format. + +Sources: `.fromNetwork(url, token:)`, `.fromAsset(path)`, `.fromFile(file)`, and +`.fromHuggingFace(repo)` when a resolver is registered. -## Always close sessions and models +## 3. maxTokens is the CONTEXT WINDOW, not the reply length -Both hold native resources — an isolate, a compiled model, GPU buffers. Leaking -them exhausts memory and, on some backends, wedges the next load. +`maxTokens` is the whole KV-cache budget: system prompt + history + the current +message + everything generated. It is not "how long the answer may be". + +Setting it small to get a short answer is the most common mistake with this +package, and on `.litertlm` it does not truncate — it crashes, with a message +naming an internal executor file and nothing else. ```dart -final session = await model.createSession(); -try { - // … -} finally { - await session.close(); -} -await model.close(); +// WRONG — meant "a 100-token reply", actually a 100-token context. +final model = await FlutterGemma.getActiveModel(maxTokens: 100); + +// RIGHT — roomy context, capped output. +final model = await FlutterGemma.getActiveModel(maxTokens: 1024); +final session = await model.createSession(maxOutputTokens: 100); ``` -## One-shot generation +Use 4096+ for vision or audio — one image is worth hundreds of tokens. + +## 4. Message.isUser defaults to false + +```dart +const Message(text: 'Hello') // WRONG — empty response +const Message(text: 'Hello', isUser: true) // RIGHT +``` + +No error is raised. The response is just empty. + +## Generating ```dart final session = await model.createSession( temperature: 0.8, - randomSeed: 1, topK: 40, maxOutputTokens: 256, ); try { await session.addQueryChunk( - const Message(text: 'Explain isolates in one paragraph.', isUser: true), + const Message(text: 'Explain isolates briefly.', isUser: true), ); final answer = await session.getResponse(); + // streaming: await for (final chunk in session.getResponseAsync()) … } finally { await session.close(); } +await model.close(); ``` -Streaming, same session shape: - -```dart -await for (final chunk in session.getResponseAsync()) { - stdout.write(chunk); -} -``` +Sessions and models hold native resources — an isolate, a compiled model, GPU +buffers. Always close them, in a `finally`. -## Multi-turn conversation +## Multi-turn -`InferenceChat` keeps the history and applies the model's chat template. +`InferenceChat` keeps history and applies the model's chat template: ```dart -final chat = await model.createChat( - tokenBuffer: 256, - maxOutputTokens: 512, -); +final chat = await model.createChat(tokenBuffer: 256, maxOutputTokens: 512); try { await chat.addQueryChunk(const Message(text: 'Hi', isUser: true)); - final reply = await chat.generateChatResponse(); + final response = await chat.generateChatResponse(); + if (response is TextResponse) print(response.token); } finally { await chat.close(); } ``` -`generateChatResponse()` returns a `ModelResponse`, not a `String`. Switch on -it — with tools enabled it may be a function call: +`generateChatResponse()` returns a `ModelResponse`, not a `String`. With tools +enabled it may be a function call — see the `flutter-gemma-tools` skill. -```dart -switch (response) { - case TextResponse(:final token): // plain text - case FunctionCallResponse(:final name, :final args): // one tool call - case ParallelFunctionCallResponse(:final calls): // several at once -} -``` +Thinking models (Qwen3, DeepSeek R1) emit `` blocks; pass +`isThinking: true` to surface them as `ThinkingResponse`, or `false` to have +them stripped. -## Function calling +## Multimodal -Two arguments beyond `tools` are load-bearing. Without `supportsFunctionCalls` -the tools are dropped with a warning; without `modelType` the correct call -format cannot be derived. - -```dart -final chat = await model.createChat( - tools: [ - const Tool( - name: 'change_color', - description: 'Change the UI background colour.', - parameters: { - 'type': 'object', - 'properties': { - 'color': {'type': 'string', 'description': 'A colour name.'}, - }, - 'required': ['color'], - }, - ), - ], - supportsFunctionCalls: true, - modelType: ModelType.gemma4, -); - -final response = await chat.generateChatResponse(); -if (response is FunctionCallResponse) { - final result = await runTool(response.name, response.args); - await chat.addQueryChunk( - Message.toolResponse(toolName: response.name, response: result), - ); - final followUp = await chat.generateChatResponse(); -} -``` - -Not every model supports tools — check the package README's support table before -enabling them. - -## Thinking models - -Qwen3 and DeepSeek R1 emit reasoning in `` tags. Pass `isThinking` so the -tags are handled rather than shown to the user: - -```dart -final chat = await model.createChat(isThinking: true, modelType: ModelType.qwen3); -``` - -With `isThinking: false` on Qwen3 the tags are stripped automatically. - -## Vision and audio - -Declare support at model creation, then attach bytes to a message: +Declare support at model creation, then attach bytes: ```dart final model = await FlutterGemma.getActiveModel( @@ -183,16 +146,20 @@ await session.addQueryChunk( ); ``` -Multimodal models need a much larger context than text-only ones — an image is -worth hundreds of tokens. Start at 4096. +## Cost model -## Sessions are not free - -Model loading is expensive; session creation is cheap by comparison. Load the -model once and keep it, create and close a session per interaction. Do not call -`getActiveModel` per message. +Loading a model is expensive; creating a session is cheap. Load once, keep the +model, create and close a session per interaction. Never call `getActiveModel` +per message. `getActiveModel` returns a process-wide singleton. Calling it again with -different runtime arguments rebuilds it and closes the previous instance — so a +different runtime arguments rebuilds it and closes the previous instance — a handle you are still holding becomes unusable. Decide the runtime configuration once. + +## Engine-specific rules + +Platform floors, backend selection and format quirks live with each engine: +`flutter-gemma-litertlm`, `flutter-gemma-mediapipe`, `flutter-gemma-onnx`, +`flutter-gemma-builtin-ai`. Read the one for the engine in use — this skill +covers only what is common to all of them. diff --git a/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md new file mode 100644 index 00000000..4667fe0a --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md @@ -0,0 +1,87 @@ +--- +name: flutter-gemma-litertlm +description: Use when the app runs .litertlm models through flutter_gemma_litertlm — the default engine on all six platforms. Covers the 1024-token KV cache floor that crashes below it, the Android minSdk 30 requirement, and backend selection. +--- + +# The .litertlm engine + +`flutter_gemma_litertlm` is the main inference path: Dart FFI straight into the +LiteRT-LM C API on Android, iOS, macOS, Windows and Linux, plus a web arm. No +JVM, no gRPC, no separate process. + +```dart +await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); + +await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, // required — the default is `task` +).fromNetwork(url).install(); +``` + +## maxTokens below 1024 crashes, it does not truncate + +Every supported `.litertlm` model bakes `kv_cache_max_len = 1024`. A `maxTokens` +below that underflows the native KV-cache resize and tensor allocation fails at +generation with a message naming an internal executor file. + +Measured on a Pixel 8a (CPU): 100 / 256 / 512 crash; 1024 and 4096 work. + +The engine now clamps values below 1024 upward and logs a warning, but do not +rely on that. Pass a real context size and cap the reply with +`maxOutputTokens` on the session — that one is `.litertlm`-only and is the +correct knob for reply length. + +## Android needs minSdk 30 + +`libLiteRtLm.so` uses `pthread_cond_clockwait` and `sem_clockwait`, which are +API-30-only Bionic symbols. On API 29 it fails at `dlopen`, not at build time. +Set `minSdk 30` in `android/app/build.gradle(.kts)` for any app that runs +`.litertlm`. + +Only `arm64-v8a` is shipped. A build implying other ABIs produces an APK +without the native library. + +## Backends + +```dart +final model = await FlutterGemma.getActiveModel( + maxTokens: 1024, + preferredBackend: PreferredBackend.gpu, +); +``` + +| | Android | iOS | Desktop | Web | +| --- | --- | --- | --- | --- | +| `cpu` | yes | yes | yes | no | +| `gpu` | yes | device only | yes | required | +| `npu` | Snapdragon | no | Windows, Intel Lunar/Panther Lake | no | + +Passing an explicit backend disables fallback — if that backend cannot load the +model, it fails rather than quietly trying another. Leaving it null tries GPU +then CPU. Some models bake a constraint: the 12B build declares +`section_backend_constraint: gpu` and an explicit `cpu` on it simply fails. + +**iOS Simulator is CPU-only.** Metal there has a 256 MB single-allocation cap +and model weights exceed it. Test GPU on a real device. + +## Desktop is `.litertlm` only + +There is no `.task` support on macOS, Windows or Linux. Windows GPU needs +`dxil.dll` and `dxcompiler.dll`, and Windows NPU needs Intel Lunar/Panther Lake +silicon — both ship inside the bundled native archive, nothing to install. + +## Native libraries are fetched at build time + +`hook/build.dart` downloads a per-platform archive from a pinned GitHub release +and verifies its SHA256 (Native Assets). Nothing is committed to the repo and +nothing ships in the pub package, so a first build needs network access. + +If a build fails on a missing symbol after upgrading, the shared-bundle owner +marker went stale: delete +`~/Library/Caches/flutter_gemma/native/.flutter_gemma_native_version` and run +`flutter clean`. + +## Web is an early preview + +Text only. No vision, audio, thinking, function calling or LoRA on the +`.litertlm` web arm — those work on native. Web is also GPU-only. diff --git a/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md new file mode 100644 index 00000000..92767361 --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md @@ -0,0 +1,87 @@ +--- +name: flutter-gemma-mediapipe +description: Use when the app runs .task or .bin models through flutter_gemma_mediapipe — mobile and web only, no desktop. Covers the iOS 16 floor this package alone imposes, the web script tags, and why maxOutputTokens is ignored here. +--- + +# The MediaPipe engine + +`flutter_gemma_mediapipe` runs `.task` and `.bin` models through MediaPipe +GenAI on Android, iOS and web. **There is no desktop support** — macOS, Windows +and Linux need `.litertlm`. + +```dart +await FlutterGemma.initialize(inferenceEngines: [MediaPipeEngine()]); + +await FlutterGemma.installModel( + modelType: ModelType.gemma3, + fileType: ModelFileType.task, // the default, but say it anyway +).fromNetwork(url).install(); +``` + +`.task` is the default `fileType`, so this is the one engine where forgetting to +declare it happens to work. Declare it regardless — it documents intent, and a +later switch to `.litertlm` then fails loudly instead of silently routing here. + +## This package alone requires iOS 16 + +Core, litertlm, embeddings and builtin_ai all build from iOS 15.0. MediaPipe +GenAI raises the floor to **16.0** for the whole app: + +```ruby +platform :ios, '16.0' +use_frameworks! :linkage => :static +``` + +If the app does not use `.task` models, drop this package and stay on 15. + +## maxOutputTokens is ignored + +MediaPipe has no session-level output cap. Passing `maxOutputTokens` logs that +it was ignored and generation runs to the model's own limit. To bound output +here, stop consuming the stream yourself. + +`maxTokens` still means the context window, and MediaPipe tolerates small values +rather than crashing — unlike `.litertlm`. That difference is a property of the +engine, not of the API. + +## Web + +The web arm needs the MediaPipe runtime loaded before Flutter starts. Add to +`web/index.html`: + +```html + +``` + +Pin the version. An unpinned CDN import takes whatever published last, which has +shipped broken before. + +Web is **GPU-only** — there is no CPU backend for MediaPipe in the browser — and +large models need the streaming storage mode: + +```dart +await FlutterGemma.initialize( + webStorageMode: WebStorageMode.streaming, // OPFS; cacheApi caps near 2 GB + inferenceEngines: [MediaPipeEngine()], +); +``` + +The runtime JS files (`cache_api.js`, `opfs_helper.js`) are not injected +automatically — copy them into the app's own `web/` directory. + +## Android + +The plugin ships its own native layer and its own pigeon; nothing to configure +beyond the usual OpenCL `` entries, which the core plugin's +manifest merges in for you. + +Only `arm64-v8a` is shipped. + +## Vision + +Multimodal `.task` models work on Android, iOS and web. Audio input does not — +that is `.litertlm` only, and only on native. diff --git a/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md new file mode 100644 index 00000000..8aeb19eb --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md @@ -0,0 +1,72 @@ +--- +name: flutter-gemma-onnx +description: Use when running ONNX models through flutter_gemma_onnx — ORT-GenAI text generation or ORT embeddings. An ORT-GenAI model is a DIRECTORY, not a single file, so the ordinary single-file network install does not apply; the package is also gated to five specific host architectures. +--- + +# The ONNX engine + +`flutter_gemma_onnx` provides two things over `dart:ffi` in a worker isolate: +text generation via ORT-GenAI (`OnnxEngine`) and embeddings via plain ORT +(`OnnxEmbeddingBackend`). + +```dart +await FlutterGemma.initialize( + inferenceEngines: [OnnxEngine()], + embeddingBackends: [OnnxEmbeddingBackend()], +); +``` + +## An ORT-GenAI model is a DIRECTORY + +This is the difference that breaks the usual mental model. The model is not one +file: + +``` +phi-3.5-mini/ + genai_config.json + model.onnx + model.onnx_data # weights, often several GB + tokenizer.json +``` + +`OnnxEngine.createModel` takes that directory's `genai_config.json` and loads +the **parent directory**. The single-file `.fromNetwork(url)` install used for +`.litertlm` and `.task` does not cover this — the files must arrive together, +by bundling them as assets or fetching them into one directory yourself. + +Embeddings are the exception: a plain `.onnx` embedding model is a single file +and installs normally. + +## Only five host architectures + +`OnnxEngine.canHandle` is gated to **macOS arm64, Linux x64, Windows x64, +Android arm64 and iOS arm64**, in lockstep with the build hook that bundles the +native archives. Anywhere else it declines and logs why, so the registry falls +through to another engine rather than failing at load. + +`OnnxEmbeddingBackend.canHandle` stays extension-based on every platform — so +that LiteRT's catch-all cannot silently claim an `.onnx` file — and gates inside +`createModel` instead. The error therefore arrives at model creation, not at +registration. + +## Memory + +Phi-3.5-mini 3.8B int4 peaks around **3.74 GB RSS**. That needs a 6 GB+ phone; +below that the OS kills the app during load rather than reporting an error you +can catch. + +Measured throughput: macOS M4 Pro ~54 tok/s, Pixel 8 Pro ~10.4 tok/s, Linux +~5.3-5.8 tok/s, Windows ~3.3 tok/s on CPU test VMs. Treat ONNX as the +portability option, not the fast one — `.litertlm` is faster where both run. + +## No web + +There is no web arm. Use `.task` through MediaPipe or `.litertlm` in the +browser. + +## Native libraries + +Fetched at build time from Microsoft's own GitHub releases by the package's +`hook/build.dart` (Native Assets), not from a repo tag. On iOS the ORT runtime +is statically linked into the GenAI framework, so there is one binary rather +than two. diff --git a/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md new file mode 100644 index 00000000..8415f9ae --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md @@ -0,0 +1,94 @@ +--- +name: flutter-gemma-rag +description: Use when building retrieval over on-device documents with flutter_gemma — embeddings via flutter_gemma_embeddings plus a vector store (rag_sqlite or rag_qdrant). Query and document embeddings need DIFFERENT TaskType prefixes or retrieval quality collapses, and embedding is CPU-only by design. +--- + +# On-device RAG with flutter_gemma + +Three pieces: an embedding model, a vector store, and the retrieval call. All +three are opt-in packages. + +```dart +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], + embeddingBackends: [LiteRtEmbeddingBackend()], // flutter_gemma_embeddings + vectorStore: SqliteVectorStore(), // flutter_gemma_rag_sqlite +); +``` + +Without a `vectorStore` the default sentinel throws a clear "add a RAG package" +error on first use. Without an embedding backend, `getActiveEmbedder` throws the +same way. + +## Query and document must use DIFFERENT task types + +This is the mistake that silently ruins retrieval. Embedding models are trained +asymmetrically: a question and the passage that answers it are encoded with +different prefixes, and using one prefix for both collapses the similarity +signal. Nothing errors — results are just bad. + +```dart +// Indexing a document +final docVector = await embedder.generateEmbedding( + chunk, + taskType: TaskType.retrievalDocument, +); + +// Searching with a question +final queryVector = await embedder.generateEmbedding( + question, + taskType: TaskType.retrievalQuery, +); +``` + +The prefix strings live in one place in Dart and are applied for you — pass the +right `TaskType` and do not prepend anything yourself. + +## Embedding is CPU-only, and that is permanent + +EmbeddingGemma ships as int4, and the TFLite GPU delegate cannot execute int4. +This is not a missing feature or a bug to work around: there is no GPU path. +Asking for `PreferredBackend.gpu` on an embedder gains nothing. + +Budget accordingly — embedding a large corpus on device is minutes of CPU, so +do it in the background, batched, and persist the vectors rather than +recomputing at startup. + +## Picking a vector store + +| Package | Platforms | Notes | +| --- | --- | --- | +| `flutter_gemma_rag_sqlite` | all six, web included | `sqlite-vec` KNN inside SQLite | +| `flutter_gemma_rag_qdrant` | native only, no web | the official `qdrant_edge` SDK | + +`rag_sqlite` on web needs a custom `sqlite3.wasm` with `vec0` linked in — copy +it into the app's own `web/` directory. Nothing does that automatically. + +## Filtering + +Both stores take the same sealed `Filter` DSL from core, so a query written +against one works against the other: + +```dart +final results = await FlutterGemma.rag.search( + queryVector, + limit: 5, + filter: Filter(must: [Condition.equals('lang', 'en')]), +); +``` + +## Chunking is yours + +The package embeds what you give it. Splitting documents, choosing chunk size +and overlap, and storing the text alongside the vector are all application +decisions. A chunk longer than the model's input window is truncated silently — +check the model's limit rather than assuming. + +## Do not double-normalise + +`meanPoolAndNormalize` accepts only token-level output shaped `[1, seq, dim]` +and deliberately rejects rank-2 input. A model that already returns a pooled, +normalised vector must not be pooled again — doing so distorts every distance in +the index, and the failure is invisible until retrieval quality is measured. If +a rank-2 rejection fires, the model's output contract is pooled-final; wire it +as such rather than reshaping to get past the check. diff --git a/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md deleted file mode 100644 index a7fa5eed..00000000 --- a/packages/flutter_gemma/skills/flutter-gemma-setup/SKILL.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -name: flutter-gemma-setup -description: Use when adding flutter_gemma to a Flutter app, calling FlutterGemma.initialize, or installing a model — core registers no inference engine by default and routes by the DECLARED file type, so both are easy to get silently wrong. ---- - -# Setting up flutter_gemma - -Two decisions here fail quietly rather than loudly. Get them right first. - -## 1. Core ships no engine — you must register one - -`flutter_gemma` is the contracts, the registry and the platform shells. It -contains no inference runtime. Adding only `flutter_gemma` to `pubspec.yaml` -compiles fine and throws on the first `getActiveModel()`. - -Pick the engine package for the model format you actually have: - -| Package | Handles | Platforms | -| --- | --- | --- | -| `flutter_gemma_litertlm` | `.litertlm` | Android, iOS, macOS, Windows, Linux, web | -| `flutter_gemma_mediapipe` | `.task`, `.bin` | Android, iOS, web | -| `flutter_gemma_builtin_ai` | the OS model (Gemini Nano, Apple Foundation Models) | Android 26+, iOS, macOS | -| `flutter_gemma_onnx` | ONNX / ORT-GenAI | macOS arm64, Linux x64, Windows x64, Android arm64, iOS arm64 | - -Register the providers before anything else runs: - -```dart -import 'package:flutter_gemma/flutter_gemma.dart'; -import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; - -await FlutterGemma.initialize( - inferenceEngines: [LiteRtLmEngine()], -); -``` - -Every capability is opt-in the same way and defaults to an empty list: - -```dart -await FlutterGemma.initialize( - inferenceEngines: [LiteRtLmEngine()], // text / vision / audio - embeddingBackends: [LiteRtEmbeddingBackend()], // flutter_gemma_embeddings - sttBackends: [LiteRtSttBackend()], // flutter_gemma_speech - ttsBackends: [LiteRtTtsBackend()], // flutter_gemma_speech - huggingFaceToken: 'hf_…', // only for gated repos -); -``` - -If a list is empty, the corresponding first call throws a `StateError` naming -the package to add. That message is the intended diagnostic — read it rather -than guessing. - -## 2. The engine is chosen by the DECLARED file type, never the filename - -`installModel` defaults to `ModelFileType.task`. A `.litertlm` file installed -without declaring its type is routed to MediaPipe, which cannot read it. - -```dart -// WRONG — the name says .litertlm, the declaration says .task, -// and the declaration is what routes it. -await FlutterGemma.installModel(modelType: ModelType.gemma4) - .fromNetwork('https://example.com/model.litertlm') - .install(); - -// RIGHT -await FlutterGemma.installModel( - modelType: ModelType.gemma4, - fileType: ModelFileType.litertlm, -).fromNetwork('https://example.com/model.litertlm').install(); -``` - -`modelType` is a separate axis and drives the chat template and capabilities -(`ModelType.gemma4`, `.gemma3`, `.qwen3`, `.deepSeek`, `.general`, …). Getting -it wrong does not fail loudly either — it produces a model that generates, but -with the wrong prompt format. - -## Installing from Hugging Face - -When the repo publishes a deployment manifest, one call resolves the variant, -the revision and the runtime defaults: - -```dart -await FlutterGemma.initialize( - inferenceEngines: [LiteRtLmEngine()], - huggingFaceResolvers: [LitertlmManifestResolver()], -); - -final install = await FlutterGemma.installModel( - modelType: ModelType.general, - fileType: ModelFileType.litertlm, -).fromHuggingFace('litert-community/LFM2.5-230M').install(); - -// The manifest's own runtime defaults — pass them through rather than -// inventing values. -final model = await FlutterGemma.getActiveModel(defaults: install.runtime); -``` - -Without a registered resolver `fromHuggingFace` throws a clear error. `defaults` -carries what the model publisher tested; an explicit argument you pass alongside -it always wins. - -## Other model sources - -```dart -.fromNetwork(url, token: hfToken) // download, resumable -.fromAsset('assets/model.litertlm') // bundled in the app -.fromFile(File(path)) // already on disk -``` - -## Platform notes that bite - -- **Android + `.litertlm`** needs `minSdk 30`. The native library uses - API-30-only Bionic symbols; on 29 it fails at `dlopen`. -- **`flutter_gemma_builtin_ai`** needs `minSdk 26`, or the manifest merger - fails. -- **iOS** builds from 15.0, except with `flutter_gemma_mediapipe`, which - requires 16.0. -- **Web** is GPU-only and needs the CDN script tags plus the runtime JS files - copied into the app's `web/` directory. See the package README. -- **iOS Simulator** cannot run GPU inference — Metal there has a 256 MB - single-allocation cap and model weights exceed it. Use CPU or a real device. - -## Verifying setup - -```dart -await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); -await FlutterGemma.installModel( - modelType: ModelType.gemma4, - fileType: ModelFileType.litertlm, -).fromNetwork(url).install(); - -final model = await FlutterGemma.getActiveModel(maxTokens: 1024); -final session = await model.createSession(); -try { - await session.addQueryChunk( - const Message(text: 'Say hello.', isUser: true), - ); - print(await session.getResponse()); -} finally { - await session.close(); -} -await model.close(); -``` - -If that runs, the engine is registered, the file type routed correctly and the -model loaded. See `flutter-gemma-inference` for what to do next, and why -`maxTokens: 1024` rather than something smaller. diff --git a/packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md similarity index 62% rename from packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md rename to packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md index 05024620..e3c6a498 100644 --- a/packages/flutter_gemma_speech/skills/flutter-gemma-speech-stt/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md @@ -1,9 +1,9 @@ --- -name: flutter-gemma-speech-stt -description: Use when transcribing audio with flutter_gemma_speech — installing an STT model, choosing between moonshine/Whisper/Parakeet, or setting the output language. Whisper is multilingual and the language is a per-transcription property, not a property of the loaded model. +name: flutter-gemma-speech +description: Use when adding speech to a flutter_gemma app — transcription (moonshine/Whisper/Parakeet), synthesis (Matcha/Qwen3/Inflect), or the VoiceSession loop. Audio must be 16 kHz mono 16-bit PCM, and the Whisper output language is a property of a transcription rather than of the loaded model. --- -# Speech-to-text with flutter_gemma_speech +# Speech with flutter_gemma_speech ## Register the backend first @@ -13,7 +13,10 @@ STT is opt-in. Core registers nothing: import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_speech/flutter_gemma_speech.dart'; -await FlutterGemma.initialize(sttBackends: [LiteRtSttBackend()]); +await FlutterGemma.initialize( + sttBackends: [LiteRtSttBackend()], // transcription + ttsBackends: [LiteRtTtsBackend()], // synthesis +); ``` Native only — Android, iOS, macOS, Windows, Linux. The web arm is a stub that @@ -116,10 +119,54 @@ language is indistinguishable from success. - `flutter_gemma_speech` requires a matching core — check its `flutter_gemma` constraint. A core too old accepts `language:` and drops it. +## Text-to-speech + +```dart +await FlutterGemma.installTts() + .fromNetwork('https://huggingface.co/litert-community/Matcha-TTS/resolve/main/') + .ofType(TtsModelType.matcha) + .install(); + +final synth = await FlutterGemma.getActiveTts(); +try { + final pcm = await synth.synthesize('Hello world.'); // Uint8List, 16-bit PCM + print(synth.sampleRate); // 22050 for Matcha +} finally { + await synth.close(); +} +``` + +`sampleRate` differs per model — read it rather than assuming, or playback is +pitched wrong. + +| `TtsModelType` | Languages | Notes | +| --- | --- | --- | +| `matcha` | its bundle's locale | fast, no runtime language parameter | +| `qwen3` | many, selectable | pass `language:` to `getActiveTts` | +| `inflect` | English only | ~90x real time on CPU | + +## TTS language fails LOUD, unlike STT + +`getActiveTts` returns a process-wide singleton, and asking an existing +synthesizer for a different language **throws** a `StateError` telling you to +`close()` first. That is deliberate: reusing it would emit wrong-language audio +with no error. + +```dart +final en = await FlutterGemma.getActiveTts(language: 'english'); +await en.close(); // required +final de = await FlutterGemma.getActiveTts(language: 'german'); +``` + +Note the asymmetry with STT, which retargets silently and cheaply instead: a +Whisper decoder prompt is rebuilt per transcription, a TTS voice is not. Values +here are full lowercase names (`'english'`, `'german'`), not the ISO codes STT +uses. + ## Voice loop -`VoiceSession` chains STT to an LLM to TTS for a push-to-talk turn. It inherits -the recognizer's current language, so set it before starting the session: +`VoiceSession` chains STT to an LLM to TTS for one push-to-talk turn, with +barge-in. ```dart final session = VoiceSession.fromChat( @@ -127,5 +174,19 @@ final session = VoiceSession.fromChat( chat: chat, synthesizer: await FlutterGemma.getActiveTts(), ); -await for (final event in session.runTurn(pcm)) { /* … */ } + +await for (final event in session.runTurn(pcm16kMono)) { + switch (event) { + case VoiceTranscriptEvent(:final text): // show it + case VoiceReplyTextEvent(:final chunk): // stream it + case VoiceReplyAudioEvent(:final pcm, :final sampleRate): // play it + case VoiceTurnInterruptedEvent(): // stop the player + case VoiceTurnCompleteEvent(): + case VoiceErrorEvent(): + } +} ``` + +The session inherits the recognizer's current language, so set it before +starting. A chat with tools is supported — pass `onToolCall`; a tools-enabled +chat arriving without a handler throws. diff --git a/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md new file mode 100644 index 00000000..5196cbc6 --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md @@ -0,0 +1,112 @@ +--- +name: flutter-gemma-tools +description: Use when giving a flutter_gemma model access to functions — declaring Tool objects, calling createChat with tools, handling FunctionCallResponse, or feeding a tool result back. Both supportsFunctionCalls and modelType are required or the tools are dropped with no error. +--- + +# Function calling with flutter_gemma + +## Three arguments, not one + +Passing `tools` alone does nothing. Without `supportsFunctionCalls` the tools +are dropped with a debug-only warning; without `modelType` the correct call +format cannot be derived. All three are required: + +```dart +final chat = await model.createChat( + tools: myTools, + supportsFunctionCalls: true, + modelType: ModelType.gemma4, +); +``` + +Get this wrong and the model answers in prose, describing what it would do +instead of calling anything. That looks like "the model is bad at tools" and is +actually a missing argument. + +## Declaring a tool + +`parameters` is a JSON Schema object: + +```dart +const changeColor = Tool( + name: 'change_color', + description: 'Change the UI background colour.', + parameters: { + 'type': 'object', + 'properties': { + 'color': {'type': 'string', 'description': 'A colour name like red.'}, + }, + 'required': ['color'], + }, +); +``` + +The `description` is what the model matches the user's intent against. Write it +as an action, and describe every parameter — a bare `{'type': 'string'}` gives +the model nothing to reason with. + +## The response is a type, not a string + +`generateChatResponse()` returns `ModelResponse`. Switch on it: + +```dart +final response = await chat.generateChatResponse(); + +switch (response) { + case FunctionCallResponse(:final name, :final args): + final result = await runTool(name, args); + await chat.addQueryChunk( + Message.toolResponse(toolName: name, response: result), + ); + final followUp = await chat.generateChatResponse(); + + case ParallelFunctionCallResponse(:final calls): + // several calls in one turn — run them, then feed each result back + for (final call in calls) { /* … */ } + + case TextResponse(:final token): + // the model chose to answer directly, which is a valid outcome +} +``` + +Treat a plain `TextResponse` as normal. A model that calls a tool for every +message is worse than one that decides. + +Streaming works the same way — `generateChatResponseAsync()` yields +`ModelResponse` events, and a `FunctionCallResponse` arrives as one of them. +Raw `<|tool_call>` markers must never appear in the text stream; if they do, +the model type is wrong. + +## Feeding the result back + +```dart +Message.toolResponse( + toolName: 'change_color', + response: {'status': 'success', 'applied_color': 'purple'}, +) +``` + +The response map is serialised into the transcript, so keep it small and +factual. Return an error field rather than throwing — the model can recover +from `{'error': 'unknown colour'}` and cannot recover from an exception. + +## Not every model supports tools + +Check the package README's support table before enabling them. Gemma 4, Gemma +3 1B, FunctionGemma, Phi-4 Mini, Qwen 2.5/3 and DeepSeek R1 do; Gemma 3 270M +and SmolLM do not. Enabling tools on a model that cannot use them produces +prose, not an error. + +## Escaped tokens in arguments + +Some runtimes emit `<|"|>` escape markers inside argument strings. The package +strips them before the call reaches you, so `args` values should be clean — +if you see them, report it rather than stripping them yourself. + +## Web + +Function calling on the `.litertlm` web path works, but the constrained-decoding +grammar does not reset after a completed tool-call block: any subsequent turn in +that same chat aborts with `Invalid token at state N`. Until the upstream fix +lands, treat a tools-enabled web chat as single-turn — create a fresh chat after +a call. A conversation that never emits a call is unaffected. diff --git a/packages/flutter_gemma_speech/CHANGELOG.md b/packages/flutter_gemma_speech/CHANGELOG.md index 9e62a572..b32cdaf8 100644 --- a/packages/flutter_gemma_speech/CHANGELOG.md +++ b/packages/flutter_gemma_speech/CHANGELOG.md @@ -1,6 +1,3 @@ -## 0.5.1 -- Ship an agent skill for speech-to-text (`dart run skills@ get`). - ## 0.5.0 - Whisper: output language is a parameter, no longer hardcoded English (#500). diff --git a/packages/flutter_gemma_speech/pubspec.yaml b/packages/flutter_gemma_speech/pubspec.yaml index 960a91f7..deb02eb4 100644 --- a/packages/flutter_gemma_speech/pubspec.yaml +++ b/packages/flutter_gemma_speech/pubspec.yaml @@ -1,6 +1,6 @@ name: flutter_gemma_speech description: "On-device speech (STT + TTS) for flutter_gemma via the LiteRT C API + dart:ffi. Opt-in Stt/TtsBackendProvider." -version: 0.5.1 +version: 0.5.0 homepage: https://fluttergemma.dev repository: https://github.com/DenisovAV/flutter_gemma/tree/main/packages/flutter_gemma_speech topics: [speech, stt, gemma, litert, on-device] From 54c225b0c45b40c9f721128d2facf3a8d2b8b841 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Thu, 10 Sep 2026 22:29:20 +0200 Subject: [PATCH 03/21] chore(release): keep the shipped skills from going stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Skills are read by a machine. A stale README makes a human squint; a stale SKILL.md makes their agent write confident code against an API that moved, and nothing in the build notices — markdown keeps analyze, test and format green. Two additions. `tool/check_skills.sh` extracts every API symbol the skills name and fails if one is gone from packages/*/lib/. It reads three shapes, because between them they cover where names actually appear: inline backticks, named arguments inside ```dart fences, and dotted members inside them. The first shape alone is not enough — it misses maxOutputTokens entirely, which the skills mention only inside code blocks. Currently 135 symbols across eight skills, zero missing. Mutation-checked rather than assumed: renaming maxOutputTokens, supportsFunctionCalls, TaskType.retrievalQuery, BuiltInAi.availability and Message.toolResponse each turns it red. It also exits 2 rather than 0 when it extracts nothing, so a check that examined nothing cannot pass for a clean one — that failure mode is why the first draft of this script looked fine while checking almost nothing. Release skill gains step 12d plus a Definition-of-Done line: map the change to the skill that covers it, then run the gate. The section says plainly what the gate cannot do — catch a symbol that still exists but changed meaning. getActiveStt(language:) went from a load-time property to a per-transcription one with no rename at all, and the script would have stayed green through it. Step 1d-bis needs no change: its repo-wide grep already covers packages/**/*.md, which is where the skills live. Verified rather than assumed. --- .claude/skills/release/SKILL.md | 46 ++++++++++++++++ tool/check_skills.sh | 97 +++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+) create mode 100755 tool/check_skills.sh diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 8c8b6c43..82662d99 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -52,6 +52,7 @@ silently do the other thing. [ ] 8 dart pub publish --dry-run → 0 warnings, every package [ ] 12a website + README version pins bumped to the just-published versions [ ] 12b new/changed public API + behavior documented (README + website) ← SAME PR +[ ] 12d skills/ updated for the change, and `bash tool/check_skills.sh` green [ ] 12c after merge: firebase-hosting-merge run == success (not just triggered) ``` @@ -615,6 +616,51 @@ Update each `^X.Y.Z` for the core packages (`flutter_gemma`, `flutter_gemma_lite - **Breaking changes / migrations** → `migration.md`. - **A bug class users hit** → `troubleshooting.md` (e.g. the #318 `maxTokens` vs `maxOutputTokens` confusion belongs here). +### 12d. Update the shipped agent skills — they are read by a MACHINE + +`packages/flutter_gemma/skills/` holds eight `SKILL.md` files that ship inside +the core archive and are installed into users' coding agents by +`dart run skills@ get`. They are not a nice-to-have copy of the docs: an agent +follows them literally when writing code against this package. + +That makes stale skills worse than stale docs. A human reading an outdated +README notices the mismatch; an agent does not — it writes confident, wrong code +against an API that moved, and the user blames the package. + +**If this release changed public API or behaviour, the skills change with it.** +Map the change to the skill that covers it: + +| Area | Skill | +|------|-------| +| registry, install, `ModelFileType`, `maxTokens`, sessions, chat | `flutter-gemma-inference` | +| function calling | `flutter-gemma-tools` | +| `.litertlm` engine, backends, platform floors | `flutter-gemma-litertlm` | +| `.task`/`.bin`, MediaPipe web | `flutter-gemma-mediapipe` | +| ONNX / ORT-GenAI | `flutter-gemma-onnx` | +| the OS built-in model | `flutter-gemma-builtin-ai` | +| STT, TTS, `VoiceSession` | `flutter-gemma-speech` | +| embeddings, vector stores | `flutter-gemma-rag` | + +Then run the gate: + +```bash +bash tool/check_skills.sh # exit 0 required +``` + +It extracts every API symbol the skills name — inline backticks, named arguments +inside ```dart fences, and dotted members — and fails if one no longer exists in +`packages/*/lib/`. Read the count it prints, not just the exit code: a run that +examined nothing exits 2 rather than reporting a pass. + +**What the gate cannot catch, and you must:** a symbol that still exists but +changed MEANING. `getActiveStt(language:)` went from "the language this +recognizer was built with" to "the default for its transcriptions" without a +single rename — the script stayed green through both. When a behaviour changes, +open the skill and read it. + +Skills live only in `flutter_gemma`, so a fix to any of them is one publish of +core. That is why they are all there rather than in the packages they describe. + ### 12c. Deploy — it's automatic on merge to main **You do NOT run a manual deploy.** `.github/workflows/firebase-hosting-merge.yml` auto-deploys to Firebase Hosting (`aichat-c0c27`, target `fluttergemma`, https://fluttergemma.dev → live channel) on every push to `main` that touches `website/**` or `packages/flutter_gemma/example/**`. So: diff --git a/tool/check_skills.sh b/tool/check_skills.sh new file mode 100755 index 00000000..8508ab1b --- /dev/null +++ b/tool/check_skills.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +# +# Every API symbol the shipped agent skills name must still exist in the +# workspace sources. +# +# Skills under packages/flutter_gemma/skills/ are read by a MACHINE, not a +# person. A renamed symbol does not make them look odd — it makes them +# confidently wrong, and the user's agent writes code against an API that is +# gone. Nothing else in the build notices: skills are markdown, so analyze, +# test and format all stay green. +# +# Three shapes are checked, because between them they cover where API names +# actually appear in a SKILL.md: +# +# inline `backticked` identifiers CamelCase, or camelCase with a capital +# named arguments in ```dart fences maxOutputTokens: +# dotted members in ```dart fences TaskType.retrievalQuery +# +# The first shape alone is not enough — it misses `maxOutputTokens` entirely, +# because that symbol appears in the skills only inside code blocks. Verified by +# mutation: renaming maxOutputTokens, supportsFunctionCalls, +# TaskType.retrievalQuery, BuiltInAi.availability or Message.toolResponse each +# turns this script red. +# +# What it CANNOT catch: a symbol that still exists but changed meaning. The STT +# language went from a load-time property to a per-transcription one without a +# single rename — for that class, read the skill. +# +# Usage, from the repo root: +# bash tool/check_skills.sh +# Exit 1 means a skill names something that no longer exists. + +set -uo pipefail + +SKILLS_DIR=packages/flutter_gemma/skills + +if [ ! -d "$SKILLS_DIR" ]; then + echo "no skills directory at $SKILLS_DIR — run from the repo root" >&2 + exit 2 +fi + +LIB=$(mktemp) +trap 'rm -f "$LIB"' EXIT +find packages -path '*/lib/*' -name '*.dart' -not -path '*/build/*' \ + -exec cat {} + > "$LIB" 2>/dev/null + +if [ ! -s "$LIB" ]; then + echo "collected no Dart sources — the check would pass vacuously" >&2 + exit 2 +fi + +# Not Dart API: build-config keys and prose that survives the shape filter. +SKIP=' minSdk platform dependencies ' + +total=0 +missing=0 + +for f in "$SKILLS_DIR"/*/SKILL.md; do + skill=$(basename "$(dirname "$f")") + code=$(awk '/^```dart/{c=1;next} /^```/{c=0} c' "$f") + + inline=$(grep -oE '`[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*`' "$f" \ + | tr -d '`' | grep -E '^[A-Z]|[a-z][A-Z]') + named=$(printf '%s' "$code" \ + | grep -oE '(^|[ (,])[a-z][A-Za-z0-9_]*:' | tr -d ' (,:') + dotted=$(printf '%s' "$code" \ + | grep -oE '\b[A-Z][A-Za-z0-9_]*\.[a-z][A-Za-z0-9_]*') + + for sym in $(printf '%s\n%s\n%s\n' "$inline" "$named" "$dotted" | sort -u); do + [ -z "$sym" ] && continue + case "$SKIP" in *" $sym "*) continue ;; esac + + total=$((total + 1)) + base="${sym%%.*}" + leaf="${sym##*.}" + + if ! grep -q "\b${base}\b" "$LIB"; then + echo " MISSING ${sym} (${skill})" + missing=$((missing + 1)) + elif [ "$base" != "$leaf" ] && ! grep -q "\b${leaf}\b" "$LIB"; then + echo " MISSING ${sym} (${skill})" + missing=$((missing + 1)) + fi + done +done + +skills_count=$(find "$SKILLS_DIR" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ') +echo "checked ${total} symbol(s) across ${skills_count} skill(s), ${missing} missing" + +# A run that examined nothing is indistinguishable from a clean run — refuse to +# report success in that case. +if [ "$total" -eq 0 ]; then + echo "extracted no symbols — the check is broken, not the skills" >&2 + exit 2 +fi + +[ "$missing" -eq 0 ] || exit 1 From 9047050a3078b11a1f83b63e5b25465ed14eba1c Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Thu, 10 Sep 2026 22:35:51 +0200 Subject: [PATCH 04/21] chore(release): route the diff to the skills that need re-reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A gate that checks names is not enough, and "read the skills carefully" in a checklist is not a step anyone performs. The missing piece was deciding WHICH skills a given release puts in doubt. tool/skills_review.sh takes the release range and, per skill, prints the API symbols that skill NAMES and this diff TOUCHED — added or removed lines only, so a symbol merely sitting near a change is not evidence. A skill with hits gets opened; a skill without gets skipped deliberately rather than forgotten. Validated against two real releases rather than reasoned about: 01801ef2 (STT language) flags flutter-gemma-speech with getActiveStt, language, SttModelType.whisper — exactly the skill whose prose the change invalidated a764009e (createChat flags flutter-gemma-tools, and leaves speech never passed tools) alone First run flagged 5 of 8 for an STT-only change, because String, StateError, name, text and must appear in every diff. Those carry no routing power, so they are filtered; it is 2 of 8 now. Flagging everything is the same as flagging nothing. Release 12d now reads: run the router, READ every flagged skill against the change, then run check_skills.sh. The section says which of the two scripts answers which question — names versus meaning — and that only reading closes the second. --- .claude/skills/release/SKILL.md | 33 ++++++++--- tool/skills_review.sh | 101 ++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+), 7 deletions(-) create mode 100755 tool/skills_review.sh diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 82662d99..43ed1b5c 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -52,7 +52,8 @@ silently do the other thing. [ ] 8 dart pub publish --dry-run → 0 warnings, every package [ ] 12a website + README version pins bumped to the just-published versions [ ] 12b new/changed public API + behavior documented (README + website) ← SAME PR -[ ] 12d skills/ updated for the change, and `bash tool/check_skills.sh` green +[ ] 12d skills/: `skills_review.sh ` run, every flagged skill READ, + updated where the prose drifted, and `check_skills.sh` green [ ] 12c after merge: firebase-hosting-merge run == success (not just triggered) ``` @@ -641,7 +642,24 @@ Map the change to the skill that covers it: | STT, TTS, `VoiceSession` | `flutter-gemma-speech` | | embeddings, vector stores | `flutter-gemma-rag` | -Then run the gate: +**Do not go looking by hand.** Ask the diff which skills it puts in doubt: + +```bash +bash tool/skills_review.sh # e.g. v1.8.0 +``` + +For each skill it prints the symbols that skill NAMES and this release TOUCHED. +Run against the STT release it names `flutter-gemma-speech` with +`getActiveStt`, `language`, `SttModelType.whisper`; against the +`createChat`-tools fix it names `flutter-gemma-tools` and leaves speech alone. +That is the routing — a skill with hits gets opened, a skill without one gets +skipped with a clear conscience. + +**Then open every flagged skill and read it against the change.** This is the +step, not the script. The script cannot tell whether the prose is still true; +it only says where to look. + +Finally the mechanical gate: ```bash bash tool/check_skills.sh # exit 0 required @@ -652,11 +670,12 @@ inside ```dart fences, and dotted members — and fails if one no longer exists `packages/*/lib/`. Read the count it prints, not just the exit code: a run that examined nothing exits 2 rather than reporting a pass. -**What the gate cannot catch, and you must:** a symbol that still exists but -changed MEANING. `getActiveStt(language:)` went from "the language this -recognizer was built with" to "the default for its transcriptions" without a -single rename — the script stayed green through both. When a behaviour changes, -open the skill and read it. +**Why both.** `check_skills.sh` answers "does every name still exist" — renames +and deletions. It stays green when a symbol survives and its MEANING moves, +which is the failure that actually happened here: `getActiveStt(language:)` went +from "the language this recognizer was built with" to "the default for its +transcriptions" with no rename anywhere. `skills_review.sh` is what puts that +change in front of your eyes; only reading closes it. Skills live only in `flutter_gemma`, so a fix to any of them is one publish of core. That is why they are all there rather than in the packages they describe. diff --git a/tool/skills_review.sh b/tool/skills_review.sh new file mode 100755 index 00000000..a3e42257 --- /dev/null +++ b/tool/skills_review.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# +# Which shipped skills does this release's diff put in doubt, and why. +# +# `check_skills.sh` answers "does every name still exist" — renames and +# deletions. It stays green when a symbol survives but changes MEANING, which is +# the failure that actually bit us: getActiveStt(language:) went from a +# load-time property of the recognizer to a per-transcription one, with no +# rename anywhere. +# +# Only reading catches that. This script decides WHAT to read: for each skill it +# lists the API symbols that skill names AND that this diff touched. A skill +# with hits is one you must open; a skill with none you can skip with a clear +# conscience. +# +# Usage, from the repo root: +# bash tool/skills_review.sh # e.g. v1.8.0, or origin/main +# bash tool/skills_review.sh v1.8.0 HEAD +# +# Exit 0 always — this routes attention, it does not pass or fail. + +set -uo pipefail + +SKILLS_DIR=packages/flutter_gemma/skills +FROM=${1:-} +TO=${2:-HEAD} + +if [ -z "$FROM" ]; then + echo "usage: bash tool/skills_review.sh [until-ref]" >&2 + echo " e.g. bash tool/skills_review.sh v1.8.0" >&2 + exit 2 +fi + +if [ ! -d "$SKILLS_DIR" ]; then + echo "no skills directory at $SKILLS_DIR — run from the repo root" >&2 + exit 2 +fi + +# Added/removed source lines only. A symbol that merely sits near a change is +# not evidence; a symbol on a +/- line is. +DIFF=$(mktemp) +trap 'rm -f "$DIFF"' EXIT +git diff "$FROM" "$TO" -- 'packages/*/lib/**' \ + | grep -E '^[+-]' | grep -vE '^(\+\+\+|---)' > "$DIFF" + +if [ ! -s "$DIFF" ]; then + echo "no source changes in packages/*/lib between $FROM and $TO" + echo "→ no skill needs re-reading on account of code" + exit 0 +fi + +echo "Skills to re-read for $FROM..$TO" +echo + +NOISE='String StateError ArgumentError UnsupportedError Exception name text description response spec chat must limit parameters type value' + +flagged=0 +for f in "$SKILLS_DIR"/*/SKILL.md; do + skill=$(basename "$(dirname "$f")") + code=$(awk '/^```dart/{c=1;next} /^```/{c=0} c' "$f") + + syms=$( + { + grep -oE '`[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*`' "$f" \ + | tr -d '`' | grep -E '^[A-Z]|[a-z][A-Z]' + printf '%s' "$code" | grep -oE '(^|[ (,])[a-z][A-Za-z0-9_]*:' | tr -d ' (,:' + printf '%s' "$code" | grep -oE '\b[A-Z][A-Za-z0-9_]*\.[a-z][A-Za-z0-9_]*' + } | sort -u + ) + + hits="" + for sym in $syms; do + [ -z "$sym" ] && continue + # Words with no routing power: dart:core types and identifiers so generic + # that they appear in almost any diff. Leaving them in flagged 5 skills of 8 + # for an STT-only release, which is the same as flagging none. + case " $NOISE " in *" $sym "*) continue ;; esac + leaf="${sym##*.}" + if grep -q "\b${leaf}\b" "$DIFF"; then + hits="${hits}${hits:+ }${sym}" + fi + done + + if [ -n "$hits" ]; then + flagged=$((flagged + 1)) + echo " ${skill}" + for h in $hits; do echo " touched: ${h}"; done + echo + fi +done + +if [ "$flagged" -eq 0 ]; then + echo " none — the diff touches no symbol any skill names" + echo +fi + +echo "${flagged} of $(find "$SKILLS_DIR" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ') skill(s) flagged." +echo +echo "A hit means the skill DESCRIBES something this release changed. Open it and" +echo "check the prose still matches the behaviour — the symbol existing is not" +echo "the same as the skill being right." From a9168b7bcef2d2d3179095301b3e94dedec496f9 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Fri, 11 Sep 2026 08:54:48 +0200 Subject: [PATCH 05/21] fix(skills): rewrite against the source, and gate them by compiling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four independent reviews of the eight drafts found code that would not compile or would fail at runtime in five of them: ModelType.gemma3 (no such value), a switch over the sealed ModelResponse without ThinkingResponse, rag.search(vector, limit:) and Condition.equals (the API is searchSimilar(query:, topK:) and FieldEquals), getActiveModel(spec:) (no such parameter), LiteRtEmbeddingBackend attributed to the wrong package, SqliteVectorStore on web (throws), filters without filterSchema (silently ignored), and ONNX installs without fileType (routed to MediaPipe). Plus wrong claims: an explicit gpu/npu backend does fall back, .litertlm clamps rather than crashes below 1024, ONNX and built-in AI do have web arms, moonshine-tiny is English-only. Every item was checked against the source before it was changed. Seven skills now: flutter-gemma-litertlm folded into inference, because litertlm is the default engine and "add offline chat" names no engine, so the default path must be self-contained; flutter-gemma-tools renamed to function-calling. Platform setup (Podfile, entitlements, web/index.html) moved to inference/references/platform-setup.md. tool/check_skills.dart replaces the grep gate. It compiles every dart fence as a function body and every inline Type / Type.member as a declaration, then runs dart analyze inside the example app. The grep gate was green on four nonexistent APIs — gemma3 matched a model URL, limit: an unrelated argument. Verified both ways: 29 errors on the broken drafts; clean on the rewrite, and six mutations of the new text each turn it red. Also: CLAUDE.md said built-in AI and ONNX have no web (the source the drafts copied), and website models.md repeated the moonshine claim. One claim was removed rather than fixed: platform-setup.md said a Release build fails without disable-library-validation. The README asks for it in both files, but the example has it in Debug only; that needs a macOS release build to settle and is recorded, not asserted. --- .claude/skills/release/SKILL.md | 30 ++- CLAUDE.md | 4 +- .../skills/flutter-gemma-builtin-ai/SKILL.md | 102 ++++---- .../flutter-gemma-function-calling/SKILL.md | 118 +++++++++ .../skills/flutter-gemma-inference/SKILL.md | 215 +++++++++------- .../references/platform-setup.md | 107 ++++++++ .../skills/flutter-gemma-litertlm/SKILL.md | 87 ------- .../skills/flutter-gemma-mediapipe/SKILL.md | 89 +++---- .../skills/flutter-gemma-onnx/SKILL.md | 101 ++++---- .../skills/flutter-gemma-rag/SKILL.md | 137 +++++----- .../skills/flutter-gemma-speech/SKILL.md | 182 +++++--------- .../skills/flutter-gemma-tools/SKILL.md | 112 --------- tool/check_skills.dart | 237 ++++++++++++++++++ tool/check_skills.sh | 97 ------- tool/skills_review.sh | 4 +- website/content/docs/models.md | 2 +- 16 files changed, 891 insertions(+), 733 deletions(-) create mode 100644 packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md create mode 100644 packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md delete mode 100644 packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md delete mode 100644 packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md create mode 100644 tool/check_skills.dart delete mode 100755 tool/check_skills.sh diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 43ed1b5c..73ff8ac1 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -53,7 +53,7 @@ silently do the other thing. [ ] 12a website + README version pins bumped to the just-published versions [ ] 12b new/changed public API + behavior documented (README + website) ← SAME PR [ ] 12d skills/: `skills_review.sh ` run, every flagged skill READ, - updated where the prose drifted, and `check_skills.sh` green + updated where the prose drifted, and `dart tool/check_skills.dart` green [ ] 12c after merge: firebase-hosting-merge run == success (not just triggered) ``` @@ -633,9 +633,8 @@ Map the change to the skill that covers it: | Area | Skill | |------|-------| -| registry, install, `ModelFileType`, `maxTokens`, sessions, chat | `flutter-gemma-inference` | -| function calling | `flutter-gemma-tools` | -| `.litertlm` engine, backends, platform floors | `flutter-gemma-litertlm` | +| registry, install, `ModelFileType`, `maxTokens`, sessions, chat, the `.litertlm` engine, backends, platform setup | `flutter-gemma-inference` (+ `references/platform-setup.md`) | +| function calling | `flutter-gemma-function-calling` | | `.task`/`.bin`, MediaPipe web | `flutter-gemma-mediapipe` | | ONNX / ORT-GenAI | `flutter-gemma-onnx` | | the OS built-in model | `flutter-gemma-builtin-ai` | @@ -651,7 +650,8 @@ bash tool/skills_review.sh # e.g. v1.8.0 For each skill it prints the symbols that skill NAMES and this release TOUCHED. Run against the STT release it names `flutter-gemma-speech` with `getActiveStt`, `language`, `SttModelType.whisper`; against the -`createChat`-tools fix it names `flutter-gemma-tools` and leaves speech alone. +`createChat`-tools fix it names the function-calling skill and leaves speech +alone. That is the routing — a skill with hits gets opened, a skill without one gets skipped with a clear conscience. @@ -662,16 +662,22 @@ it only says where to look. Finally the mechanical gate: ```bash -bash tool/check_skills.sh # exit 0 required +dart tool/check_skills.dart # exit 0 required ``` -It extracts every API symbol the skills name — inline backticks, named arguments -inside ```dart fences, and dotted members — and fails if one no longer exists in -`packages/*/lib/`. Read the count it prints, not just the exit code: a run that -examined nothing exits 2 rather than reporting a pass. +It COMPILES the skills: every ```dart fence becomes a function body, every +inline `Type` and `Type.member` in the prose becomes a declaration, and +`dart analyze` runs over the result inside the example app, which depends on +every package. A misspelt parameter, a method that moved, a switch that is no +longer exhaustive — all fail. Read the count it prints, not just the exit code: +a run that extracted nothing exits 2 rather than reporting a pass. -**Why both.** `check_skills.sh` answers "does every name still exist" — renames -and deletions. It stays green when a symbol survives and its MEANING moves, +It replaced a grep-based check that was green on four APIs that did not exist — +`gemma3` matched a model URL, `limit:` an unrelated argument. A text search +cannot tell "this name exists" from "this code is right". + +**Why both.** `check_skills.dart` answers "does this code still compile" — +renames, deletions, signature changes. It stays green when a symbol survives and its MEANING moves, which is the failure that actually happened here: `getActiveStt(language:)` went from "the language this recognizer was built with" to "the default for its transcriptions" with no rename anywhere. `skills_review.sh` is what puts that diff --git a/CLAUDE.md b/CLAUDE.md index ccff3291..31bd7663 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -311,7 +311,7 @@ flutter analyze && dart format . && tool/test_all.sh | `hook/build.dart` | Native Assets hook — fetches the per-platform `vec0` loadable extension | | `web/rag/sqlite3.wasm` | custom `sqlite3.wasm` with `sqlite-vec`/`vec0` statically linked (app copies to its web root) | -**`packages/flutter_gemma_builtin_ai/` (OS built-in AI; Gemini Nano on Android, Apple Foundation Models on iOS/macOS; no web/desktop):** +**`packages/flutter_gemma_builtin_ai/` (OS built-in AI; Gemini Nano on Android and desktop Chrome via the Prompt API, Apple Foundation Models on iOS/macOS; no Windows/Linux):** | File | Purpose | |------|---------| @@ -324,7 +324,7 @@ flutter analyze && dart format . && tool/test_all.sh | `android/src/.../` | Android ML Kit GenAI (AICore) native layer; declares `minSdk 26` | | `darwin/Classes/` (shared iOS+macOS source via `sharedDarwinSource: true`) | Apple Foundation Models native layer | -**`packages/flutter_gemma_onnx/` (ONNX Runtime — ORT-GenAI inference + plain-ORT embeddings; macOS arm64 only in v1, no web):** +**`packages/flutter_gemma_onnx/` (ONNX Runtime — ORT-GenAI inference + plain-ORT embeddings on macOS arm64 / Linux x64 / Windows x64 / Android arm64 / iOS arm64; web via Transformers.js + onnxruntime-web):** | File | Purpose | |------|---------| diff --git a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md index 704c130c..e9e1f36d 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md @@ -1,80 +1,76 @@ --- name: flutter-gemma-builtin-ai -description: Use when running the OS's own model through flutter_gemma_builtin_ai — Gemini Nano on Android or Apple Foundation Models on iOS/macOS. There is no file to download or install; the OS owns the weights, so availability must be probed at runtime and can legitimately be "not yet downloaded". +description: Use when running the device's own model with flutter_gemma_builtin_ai — Gemini Nano on Android or in desktop Chrome, Apple Foundation Models on iPhone, iPad and Mac — with nothing to download or bundle. Also use when BuiltInAiUnavailableException is thrown, availability reports "downloadable", the Android build fails the manifest merge on minSdk, or the model is missing in Chrome. For models you download yourself, use flutter-gemma-inference. --- # The built-in OS model -`flutter_gemma_builtin_ai` runs the model the operating system already ships: -Gemini Nano through ML Kit GenAI / AICore on Android, Apple Foundation Models on -iOS and macOS. +## Rules -```dart -await FlutterGemma.initialize(inferenceEngines: [BuiltInAiEngine()]); -``` - -## There is no model file +1. The OS owns the weights, but the model is still installed — as an identity: `fileType: ModelFileType.builtIn` with `.fromBundled(...)`. Nothing is downloaded by the app. +2. Call `BuiltInAi.ensureReady()` before `getActiveModel()`, from a user action: on Android the first call downloads the model and can take minutes. +3. Catch `BuiltInAiUnavailableException` and fall back to a downloadable model. +4. Android apps need `minSdk 26`, or the manifest merge fails. +5. There is no Windows or Linux support. -This is the difference from every other engine. `ModelFileType.builtIn` means -the OS owns the weights — nothing to download, nothing to bundle, no storage -budget, and `installModel` is not part of the flow. Ready-made specs are -provided: +## Setup ```dart -final model = await FlutterGemma.getActiveModel( - spec: BuiltInAiModels.geminiNano, // or .appleFoundationModels -); -``` +import 'package:flutter/foundation.dart'; +import 'package:flutter_gemma_builtin_ai/flutter_gemma_builtin_ai.dart'; + +await FlutterGemma.initialize(inferenceEngines: [BuiltInAiEngine()]); -The trade is that availability is not yours to control. +final spec = kIsWeb || defaultTargetPlatform == TargetPlatform.android + ? BuiltInAiModels.geminiNano + : BuiltInAiModels.appleFoundationModels; + +await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.builtIn, +).fromBundled(spec.name).install(); + +try { + await BuiltInAi.ensureReady(onProgress: (percent) => print('$percent%')); + final model = await FlutterGemma.getActiveModel(maxTokens: 4096); +} on BuiltInAiUnavailableException catch (e) { + print('No built-in model here: $e — fall back to a downloadable one'); +} +``` -## Probe availability before using it +## Checking before you offer the feature ```dart final availability = await BuiltInAi.availability(); +final usable = availability == BuiltInAiAvailability.available || + availability == BuiltInAiAvailability.downloadable || + availability == BuiltInAiAvailability.downloading; ``` -`BuiltInAiAvailability` has seven states, and three of them are not failures: +`downloadable` and `downloading` mean "not yet", not "no" — `ensureReady` finishes the job. The `unavailable*` states are final for this device: `unavailableDeviceUnsupported`, `unavailableOsTooOld`, `unavailableDisabled`, `unavailableOther`. -| State | Meaning | -| --- | --- | -| `available` | ready now | -| `downloadable` | supported, weights not fetched yet | -| `downloading` | fetch in progress | -| `unavailableDeviceUnsupported` | this hardware will never support it | -| `unavailableOsTooOld` | an OS upgrade would fix it | -| `unavailableDisabled` | turned off by the user or by policy | -| `unavailableOther` | something else | +## Platforms -Treat `downloadable` and `downloading` as "not yet", not as "no". `ensureReady` -triggers and awaits the download: +| Platform | Model | Needs | +| --- | --- | --- | +| Android | Gemini Nano (AICore) | Pixel 9+, Galaxy S25+; `minSdk 26` | +| iOS, macOS | Apple Foundation Models | iPhone 15 Pro+ or an Apple Silicon Mac, Apple Intelligence turned on | +| Web | Gemini Nano (Chrome Prompt API) | desktop Chrome or Edge only — not mobile browsers, Firefox or Safari | -```dart -await BuiltInAi.ensureReady(); -``` +Images work on Android. On Apple platforms they need OS 27 — on OS 26 an image throws. The web model is text-only. -That can take minutes on first use and needs network, so drive it from an -explicit user action with visible progress — never from app start. +## Web -An unusable state throws `BuiltInAiUnavailableException`. Catch it and fall back -to a downloadable model through another engine; do not let it reach the user as -a crash. +There is no script to add: the Prompt API is part of the browser. It has to be enabled. -## Android needs minSdk 26 +- Production — register the origin for the Prompt API origin trial and add the token to `web/index.html`: -ML Kit GenAI / AICore will not merge below API 26, so the manifest merger fails -at build time with a `uses-sdk:minSdkVersion` conflict. Raise `minSdk` to 26 in -`android/app/build.gradle(.kts)` for any app that includes this package. - -## Platforms +```html + +``` -Android, iOS and macOS only — **no web, no Windows, no Linux**. The engine -declines elsewhere rather than throwing, so a registry with another engine -registered still works. +- Local development — enable `chrome://flags/#prompt-api-for-gemini-nano` and restart Chrome. -## What you give up +## Trade-offs -The OS model is small and its behaviour is set by the platform: no choice of -weights, no LoRA, no control over quantisation, and capabilities that differ by -OS version. Use it when "zero download, zero disk" matters more than -capability; use `.litertlm` when you need a specific model. +No choice of weights, no LoRA, and capabilities that vary by OS version. Use it when zero download and zero disk matter more than picking the model. diff --git a/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md new file mode 100644 index 00000000..d1d69e01 --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md @@ -0,0 +1,118 @@ +--- +name: flutter-gemma-function-calling +description: Use when adding function calling (tool calling) to a flutter_gemma chat — letting an on-device model call the app's own functions, declaring Tool objects, handling FunctionCallResponse, returning results with Message.toolResponse, or running the built-in tool loop. Also use when the model describes an action in prose instead of calling the tool, or a switch over ModelResponse fails to compile. For plain chat, use flutter-gemma-inference. +--- + +# Function calling with flutter_gemma + +## Rules + +1. `createChat` needs three arguments: `tools`, `supportsFunctionCalls: true` and `modelType`. Without the flag the tools are dropped with a debug-only warning; without the type the call format cannot be derived. +2. Switch over all four `ModelResponse` subtypes. It is sealed — a switch that leaves out `ThinkingResponse` does not compile. +3. Return tool results as data, errors included. Never throw from a tool. +4. Prefer `generateChatResponseWithTools` to a hand-written loop. +5. Use a tool-capable model: Gemma 4, Gemma 3 1B, FunctionGemma, Phi-4 Mini, Qwen 2.5, Qwen3, DeepSeek R1. Gemma 3 270M and SmolLM cannot call tools. + +## Declare a tool + +`parameters` is a JSON Schema object. The model matches the user's intent against `description`, so write it as an action and describe every parameter. + +```dart +const changeColor = Tool( + name: 'change_color', + description: 'Change the app background colour.', + parameters: { + 'type': 'object', + 'properties': { + 'color': {'type': 'string', 'description': 'A colour name, e.g. red.'}, + }, + 'required': ['color'], + }, +); +``` + +## Open the chat + +```dart +final chat = await model.createChat( + tools: myTools, + supportsFunctionCalls: true, + modelType: ModelType.gemma4, +); +``` + +## The built-in loop + +It calls your handler for each tool call, feeds the result back, and continues until the model answers in text or `maxToolTurns` is reached. + +```dart +await chat.addQueryChunk(Message(text: prompt, isUser: true)); +await for (final r in chat.generateChatResponseWithTools( + onToolCall: (call) => runTool(call.name, call.args), + maxToolTurns: 8, +)) { + if (r is TextResponse) stdout.write(r.token); +} +``` + +## Handling calls yourself + +```dart +final response = await chat.generateChatResponse(); +switch (response) { + case FunctionCallResponse(:final name, :final args): + final result = await runTool(name, args); + await chat.addQueryChunk( + Message.toolResponse(toolName: name, response: result), + ); + final followUp = await chat.generateChatResponse(); + case ParallelFunctionCallResponse(:final calls): + for (final call in calls) { + final result = await runTool(call.name, call.args); + await chat.addQueryChunk( + Message.toolResponse(toolName: call.name, response: result), + ); + } + final afterAll = await chat.generateChatResponse(); + case TextResponse(:final token): + print(token); // the model chose to answer directly — a valid outcome + case ThinkingResponse(): + break; +} +``` + +## Errors are results + +```dart +await chat.addQueryChunk( + Message.toolResponse( + toolName: 'change_color', + response: {'error': 'unknown colour: mauvish'}, + ), +); +``` + +The model can recover from an error it can read. An exception thrown out of a tool ends the turn instead. + +## Traps + +**Model answers in prose** +- Symptom: "I would change the colour to red" instead of a call. +- Fix: check `supportsFunctionCalls: true` and `modelType`, then check the model is tool-capable. + +**Raw markers in the text** +- Symptom: `<|tool_call>` or `` appears in `TextResponse` tokens. +- Fix: `modelType` does not match the installed model. + +## Web + +Function calling works on the `.litertlm` web engine. Close each chat before creating the next one: a chat left open makes the following one fail with `Invalid token at state N`. + +```dart +await chat.close(); +final next = await model.createChat( + tools: myTools, + supportsFunctionCalls: true, + modelType: ModelType.gemma4, +); +``` diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md index 11614485..430ae42a 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md @@ -1,93 +1,89 @@ --- name: flutter-gemma-inference -description: Use whenever writing flutter_gemma code — installing a model, calling FlutterGemma.initialize/getActiveModel/createSession, or generating text. Core registers no engine by default, maxTokens is the context window and NOT the reply length, and Message.isUser defaults to false; all three fail quietly. +description: Use when adding on-device LLM inference to a Flutter app with flutter_gemma — offline chat, running Gemma, Qwen or Phi locally, streaming replies, image prompts — or setting up the default .litertlm engine on Android, iOS, macOS, Windows, Linux or web. Also use when a reply comes back empty, maxTokens does not shorten replies, getActiveModel throws "No inference engine can handle this model", or .litertlm fails to load on Android. For function calling, RAG, speech, .task files, ONNX or the OS built-in model, also use the matching flutter-gemma-* skill. --- # Running a model with flutter_gemma -This is the path from an empty app to a generated token. Four defaults on it -produce wrong behaviour rather than an error — start with those. +## Rules -## 1. Core ships no engine — register one +1. Register an engine in `FlutterGemma.initialize(inferenceEngines: [...])`. Core ships none. +2. Declare `fileType` on `installModel`. It defaults to `ModelFileType.task`, and the declaration — never the file name — picks the engine. +3. `maxTokens` is the context window. Cap the reply with `maxOutputTokens` on the session. +4. Pass `isUser: true` on every user `Message`. +5. Close every session, chat and model in a `finally`. +6. Never put a Hugging Face token in source. Read it with `String.fromEnvironment`. +7. On Android, set `minSdk 30` for anything built on `.litertlm` — inference, embeddings, speech. -`flutter_gemma` is the contracts, the registry and the platform shells. It has -no inference runtime. Adding only `flutter_gemma` compiles fine and throws on -the first `getActiveModel()`. - -| Package | Handles | -| --- | --- | -| `flutter_gemma_litertlm` | `.litertlm` — the main path, all six platforms | -| `flutter_gemma_mediapipe` | `.task`, `.bin` — mobile + web | -| `flutter_gemma_builtin_ai` | the OS model, no file to install | -| `flutter_gemma_onnx` | ONNX / ORT-GenAI | +## Setup — the default engine (.litertlm) ```dart -import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; -await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); -``` - -Every capability is opt-in the same way and defaults to an empty list — -`embeddingBackends`, `sttBackends`, `ttsBackends`, `huggingFaceResolvers`. If a -list is empty, the matching first call throws a `StateError` naming the package -to add. Read that message rather than guessing. +const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN'); -## 2. The engine is chosen by the DECLARED file type, never the filename - -`installModel` defaults to `ModelFileType.task`. A `.litertlm` file installed -without declaring its type is routed to MediaPipe, which cannot read it. - -```dart -// WRONG — the name says .litertlm, the declaration says .task, -// and the declaration is what routes it. -await FlutterGemma.installModel(modelType: ModelType.gemma4) - .fromNetwork(url).install(); +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], + huggingFaceToken: hfToken.isEmpty ? null : hfToken, // gated repos only +); -// RIGHT await FlutterGemma.installModel( modelType: ModelType.gemma4, fileType: ModelFileType.litertlm, ).fromNetwork(url).install(); + +final model = await FlutterGemma.getActiveModel(maxTokens: 1024); ``` -`modelType` is a separate axis: it drives the chat template and the model's -capabilities (`gemma4`, `gemma3`, `qwen3`, `deepSeek`, `general`, …). Getting it -wrong also fails quietly — the model generates, with the wrong prompt format. +Build with `--dart-define=HUGGINGFACE_TOKEN=hf_...` when the model repo is gated. -Sources: `.fromNetwork(url, token:)`, `.fromAsset(path)`, `.fromFile(file)`, and -`.fromHuggingFace(repo)` when a resolver is registered. +When a Hugging Face repo publishes a deployment manifest, one call picks the variant and its tested runtime settings. The engine carries its own resolver, so registering `LiteRtLmEngine` is enough: -## 3. maxTokens is the CONTEXT WINDOW, not the reply length +```dart +final install = await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.litertlm, +).fromHuggingFace('litert-community/LFM2.5-230M').install(); -`maxTokens` is the whole KV-cache budget: system prompt + history + the current -message + everything generated. It is not "how long the answer may be". +final model = await FlutterGemma.getActiveModel(defaults: install.runtime); +``` -Setting it small to get a short answer is the most common mistake with this -package, and on `.litertlm` it does not truncate — it crashes, with a message -naming an internal executor file and nothing else. +Other sources on the same builder: `.fromAsset(path)` for a model bundled in the app, `.fromFile(path)` for one already on disk, `.fromBundled(name)` for a platform-bundled resource. + +`modelType` sets the chat template. Gemma 3 and Gemma 3n are `ModelType.gemmaIt` — there is no `gemma3`. The full set: `general`, `gemmaIt`, `gemma4`, `deepSeek`, `qwen`, `qwen3`, `llama`, `hammer`, `functionGemma`, `phi`. A wrong type still generates, with the wrong prompt format. + +## Traps + +**No engine registered** +- Symptom: `StateError: No inference engine can handle this model (ModelFileType.litertlm). Add the engine package to pubspec.yaml and pass it in inferenceEngines: of FlutterGemma.initialize(...)` +- Fix: add the engine package and register its provider — or fix `fileType` if the wrong engine is registered. + +**`maxTokens` used as a reply length** ```dart -// WRONG — meant "a 100-token reply", actually a 100-token context. +// WRONG — asks for a 100-token context, not a 100-token reply final model = await FlutterGemma.getActiveModel(maxTokens: 100); +``` + +- Symptom: replies are as long as ever. On `.litertlm` the value is raised to 1024, the smallest context those models support, and only a debug-mode log says so. +- Fix: -// RIGHT — roomy context, capped output. +```dart final model = await FlutterGemma.getActiveModel(maxTokens: 1024); final session = await model.createSession(maxOutputTokens: 100); ``` -Use 4096+ for vision or audio — one image is worth hundreds of tokens. - -## 4. Message.isUser defaults to false +Use 4096 or more with images or audio — one image costs hundreds of tokens. -```dart -const Message(text: 'Hello') // WRONG — empty response -const Message(text: 'Hello', isUser: true) // RIGHT -``` +**`isUser` left out** +- Symptom: an empty response, no error. `Message.isUser` defaults to `false`, so the prompt is read as the model's own turn. +- Fix: `Message(text: prompt, isUser: true)`. -No error is raised. The response is just empty. +**Same reply every time** +- Symptom: identical output for identical input. `createSession` defaults to `topK: 1`, which is greedy decoding. +- Fix: pass `topK` (e.g. 40) and a `temperature`. -## Generating +## Generate ```dart final session = await model.createSession( @@ -96,28 +92,30 @@ final session = await model.createSession( maxOutputTokens: 256, ); try { - await session.addQueryChunk( - const Message(text: 'Explain isolates briefly.', isUser: true), - ); - final answer = await session.getResponse(); - // streaming: await for (final chunk in session.getResponseAsync()) … + await session.addQueryChunk(Message(text: prompt, isUser: true)); + final reply = await session.getResponse(); } finally { await session.close(); } -await model.close(); ``` -Sessions and models hold native resources — an isolate, a compiled model, GPU -buffers. Always close them, in a `finally`. +Streaming: -## Multi-turn +```dart +await session.addQueryChunk(Message(text: prompt, isUser: true)); +await for (final token in session.getResponseAsync()) { + stdout.write(token); +} +``` + +To stop early, call `await session.stopGeneration()`. Cancelling the stream subscription detaches Dart but does not stop native decoding on every engine. -`InferenceChat` keeps history and applies the model's chat template: +## Multi-turn chat ```dart final chat = await model.createChat(tokenBuffer: 256, maxOutputTokens: 512); try { - await chat.addQueryChunk(const Message(text: 'Hi', isUser: true)); + await chat.addQueryChunk(Message(text: prompt, isUser: true)); final response = await chat.generateChatResponse(); if (response is TextResponse) print(response.token); } finally { @@ -125,41 +123,76 @@ try { } ``` -`generateChatResponse()` returns a `ModelResponse`, not a `String`. With tools -enabled it may be a function call — see the `flutter-gemma-tools` skill. +`generateChatResponse()` returns a sealed `ModelResponse`: `TextResponse`, `FunctionCallResponse`, `ParallelFunctionCallResponse` or `ThinkingResponse`. + +## Two conversations at once -Thinking models (Qwen3, DeepSeek R1) emit `` blocks; pass -`isThinking: true` to surface them as `ThinkingResponse`, or `false` to have -them stripped. +`createSession` and `createChat` fill a single slot on the model: a second call replaces the first, and the two chats corrupt each other. For concurrent conversations use `openSession` / `openChat`, and close each one. + +```dart +final summariser = await model.openChat(); +final assistant = await model.openChat(); +try { + await summariser.addQueryChunk(Message(text: chunk, isUser: true)); + await assistant.addQueryChunk(Message(text: question, isUser: true)); +} finally { + await summariser.close(); + await assistant.close(); +} +``` -## Multimodal +## Thinking models -Declare support at model creation, then attach bytes: +Gemma 4, Qwen3 and DeepSeek R1 can emit reasoning. Pass `isThinking: true` to `createChat`. Reasoning arrives as `ThinkingResponse` only from `generateChatResponseAsync()`; `generateChatResponse()` strips it. Not available on web. ```dart -final model = await FlutterGemma.getActiveModel( - maxTokens: 4096, - supportImage: true, -); -await session.addQueryChunk( +final chat = await model.createChat(isThinking: true, modelType: ModelType.qwen3); +await chat.addQueryChunk(Message(text: question, isUser: true)); +await for (final r in chat.generateChatResponseAsync()) { + switch (r) { + case ThinkingResponse(:final content): + print('reasoning: $content'); + case TextResponse(:final token): + stdout.write(token); + case FunctionCallResponse() || ParallelFunctionCallResponse(): + break; + } +} +await chat.close(); +``` + +## Images + +```dart +final model = await FlutterGemma.getActiveModel(maxTokens: 4096, supportImage: true); +final chat = await model.createChat(supportImage: true); +await chat.addQueryChunk( Message(text: 'What is in this photo?', isUser: true, imageBytes: bytes), ); ``` -## Cost model +## The model is a singleton + +`getActiveModel` returns one model per process. Calling it again with different runtime arguments rebuilds it and closes the previous one — a handle you still hold stops working. Load once at startup, then create and close sessions per interaction. + +## Backends -Loading a model is expensive; creating a session is cheap. Load once, keep the -model, create and close a session per interaction. Never call `getActiveModel` -per message. +```dart +final model = await FlutterGemma.getActiveModel( + maxTokens: 1024, + preferredBackend: PreferredBackend.gpu, +); +print(model.activeBackend); // what actually loaded +``` + +| `preferredBackend` | Tried in order | +| --- | --- | +| `null` or `gpu` | GPU, then CPU | +| `npu` | NPU, GPU, CPU | +| `cpu` | CPU only | -`getActiveModel` returns a process-wide singleton. Calling it again with -different runtime arguments rebuilds it and closes the previous instance — a -handle you are still holding becomes unusable. Decide the runtime configuration -once. +Read `activeBackend` rather than assuming the requested one loaded. NPU needs a Snapdragon (Android) or Intel Lunar/Panther Lake (Windows). The iOS Simulator is CPU-only; web is GPU-only. -## Engine-specific rules +## Platform setup -Platform floors, backend selection and format quirks live with each engine: -`flutter-gemma-litertlm`, `flutter-gemma-mediapipe`, `flutter-gemma-onnx`, -`flutter-gemma-builtin-ai`. Read the one for the engine in use — this skill -covers only what is common to all of them. +Android needs `minSdk 30` and ships `arm64-v8a` only. iOS and macOS need Podfile and entitlement entries; web needs script tags in `web/index.html`. Read `references/platform-setup.md` before building for iOS, macOS or web — without those entries the model fails to load or the app runs out of memory. diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md b/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md new file mode 100644 index 00000000..38e19d1a --- /dev/null +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md @@ -0,0 +1,107 @@ +# Platform setup for flutter_gemma + +Entries each platform needs before a model will load. Without them the app +builds and then fails at model load, or is killed for memory. + +## Android + +`android/app/build.gradle.kts` (or `build.gradle`): + +``` +android { + defaultConfig { + minSdk = 30 + } +} +``` + +`minSdk 30` covers everything built on `.litertlm`: inference, embeddings and +speech. On API 29 the native library fails to load at runtime — the build does +not catch it. MediaPipe `.task` models run on lower API levels. + +Only `arm64-v8a` is shipped for `.litertlm`. The OpenCL manifest entries the GPU +backend needs are merged in by the plugin; nothing to add. + +## iOS + +`ios/Podfile`, declared once: + +```ruby +platform :ios, '15.0' # '16.0' if the app includes flutter_gemma_mediapipe +use_frameworks! :linkage => :static +``` + +`ios/Runner/Runner.entitlements` — without these, large models are killed for +memory: + +```xml +com.apple.developer.kernel.extended-virtual-addressing + +com.apple.developer.kernel.increased-memory-limit + +``` + +The iOS Simulator cannot run GPU inference; use CPU there, or a real device. + +## macOS + +Add to both `macos/Runner/DebugProfile.entitlements` and +`macos/Runner/Release.entitlements`: + +```xml +com.apple.security.cs.disable-library-validation + +com.apple.security.network.client + +com.apple.developer.kernel.extended-virtual-addressing + +com.apple.developer.kernel.increased-memory-limit + +``` + +`disable-library-validation` lets the app load the bundled native frameworks; +`network.client` lets it download the model. Add them to both files — the debug +and release builds read different ones. + +## Windows and Linux + +Nothing to add. The native libraries — including the Windows GPU shader compiler +and NPU runtime — are bundled at build time. + +## Web + +All script tags go in `web/index.html` ``, before Flutter boots. + +`.litertlm` engine: + +```html + +``` + +Model storage helpers. Copy `cache_api.js` and `opfs_helper.js` from the +`flutter_gemma` package's `web/` directory into the app's `web/`, then: + +```html + + +``` + +Find the package directory with +`grep -A1 '"name": "flutter_gemma"' .dart_tool/package_config.json`. + +Storage mode, set in `FlutterGemma.initialize(webStorageMode: ...)`: + +| `WebStorageMode` | Use for | +| --- | --- | +| `cacheApi` (default) | models under about 2 GB | +| `streaming` | larger models — streams through OPFS | +| `none` | no persistence; downloads every launch | + +Web is GPU-only. The `.litertlm` web engine is text-only: no images, audio, +thinking or LoRA. diff --git a/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md deleted file mode 100644 index 4667fe0a..00000000 --- a/packages/flutter_gemma/skills/flutter-gemma-litertlm/SKILL.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -name: flutter-gemma-litertlm -description: Use when the app runs .litertlm models through flutter_gemma_litertlm — the default engine on all six platforms. Covers the 1024-token KV cache floor that crashes below it, the Android minSdk 30 requirement, and backend selection. ---- - -# The .litertlm engine - -`flutter_gemma_litertlm` is the main inference path: Dart FFI straight into the -LiteRT-LM C API on Android, iOS, macOS, Windows and Linux, plus a web arm. No -JVM, no gRPC, no separate process. - -```dart -await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); - -await FlutterGemma.installModel( - modelType: ModelType.gemma4, - fileType: ModelFileType.litertlm, // required — the default is `task` -).fromNetwork(url).install(); -``` - -## maxTokens below 1024 crashes, it does not truncate - -Every supported `.litertlm` model bakes `kv_cache_max_len = 1024`. A `maxTokens` -below that underflows the native KV-cache resize and tensor allocation fails at -generation with a message naming an internal executor file. - -Measured on a Pixel 8a (CPU): 100 / 256 / 512 crash; 1024 and 4096 work. - -The engine now clamps values below 1024 upward and logs a warning, but do not -rely on that. Pass a real context size and cap the reply with -`maxOutputTokens` on the session — that one is `.litertlm`-only and is the -correct knob for reply length. - -## Android needs minSdk 30 - -`libLiteRtLm.so` uses `pthread_cond_clockwait` and `sem_clockwait`, which are -API-30-only Bionic symbols. On API 29 it fails at `dlopen`, not at build time. -Set `minSdk 30` in `android/app/build.gradle(.kts)` for any app that runs -`.litertlm`. - -Only `arm64-v8a` is shipped. A build implying other ABIs produces an APK -without the native library. - -## Backends - -```dart -final model = await FlutterGemma.getActiveModel( - maxTokens: 1024, - preferredBackend: PreferredBackend.gpu, -); -``` - -| | Android | iOS | Desktop | Web | -| --- | --- | --- | --- | --- | -| `cpu` | yes | yes | yes | no | -| `gpu` | yes | device only | yes | required | -| `npu` | Snapdragon | no | Windows, Intel Lunar/Panther Lake | no | - -Passing an explicit backend disables fallback — if that backend cannot load the -model, it fails rather than quietly trying another. Leaving it null tries GPU -then CPU. Some models bake a constraint: the 12B build declares -`section_backend_constraint: gpu` and an explicit `cpu` on it simply fails. - -**iOS Simulator is CPU-only.** Metal there has a 256 MB single-allocation cap -and model weights exceed it. Test GPU on a real device. - -## Desktop is `.litertlm` only - -There is no `.task` support on macOS, Windows or Linux. Windows GPU needs -`dxil.dll` and `dxcompiler.dll`, and Windows NPU needs Intel Lunar/Panther Lake -silicon — both ship inside the bundled native archive, nothing to install. - -## Native libraries are fetched at build time - -`hook/build.dart` downloads a per-platform archive from a pinned GitHub release -and verifies its SHA256 (Native Assets). Nothing is committed to the repo and -nothing ships in the pub package, so a first build needs network access. - -If a build fails on a missing symbol after upgrading, the shared-bundle owner -marker went stale: delete -`~/Library/Caches/flutter_gemma/native/.flutter_gemma_native_version` and run -`flutter clean`. - -## Web is an early preview - -Text only. No vision, audio, thinking, function calling or LoRA on the -`.litertlm` web arm — those work on native. Web is also GPU-only. diff --git a/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md index 92767361..8aa3fc37 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md @@ -1,53 +1,70 @@ --- name: flutter-gemma-mediapipe -description: Use when the app runs .task or .bin models through flutter_gemma_mediapipe — mobile and web only, no desktop. Covers the iOS 16 floor this package alone imposes, the web script tags, and why maxOutputTokens is ignored here. +description: Use when running .task or .bin models (MediaPipe GenAI) with flutter_gemma_mediapipe on Android, iOS or web. Also use when CocoaPods rejects the iOS platform version, images are ignored in a MediaPipe chat, or maxOutputTokens has no effect. MediaPipe has no macOS, Windows or Linux support — use a .litertlm model there (flutter-gemma-inference). --- # The MediaPipe engine -`flutter_gemma_mediapipe` runs `.task` and `.bin` models through MediaPipe -GenAI on Android, iOS and web. **There is no desktop support** — macOS, Windows -and Linux need `.litertlm`. +## Rules + +1. Declare `fileType: ModelFileType.task` for `.task` files and `ModelFileType.binary` for `.bin` files. +2. An app that includes this package needs iOS 16.0. +3. There is no desktop support. +4. `maxOutputTokens` is ignored. Stop generation with `session.stopGeneration()`. +5. `createChat` does not inherit image support from the model — pass `supportImage: true` to the chat as well. + +## Setup ```dart +import 'package:flutter_gemma_mediapipe/flutter_gemma_mediapipe.dart'; + await FlutterGemma.initialize(inferenceEngines: [MediaPipeEngine()]); await FlutterGemma.installModel( - modelType: ModelType.gemma3, - fileType: ModelFileType.task, // the default, but say it anyway + modelType: ModelType.gemmaIt, + fileType: ModelFileType.task, ).fromNetwork(url).install(); ``` -`.task` is the default `fileType`, so this is the one engine where forgetting to -declare it happens to work. Declare it regardless — it documents intent, and a -later switch to `.litertlm` then fails loudly instead of silently routing here. - -## This package alone requires iOS 16 - -Core, litertlm, embeddings and builtin_ai all build from iOS 15.0. MediaPipe -GenAI raises the floor to **16.0** for the whole app: +`ios/Podfile`, once: ```ruby platform :ios, '16.0' -use_frameworks! :linkage => :static ``` -If the app does not use `.task` models, drop this package and stay on 15. +Core and the other engines build from iOS 15.0. If the app does not use `.task` models, leave this package out and stay on 15. -## maxOutputTokens is ignored +## Images -MediaPipe has no session-level output cap. Passing `maxOutputTokens` logs that -it was ignored and generation runs to the model's own limit. To bound output -here, stop consuming the stream yourself. +```dart +final model = await FlutterGemma.getActiveModel(maxTokens: 4096, supportImage: true); +final chat = await model.createChat(supportImage: true); +await chat.addQueryChunk( + Message(text: 'Describe this image.', isUser: true, imageBytes: bytes), +); +``` -`maxTokens` still means the context window, and MediaPipe tolerates small values -rather than crashing — unlike `.litertlm`. That difference is a property of the -engine, not of the API. +Without `supportImage: true` on the chat, the image is dropped and the model answers the text alone. Audio input is not supported on MediaPipe. + +## Bounding output + +```dart +final session = await model.createSession(); +await session.addQueryChunk(Message(text: prompt, isUser: true)); +var produced = 0; +await for (final token in session.getResponseAsync()) { + stdout.write(token); + if (++produced >= 200) { + await session.stopGeneration(); + break; + } +} +await session.close(); +``` ## Web -The web arm needs the MediaPipe runtime loaded before Flutter starts. Add to -`web/index.html`: +Load the MediaPipe runtime in `web/index.html` before Flutter starts: ```html ``` -Pin the version. An unpinned CDN import takes whatever published last, which has -shipped broken before. +Pin the version — an unpinned import takes whatever was published last. -Web is **GPU-only** — there is no CPU backend for MediaPipe in the browser — and -large models need the streaming storage mode: +Web is GPU-only. Models over about 2 GB need OPFS streaming storage: ```dart await FlutterGemma.initialize( - webStorageMode: WebStorageMode.streaming, // OPFS; cacheApi caps near 2 GB + webStorageMode: WebStorageMode.streaming, inferenceEngines: [MediaPipeEngine()], ); ``` -The runtime JS files (`cache_api.js`, `opfs_helper.js`) are not injected -automatically — copy them into the app's own `web/` directory. +The storage modes need helper scripts in `web/` — see `references/platform-setup.md` in the flutter-gemma-inference skill. ## Android -The plugin ships its own native layer and its own pigeon; nothing to configure -beyond the usual OpenCL `` entries, which the core plugin's -manifest merges in for you. - -Only `arm64-v8a` is shipped. - -## Vision - -Multimodal `.task` models work on Android, iOS and web. Audio input does not — -that is `.litertlm` only, and only on native. +Text inference runs on `arm64-v8a`, `x86_64` and `armeabi-v7a`. No manifest changes are needed. diff --git a/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md index 8aeb19eb..f494f885 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md @@ -1,72 +1,71 @@ --- name: flutter-gemma-onnx -description: Use when running ONNX models through flutter_gemma_onnx — ORT-GenAI text generation or ORT embeddings. An ORT-GenAI model is a DIRECTORY, not a single file, so the ordinary single-file network install does not apply; the package is also gated to five specific host architectures. +description: Use when running ONNX models with flutter_gemma_onnx — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. Not for .litertlm or .task models. --- # The ONNX engine -`flutter_gemma_onnx` provides two things over `dart:ffi` in a worker isolate: -text generation via ORT-GenAI (`OnnxEngine`) and embeddings via plain ORT -(`OnnxEmbeddingBackend`). +## Rules + +1. Declare `fileType: ModelFileType.onnx`. Without it the install defaults to `task` and never reaches `OnnxEngine`. +2. An ORT-GenAI model is a directory — `genai_config.json`, the `.onnx` graph, its weights and a tokenizer. Install it with `fromHuggingFace(repo)`, which downloads the whole folder, or point `fromFile` at a local `genai_config.json`. A single-file download or a Flutter asset cannot produce it. +3. Native generation runs on macOS arm64, Linux x64, Windows x64, Android arm64 and iOS arm64. Anywhere else no engine accepts the model and `getActiveModel` throws `No inference engine can handle this model`. +4. Android needs `minSdk 24`. Phi-3.5-mini peaks near 3.7 GB of RAM — target 8 GB devices. +5. Text only: no images, no audio, no LoRA. + +## Setup ```dart +import 'package:flutter_gemma_onnx/flutter_gemma_onnx.dart'; + await FlutterGemma.initialize( inferenceEngines: [OnnxEngine()], embeddingBackends: [OnnxEmbeddingBackend()], ); -``` -## An ORT-GenAI model is a DIRECTORY +await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.onnx, +).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install(); -This is the difference that breaks the usual mental model. The model is not one -file: - -``` -phi-3.5-mini/ - genai_config.json - model.onnx - model.onnx_data # weights, often several GB - tokenizer.json +final model = await FlutterGemma.getActiveModel(maxTokens: 4096); ``` -`OnnxEngine.createModel` takes that directory's `genai_config.json` and loads -the **parent directory**. The single-file `.fromNetwork(url)` install used for -`.litertlm` and `.task` does not cover this — the files must arrive together, -by bundling them as assets or fetching them into one directory yourself. - -Embeddings are the exception: a plain `.onnx` embedding model is a single file -and installs normally. - -## Only five host architectures +A repo with several execution-provider folders resolves to its CPU/mobile folder automatically — the bundled runtime is CPU-only. -`OnnxEngine.canHandle` is gated to **macOS arm64, Linux x64, Windows x64, -Android arm64 and iOS arm64**, in lockstep with the build hook that bundles the -native archives. Anywhere else it declines and logs why, so the registry falls -through to another engine rather than failing at load. +A bundle you ship yourself: -`OnnxEmbeddingBackend.canHandle` stays extension-based on every platform — so -that LiteRT's catch-all cannot silently claim an `.onnx` file — and gates inside -`createModel` instead. The error therefore arrives at model creation, not at -registration. - -## Memory - -Phi-3.5-mini 3.8B int4 peaks around **3.74 GB RSS**. That needs a 6 GB+ phone; -below that the OS kills the app during load rather than reporting an error you -can catch. - -Measured throughput: macOS M4 Pro ~54 tok/s, Pixel 8 Pro ~10.4 tok/s, Linux -~5.3-5.8 tok/s, Windows ~3.3 tok/s on CPU test VMs. Treat ONNX as the -portability option, not the fast one — `.litertlm` is faster where both run. - -## No web +```dart +await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.onnx, +).fromFile('$path/genai_config.json').install(); +``` -There is no web arm. Use `.task` through MediaPipe or `.litertlm` in the -browser. +## Web + +On web `OnnxEngine` runs the model through Transformers.js. The model is a Hugging Face repo id (e.g. `onnx-community/Qwen2.5-0.5B-Instruct`) that the browser downloads and caches on first use. `PreferredBackend.cpu` forces WASM; anything else tries WebGPU first. + +Add to `web/index.html` ``, before Flutter boots — the first script for generation, the second for embeddings: + +```html + + +``` -## Native libraries +## Embeddings -Fetched at build time from Microsoft's own GitHub releases by the package's -`hook/build.dart` (Native Assets), not from a repo tag. On iOS the ORT runtime -is statically linked into the GenAI framework, so there is one binary rather -than two. +`OnnxEmbeddingBackend` handles single-file `.onnx` embedding models, installed with `FlutterGemma.installEmbedder()` like any other — see the flutter-gemma-rag skill for the indexing flow. diff --git a/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md index 8415f9ae..4baff760 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md @@ -1,94 +1,109 @@ --- name: flutter-gemma-rag -description: Use when building retrieval over on-device documents with flutter_gemma — embeddings via flutter_gemma_embeddings plus a vector store (rag_sqlite or rag_qdrant). Query and document embeddings need DIFFERENT TaskType prefixes or retrieval quality collapses, and embedding is CPU-only by design. +description: Use when adding RAG, semantic search or text embeddings to a flutter_gemma app — searching the user's documents on-device, an embedding model plus a vector store (flutter_gemma_rag_sqlite or flutter_gemma_rag_qdrant). Also use when a metadata filter returns unfiltered results, retrieval quality is poor, addDocument throws about a missing embedding model, or the vector store throws UnimplementedError on web. --- # On-device RAG with flutter_gemma -Three pieces: an embedding model, a vector store, and the retrieval call. All -three are opt-in packages. +## Rules + +1. Use the `FlutterGemma.rag` facade: `initialize`, `addDocument`, `searchSimilar`. It embeds documents and queries with the correct task types for you. +2. Declare every field you will filter on in `filterSchema:` at `initialize`. A filter on an undeclared field — or any filter with no schema — is silently ignored and returns unfiltered results. +3. Activate an embedding model with `getActiveEmbedder()` before `addDocument`. +4. `LiteRtEmbeddingBackend` comes from `flutter_gemma_litertlm`, not `flutter_gemma_embeddings`. +5. On web use `WebSqliteVectorStore`; `SqliteVectorStore` throws `UnimplementedError` there. `flutter_gemma_rag_qdrant` is native-only. +6. Android needs `minSdk 30`. + +## Setup ```dart +import 'package:flutter/foundation.dart'; +import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; +import 'package:flutter_gemma_rag_sqlite/flutter_gemma_rag_sqlite.dart'; + await FlutterGemma.initialize( inferenceEngines: [LiteRtLmEngine()], - embeddingBackends: [LiteRtEmbeddingBackend()], // flutter_gemma_embeddings - vectorStore: SqliteVectorStore(), // flutter_gemma_rag_sqlite + embeddingBackends: [LiteRtEmbeddingBackend()], + vectorStore: kIsWeb ? WebSqliteVectorStore() : SqliteVectorStore(), + filterSchema: const FilterSchema(fields: [ + FilterField(name: 'lang', type: FilterFieldType.string), + FilterField(name: 'year', type: FilterFieldType.number), + ]), ); -``` -Without a `vectorStore` the default sentinel throws a clear "add a RAG package" -error on first use. Without an embedding backend, `getActiveEmbedder` throws the -same way. +await FlutterGemma.installEmbedder() + .modelFromNetwork(url) + .tokenizerFromNetwork(url) + .install(); +await FlutterGemma.getActiveEmbedder(); + +await FlutterGemma.rag.initialize('rag.db'); +``` -## Query and document must use DIFFERENT task types +`rag.initialize` takes a database file for sqlite and a directory for qdrant. -This is the mistake that silently ruins retrieval. Embedding models are trained -asymmetrically: a question and the passage that answers it are encoded with -different prefixes, and using one prefix for both collapses the similarity -signal. Nothing errors — results are just bad. +## Index and search ```dart -// Indexing a document -final docVector = await embedder.generateEmbedding( - chunk, - taskType: TaskType.retrievalDocument, +import 'dart:convert'; + +await FlutterGemma.rag.addDocument( + id: 'doc-1', + content: chunk, + metadata: jsonEncode({'lang': 'en', 'year': 2024}), ); -// Searching with a question -final queryVector = await embedder.generateEmbedding( - question, - taskType: TaskType.retrievalQuery, +final hits = await FlutterGemma.rag.searchSimilar( + query: question, + topK: 5, + filter: const Filter( + must: [FieldEquals(key: 'lang', value: 'en')], + mustNot: [FieldRange(key: 'year', lte: 2010)], + ), ); +for (final hit in hits) { + print('${hit.similarity.toStringAsFixed(2)} ${hit.content}'); +} ``` -The prefix strings live in one place in Dart and are applied for you — pass the -right `TaskType` and do not prepend anything yourself. - -## Embedding is CPU-only, and that is permanent +`searchSimilar` takes the question as text and embeds it itself. Filter operators: `FieldEquals`, `FieldRange` (`gte`, `lte`), `FieldMatchAny`, combined with `must`, `should` and `mustNot`. -EmbeddingGemma ships as int4, and the TFLite GPU delegate cannot execute int4. -This is not a missing feature or a bug to work around: there is no GPU path. -Asking for `PreferredBackend.gpu` on an embedder gains nothing. +## Traps -Budget accordingly — embedding a large corpus on device is minutes of CPU, so -do it in the background, batched, and persist the vectors rather than -recomputing at startup. +**Filter has no effect** +- Symptom: results ignore the filter; no error. +- Fix: declare the field in `filterSchema`. Names must match `^[A-Za-z][A-Za-z0-9_]*$` and cannot be `id`, `embedding`, `content`, `metadata`, `distance` or `k`. At most 16 fields. -## Picking a vector store +**Poor retrieval after embedding by hand** +- Query and document embeddings are trained asymmetrically. `generateEmbedding` defaults to `TaskType.retrievalQuery`, so text embedded for indexing without a task type gets the query prefix. +- Fix: pass `TaskType.retrievalDocument` when indexing yourself: -| Package | Platforms | Notes | -| --- | --- | --- | -| `flutter_gemma_rag_sqlite` | all six, web included | `sqlite-vec` KNN inside SQLite | -| `flutter_gemma_rag_qdrant` | native only, no web | the official `qdrant_edge` SDK | +```dart +final vector = await embedder.generateEmbedding( + chunk, + taskType: TaskType.retrievalDocument, +); +await FlutterGemma.rag.addDocumentWithEmbedding( + id: 'doc-2', + content: chunk, + embedding: vector, +); +``` -`rag_sqlite` on web needs a custom `sqlite3.wasm` with `vec0` linked in — copy -it into the app's own `web/` directory. Nothing does that automatically. +**`addDocument` throws** +- Cause: no active embedding model. Call `FlutterGemma.getActiveEmbedder()` after installing one. -## Filtering +## Backend -Both stores take the same sealed `Filter` DSL from core, so a query written -against one works against the other: +Leave the embedder on the default CPU backend. The GPU delegate does not produce valid vectors for EmbeddingGemma. -```dart -final results = await FlutterGemma.rag.search( - queryVector, - limit: 5, - filter: Filter(must: [Condition.equals('lang', 'en')]), -); -``` +## Web -## Chunking is yours +- Copy `web/rag/sqlite3.wasm` from the `flutter_gemma_rag_sqlite` package into the app as `web/rag/sqlite3.wasm`. +- Web embeddings need `litert_embeddings.js` and `sentencepiece.js` from the `flutter_gemma_embeddings` package's `web/` directory, copied into the app's `web/`, plus `` in `web/index.html`. -The package embeds what you give it. Splitting documents, choosing chunk size -and overlap, and storing the text alongside the vector are all application -decisions. A chunk longer than the model's input window is truncated silently — -check the model's limit rather than assuming. +Find a package's directory with `grep -A1 '"name": "flutter_gemma_rag_sqlite"' .dart_tool/package_config.json`. -## Do not double-normalise +## Chunking -`meanPoolAndNormalize` accepts only token-level output shaped `[1, seq, dim]` -and deliberately rejects rank-2 input. A model that already returns a pooled, -normalised vector must not be pooled again — doing so distorts every distance in -the index, and the failure is invisible until retrieval quality is measured. If -a rank-2 rejection fires, the model's output contract is pooled-final; wire it -as such rather than reshaping to get past the check. +Splitting documents, chunk size and overlap are yours to decide. A chunk longer than the embedding model's input window is truncated without an error. diff --git a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md index e3c6a498..c8766b95 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md @@ -1,39 +1,34 @@ --- name: flutter-gemma-speech -description: Use when adding speech to a flutter_gemma app — transcription (moonshine/Whisper/Parakeet), synthesis (Matcha/Qwen3/Inflect), or the VoiceSession loop. Audio must be 16 kHz mono 16-bit PCM, and the Whisper output language is a property of a transcription rather than of the loaded model. +description: Use when adding speech to a flutter_gemma app — speech-to-text (transcribe a voice note, dictation, Whisper, moonshine, Parakeet), text-to-speech (Matcha, Qwen3-TTS, Inflect), or a push-to-talk voice assistant with VoiceSession. Also use when transcripts come back in English for non-English audio, synthesized audio plays at the wrong pitch, or getActiveTts throws a StateError about the language. For LLM text generation, use flutter-gemma-inference. --- # Speech with flutter_gemma_speech -## Register the backend first +## Rules -STT is opt-in. Core registers nothing: +1. Audio input is 16 kHz, mono, 16-bit little-endian PCM. Not a WAV file (strip its 44-byte header), not 44.1 or 48 kHz. Nothing is resampled for you. +2. Play synthesized audio at `synth.sampleRate`. It differs per model. +3. Only Whisper has a selectable output language. moonshine-tiny and Parakeet are English-only, and passing a language to them throws `ArgumentError`. +4. STT language: set a default with `getActiveStt(language:)` or override one call with `transcribe(pcm, language:)`. Nothing reloads. +5. TTS language: `close()` the synthesizer first. Asking a live synthesizer for another language throws `StateError`. +6. Android needs `minSdk 30`. There is no web support — the web backends throw `UnsupportedError`. +7. Close recognizers and synthesizers. + +## Setup ```dart -import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_speech/flutter_gemma_speech.dart'; await FlutterGemma.initialize( - sttBackends: [LiteRtSttBackend()], // transcription - ttsBackends: [LiteRtTtsBackend()], // synthesis + sttBackends: [LiteRtSttBackend()], + ttsBackends: [LiteRtTtsBackend()], ); ``` -Native only — Android, iOS, macOS, Windows, Linux. The web arm is a stub that -throws `UnsupportedError`. - -## Pick the model deliberately - -| `SttModelType` | Input | Languages | Notes | -| --- | --- | --- | --- | -| `moonshine` | raw 16 kHz PCM | the language it hears | ~104 MB, 5 s window, fastest | -| `whisper` | log-mel | 99, selectable | tiny / base, 30 s window | -| `parakeet` | log-mel | English only | CTC 0.6B, 2.35 GB f32, desktop | +## Speech-to-text -Only Whisper has a selectable output language. The other two transcribe -whatever they hear and **reject** a language argument rather than ignoring it. - -## Install and transcribe +An STT model is two files — the model and its tokenizer — usually from different repos. ```dart await FlutterGemma.installStt() @@ -42,82 +37,33 @@ await FlutterGemma.installStt() .ofType(SttModelType.whisper) .install(); -final recognizer = await FlutterGemma.getActiveStt(); +final recognizer = await FlutterGemma.getActiveStt(language: 'de'); try { - // pcm: 16 kHz mono 16-bit little-endian PCM — the data chunk of a WAV, - // or frames from a recorder. NOT the WAV file itself. - final transcript = await recognizer.transcribe(pcm); + final german = await recognizer.transcribe(germanPcm); + final french = await recognizer.transcribe(frenchPcm, language: 'fr'); } finally { await recognizer.close(); } ``` -An STT model needs **two** files, a model and a tokenizer, and they usually come -from different repos: the LiteRT conversion of the weights, and the original -publisher's `tokenizer.json`. - -## The output language is per transcription - -This is the part that is easy to model wrongly. The language is one token in -Whisper's decoder seed prompt, and that prompt is rebuilt on every -transcription. Changing it costs a map lookup — it never reloads the model and -never invalidates a recognizer you are holding. - -```dart -// A default for this recognizer. -final stt = await FlutterGemma.getActiveStt(language: 'de'); -final german = await stt.transcribe(germanPcm); - -// One call in another language — same recognizer, nothing reloaded. -final french = await stt.transcribe(frenchPcm, language: 'fr'); -``` - -`getActiveStt` returns a process-wide singleton, and calling it again with a new -`language` retargets that recognizer. You never need to `close()` just to change -language. - -Codes are Whisper's own, without the delimiters — `'en'`, `'de'`, `'uk'`, any of -the 99 — and the default is `'en'`. - -## Language decides the OUTPUT, not comprehension - -The shipped Whisper checkpoints are the multilingual ones (no `.en` suffix), so -the weights understand the audio either way. The token only decides what the -model writes. Measured on one German clip, same audio and build, one token -apart: - -``` -'en' -> " This weather is very beautiful and the sun is shining." -'de' -> " Das Wetter ist heute sehr schön und die Sonne scheint." -``` - -So asking for the wrong language does not garble the output — it translates, -fluently and without any error. If a user reports "it always answers in -English", the language was never applied; it is not a model failure. - -## Bad values throw, they are never ignored +| `SttModelType` | Languages | Window | +| --- | --- | --- | +| `moonshine` | English | 5 s | +| `whisper` | 99, selectable, default `'en'` | 30 s | +| `parakeet` | English | 5 s, desktop only (2.35 GB) | -- A malformed code (`'de-DE'`, `'DE'`, `'german'`, `''`) is rejected before the - model is loaded. -- A well-formed code the installed checkpoint does not have (`'zz'`) is rejected - against that checkpoint's own tokenizer, with the valid set named in the - error. -- Any language on `moonshine` or `parakeet` throws `ArgumentError` — those - models have no language token to set. +Longer audio has to be split by the caller. -Catch `ArgumentError` around a user-supplied language. Do not fall back to a -default silently; the whole design here exists because a silently ignored -language is indistinguishable from success. +## Traps -## Requirements +**Transcript comes back in English** +- Symptom: German audio, fluent English text, no error. +- Cause: Whisper's language token decides the output language, not what it understands — with `'en'` it translates. moonshine only ever produces English. +- Fix: use Whisper and pass `language:`. -- Audio must be **16 kHz mono 16-bit little-endian PCM**. Resample first; there - is no conversion inside the package. -- Clips are padded or trimmed to the model's fixed window (moonshine 5 s, - Whisper 30 s). Longer audio needs chunking by the caller. -- Transcription runs in a background isolate, so it does not block the UI. -- `flutter_gemma_speech` requires a matching core — check its `flutter_gemma` - constraint. A core too old accepts `language:` and drops it. +**A language is rejected** +- Whisper codes are bare and lowercase: `'de'`, not `'de-DE'`, `'DE'` or `'german'`. Malformed codes throw `ArgumentError` from `getActiveStt`. +- A well-formed code the installed checkpoint lacks (e.g. `'zz'`) throws `ArgumentError` from `transcribe`. ## Text-to-speech @@ -129,64 +75,56 @@ await FlutterGemma.installTts() final synth = await FlutterGemma.getActiveTts(); try { - final pcm = await synth.synthesize('Hello world.'); // Uint8List, 16-bit PCM - print(synth.sampleRate); // 22050 for Matcha + final audio = await synth.synthesize('Hello world.'); // 16-bit PCM + final rate = synth.sampleRate; // 22050 for Matcha } finally { await synth.close(); } ``` -`sampleRate` differs per model — read it rather than assuming, or playback is -pitched wrong. +| `TtsModelType` | Languages | +| --- | --- | +| `matcha` | fixed by the installed bundle | +| `qwen3` | `chinese`, `english`, `german`, `italian`, `portuguese`, `spanish`, `japanese`, `korean`, `french`, `russian`, or `auto` | +| `inflect` | English | -| `TtsModelType` | Languages | Notes | -| --- | --- | --- | -| `matcha` | its bundle's locale | fast, no runtime language parameter | -| `qwen3` | many, selectable | pass `language:` to `getActiveTts` | -| `inflect` | English only | ~90x real time on CPU | +`supertonic` and `kokoro` are in the enum but throw `UnimplementedError` — do not use them. -## TTS language fails LOUD, unlike STT - -`getActiveTts` returns a process-wide singleton, and asking an existing -synthesizer for a different language **throws** a `StateError` telling you to -`close()` first. That is deliberate: reusing it would emit wrong-language audio -with no error. +Switching the Qwen3 language — full lowercase names, not ISO codes: ```dart -final en = await FlutterGemma.getActiveTts(language: 'english'); -await en.close(); // required -final de = await FlutterGemma.getActiveTts(language: 'german'); +final english = await FlutterGemma.getActiveTts(language: 'english'); +await english.close(); +final german = await FlutterGemma.getActiveTts(language: 'german'); ``` -Note the asymmetry with STT, which retargets silently and cheaply instead: a -Whisper decoder prompt is rebuilt per transcription, a TTS voice is not. Values -here are full lowercase names (`'english'`, `'german'`), not the ISO codes STT -uses. - -## Voice loop +## Voice assistant -`VoiceSession` chains STT to an LLM to TTS for one push-to-talk turn, with -barge-in. +`VoiceSession` runs one push-to-talk turn: transcribe, generate, speak, with barge-in. It uses the recognizer's current language. ```dart -final session = VoiceSession.fromChat( +final voice = VoiceSession.fromChat( recognizer: await FlutterGemma.getActiveStt(language: 'de'), chat: chat, synthesizer: await FlutterGemma.getActiveTts(), ); -await for (final event in session.runTurn(pcm16kMono)) { +await for (final event in voice.runTurn(pcm16kMono)) { switch (event) { - case VoiceTranscriptEvent(:final text): // show it - case VoiceReplyTextEvent(:final chunk): // stream it - case VoiceReplyAudioEvent(:final pcm, :final sampleRate): // play it - case VoiceTurnInterruptedEvent(): // stop the player + case VoiceTranscriptEvent(:final text): + print('heard: $text'); + case VoiceReplyTextEvent(:final chunk): + stdout.write(chunk); + case VoiceReplyAudioEvent(:final sampleRate): + print('audio at $sampleRate Hz'); + case VoiceTurnInterruptedEvent(): + print('interrupted — stop the player'); case VoiceTurnCompleteEvent(): - case VoiceErrorEvent(): + print('done'); + case VoiceErrorEvent(:final error): + print('failed: $error'); } } ``` -The session inherits the recognizer's current language, so set it before -starting. A chat with tools is supported — pass `onToolCall`; a tools-enabled -chat arriving without a handler throws. +A chat created with tools also needs `onToolCall:` — without it `fromChat` throws. diff --git a/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md deleted file mode 100644 index 5196cbc6..00000000 --- a/packages/flutter_gemma/skills/flutter-gemma-tools/SKILL.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -name: flutter-gemma-tools -description: Use when giving a flutter_gemma model access to functions — declaring Tool objects, calling createChat with tools, handling FunctionCallResponse, or feeding a tool result back. Both supportsFunctionCalls and modelType are required or the tools are dropped with no error. ---- - -# Function calling with flutter_gemma - -## Three arguments, not one - -Passing `tools` alone does nothing. Without `supportsFunctionCalls` the tools -are dropped with a debug-only warning; without `modelType` the correct call -format cannot be derived. All three are required: - -```dart -final chat = await model.createChat( - tools: myTools, - supportsFunctionCalls: true, - modelType: ModelType.gemma4, -); -``` - -Get this wrong and the model answers in prose, describing what it would do -instead of calling anything. That looks like "the model is bad at tools" and is -actually a missing argument. - -## Declaring a tool - -`parameters` is a JSON Schema object: - -```dart -const changeColor = Tool( - name: 'change_color', - description: 'Change the UI background colour.', - parameters: { - 'type': 'object', - 'properties': { - 'color': {'type': 'string', 'description': 'A colour name like red.'}, - }, - 'required': ['color'], - }, -); -``` - -The `description` is what the model matches the user's intent against. Write it -as an action, and describe every parameter — a bare `{'type': 'string'}` gives -the model nothing to reason with. - -## The response is a type, not a string - -`generateChatResponse()` returns `ModelResponse`. Switch on it: - -```dart -final response = await chat.generateChatResponse(); - -switch (response) { - case FunctionCallResponse(:final name, :final args): - final result = await runTool(name, args); - await chat.addQueryChunk( - Message.toolResponse(toolName: name, response: result), - ); - final followUp = await chat.generateChatResponse(); - - case ParallelFunctionCallResponse(:final calls): - // several calls in one turn — run them, then feed each result back - for (final call in calls) { /* … */ } - - case TextResponse(:final token): - // the model chose to answer directly, which is a valid outcome -} -``` - -Treat a plain `TextResponse` as normal. A model that calls a tool for every -message is worse than one that decides. - -Streaming works the same way — `generateChatResponseAsync()` yields -`ModelResponse` events, and a `FunctionCallResponse` arrives as one of them. -Raw `<|tool_call>` markers must never appear in the text stream; if they do, -the model type is wrong. - -## Feeding the result back - -```dart -Message.toolResponse( - toolName: 'change_color', - response: {'status': 'success', 'applied_color': 'purple'}, -) -``` - -The response map is serialised into the transcript, so keep it small and -factual. Return an error field rather than throwing — the model can recover -from `{'error': 'unknown colour'}` and cannot recover from an exception. - -## Not every model supports tools - -Check the package README's support table before enabling them. Gemma 4, Gemma -3 1B, FunctionGemma, Phi-4 Mini, Qwen 2.5/3 and DeepSeek R1 do; Gemma 3 270M -and SmolLM do not. Enabling tools on a model that cannot use them produces -prose, not an error. - -## Escaped tokens in arguments - -Some runtimes emit `<|"|>` escape markers inside argument strings. The package -strips them before the call reaches you, so `args` values should be clean — -if you see them, report it rather than stripping them yourself. - -## Web - -Function calling on the `.litertlm` web path works, but the constrained-decoding -grammar does not reset after a completed tool-call block: any subsequent turn in -that same chat aborts with `Invalid token at state N`. Until the upstream fix -lands, treat a tools-enabled web chat as single-turn — create a fresh chat after -a call. A conversation that never emits a call is unaffected. diff --git a/tool/check_skills.dart b/tool/check_skills.dart new file mode 100644 index 00000000..e5ec285e --- /dev/null +++ b/tool/check_skills.dart @@ -0,0 +1,237 @@ +// Compile every API reference in the shipped agent skills against the real +// packages. +// +// Skills under packages/flutter_gemma/skills/ are read by a coding agent that +// then writes code from them. A wrong name there does not look wrong — it turns +// into confident, broken code in someone else's app. So the skills are checked +// by the only thing that actually knows what the API is: the compiler. +// +// The previous gate grepped for each name anywhere in packages/*/lib and was +// green on four APIs that do not exist: `ModelType.gemma3` matched a model URL, +// `limit:` an unrelated named argument, `spec:` a parameter name elsewhere, and +// `isThinking` a doc comment. A text search cannot tell "this name exists" from +// "this code is right". Compilation can. +// +// What is compiled: +// - every ```dart fence, each as its own async function body. Imports written +// in any fence of a skill are hoisted to that skill's file, so a snippet +// using a satellite class must show the import that provides it — which is +// the thing an agent needs to know anyway. +// - every inline `TypeName` in the prose, as a typed declaration. +// - every inline `TypeName.member`, as an expression. An instance member +// referenced this way reports `static_access_to_instance_member`, which the +// analyzer only emits when the member EXISTS — so that code counts as found. +// +// Snippets may use the placeholder inputs declared in [_placeholders] without +// declaring them. That list is inputs only — data a real app would already +// have. An API name is never a placeholder; adding one to silence an error is +// exactly the false green this script exists to prevent. +// +// What it cannot check: a symbol that still exists but changed meaning, and +// bare lower-case names mentioned only in prose. For those, read the skill — +// tool/skills_review.sh says which ones a release put in doubt. +// +// Run from the repo root: +// dart tool/check_skills.dart +// Exit 0: everything compiles. Exit 1: a skill references something that does +// not compile. Exit 2: nothing was checked, which is not a pass. + +import 'dart:io'; + +const _skillsDir = 'packages/flutter_gemma/skills'; + +/// Where the generated files are analysed. The example app depends on every +/// package in the workspace, so any import a skill shows can resolve there. +const _hostPackage = 'packages/flutter_gemma/example'; + +const _preamble = ''' +// GENERATED by tool/check_skills.dart — deleted after each run. Do not edit. +// ignore_for_file: type=lint, unused_import, unused_local_variable, unused_element, dead_code, unused_catch_clause, dead_code_on_catch_subtype +import 'dart:async'; +import 'dart:io'; +import 'dart:typed_data'; +import 'package:flutter_gemma/flutter_gemma.dart'; +'''; + +/// Inputs a snippet may use without declaring them. Data and already-built +/// objects only — never an API name. +const _placeholders = ''' +late String url; +late String prompt; +late String question; +late String chunk; +late String repo; +late String path; +late Uint8List bytes; +late Uint8List pcm; +late Uint8List pcm16kMono; +late Uint8List germanPcm; +late Uint8List frenchPcm; +late InferenceModel model; +late InferenceModelSession session; +late InferenceChat chat; +late EmbeddingModel embedder; +late SpeechRecognizer recognizer; +late SpeechSynthesizer synth; +late List myTools; +Future> runTool(String name, Map args) async => {}; +'''; + +class _Origin { + _Origin(this.skill, this.line, this.what); + final String skill; + final int line; + final String what; +} + +Future main() async { + final skillsDir = Directory(_skillsDir); + if (!skillsDir.existsSync()) { + stderr.writeln('no skills at $_skillsDir — run from the repo root'); + exit(2); + } + + final skills = + skillsDir + .listSync() + .whereType() + .where((d) => File('${d.path}/SKILL.md').existsSync()) + .toList() + ..sort((a, b) => a.path.compareTo(b.path)); + + final work = Directory('$_hostPackage/tool/_skills_check_$pid'); + work.createSync(recursive: true); + + // generated file path -> (generated line -> where it came from) + final origins = >{}; + var blocks = 0; + var inlineRefs = 0; + + try { + for (final dir in skills) { + final skill = dir.uri.pathSegments.where((s) => s.isNotEmpty).last; + final src = File('${dir.path}/SKILL.md').readAsLinesSync(); + + final imports = {}; + final fences = >[]; + final inline = <(int, String)>{}; + + var inFence = false; + var lang = ''; + List<(int, String)>? current; + for (var i = 0; i < src.length; i++) { + final line = src[i]; + final fence = RegExp(r'^```(\w*)\s*$').firstMatch(line); + if (fence != null) { + if (!inFence) { + inFence = true; + lang = fence.group(1)!; + current = lang == 'dart' ? <(int, String)>[] : null; + } else { + inFence = false; + if (current != null) fences.add(current); + current = null; + } + continue; + } + if (inFence) { + if (current == null) continue; + if (RegExp(r"^\s*import\s+'").hasMatch(line)) { + imports.add(line.trim()); + } else { + current.add((i + 1, line)); + } + continue; + } + // Prose: inline `Type` and `Type.member` references. + for (final m in RegExp( + r'`([A-Z][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)`', + ).allMatches(line)) { + inline.add((i + 1, m.group(1)!)); + } + } + + final out = StringBuffer(_preamble); + for (final imp in imports) { + out.writeln(imp); + } + out.writeln(); + out.write(_placeholders); + out.writeln(); + + final map = {}; + var genLine = out.toString().split('\n').length; + + void emit(String code, [_Origin? origin]) { + out.writeln(code); + if (origin != null) map[genLine] = origin; + genLine++; + } + + for (var b = 0; b < fences.length; b++) { + blocks++; + emit('Future _block$b() async {'); + for (final (n, code) in fences[b]) { + emit(code, _Origin(skill, n, 'code block')); + } + emit('}'); + } + + emit('void _inline() {'); + var r = 0; + for (final (n, ref) in inline) { + inlineRefs++; + final origin = _Origin(skill, n, '`$ref`'); + if (ref.contains('.')) { + emit(' $ref;', origin); + } else { + emit(' $ref? _r${r++};', origin); + } + } + emit('}'); + + final file = File('${work.path}/${skill.replaceAll('-', '_')}.dart'); + file.writeAsStringSync(out.toString()); + origins[file.absolute.path] = map; + } + + if (blocks == 0 && inlineRefs == 0) { + stderr.writeln( + 'extracted nothing to check — the checker is broken, not the skills', + ); + exit(2); + } + + final result = await Process.run('dart', [ + 'analyze', + '--format=machine', + work.absolute.path, + ], workingDirectory: _hostPackage); + + final problems = []; + for (final row in '${result.stdout}\n${result.stderr}'.split('\n')) { + final cols = row.split('|'); + if (cols.length < 8 || cols[0] != 'ERROR') continue; + final code = cols[2].toLowerCase(); + // Only emitted when the member exists: an instance member named in prose + // as `Type.member`. Found, not missing. + if (code == 'static_access_to_instance_member') continue; + final origin = origins[cols[3]]?[int.parse(cols[4])]; + final where = origin == null + ? '${cols[3]}:${cols[4]}' + : '${origin.skill}/SKILL.md:${origin.line} (${origin.what})'; + problems.add(' $where\n ${cols.sublist(7).join('|')}'); + } + + for (final p in problems) { + stdout.writeln(p); + } + stdout.writeln( + 'compiled $blocks code block(s) and $inlineRefs inline reference(s) ' + 'across ${skills.length} skill(s): ${problems.length} error(s)', + ); + exitCode = problems.isEmpty ? 0 : 1; + } finally { + work.deleteSync(recursive: true); + } +} diff --git a/tool/check_skills.sh b/tool/check_skills.sh deleted file mode 100755 index 8508ab1b..00000000 --- a/tool/check_skills.sh +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env bash -# -# Every API symbol the shipped agent skills name must still exist in the -# workspace sources. -# -# Skills under packages/flutter_gemma/skills/ are read by a MACHINE, not a -# person. A renamed symbol does not make them look odd — it makes them -# confidently wrong, and the user's agent writes code against an API that is -# gone. Nothing else in the build notices: skills are markdown, so analyze, -# test and format all stay green. -# -# Three shapes are checked, because between them they cover where API names -# actually appear in a SKILL.md: -# -# inline `backticked` identifiers CamelCase, or camelCase with a capital -# named arguments in ```dart fences maxOutputTokens: -# dotted members in ```dart fences TaskType.retrievalQuery -# -# The first shape alone is not enough — it misses `maxOutputTokens` entirely, -# because that symbol appears in the skills only inside code blocks. Verified by -# mutation: renaming maxOutputTokens, supportsFunctionCalls, -# TaskType.retrievalQuery, BuiltInAi.availability or Message.toolResponse each -# turns this script red. -# -# What it CANNOT catch: a symbol that still exists but changed meaning. The STT -# language went from a load-time property to a per-transcription one without a -# single rename — for that class, read the skill. -# -# Usage, from the repo root: -# bash tool/check_skills.sh -# Exit 1 means a skill names something that no longer exists. - -set -uo pipefail - -SKILLS_DIR=packages/flutter_gemma/skills - -if [ ! -d "$SKILLS_DIR" ]; then - echo "no skills directory at $SKILLS_DIR — run from the repo root" >&2 - exit 2 -fi - -LIB=$(mktemp) -trap 'rm -f "$LIB"' EXIT -find packages -path '*/lib/*' -name '*.dart' -not -path '*/build/*' \ - -exec cat {} + > "$LIB" 2>/dev/null - -if [ ! -s "$LIB" ]; then - echo "collected no Dart sources — the check would pass vacuously" >&2 - exit 2 -fi - -# Not Dart API: build-config keys and prose that survives the shape filter. -SKIP=' minSdk platform dependencies ' - -total=0 -missing=0 - -for f in "$SKILLS_DIR"/*/SKILL.md; do - skill=$(basename "$(dirname "$f")") - code=$(awk '/^```dart/{c=1;next} /^```/{c=0} c' "$f") - - inline=$(grep -oE '`[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*`' "$f" \ - | tr -d '`' | grep -E '^[A-Z]|[a-z][A-Z]') - named=$(printf '%s' "$code" \ - | grep -oE '(^|[ (,])[a-z][A-Za-z0-9_]*:' | tr -d ' (,:') - dotted=$(printf '%s' "$code" \ - | grep -oE '\b[A-Z][A-Za-z0-9_]*\.[a-z][A-Za-z0-9_]*') - - for sym in $(printf '%s\n%s\n%s\n' "$inline" "$named" "$dotted" | sort -u); do - [ -z "$sym" ] && continue - case "$SKIP" in *" $sym "*) continue ;; esac - - total=$((total + 1)) - base="${sym%%.*}" - leaf="${sym##*.}" - - if ! grep -q "\b${base}\b" "$LIB"; then - echo " MISSING ${sym} (${skill})" - missing=$((missing + 1)) - elif [ "$base" != "$leaf" ] && ! grep -q "\b${leaf}\b" "$LIB"; then - echo " MISSING ${sym} (${skill})" - missing=$((missing + 1)) - fi - done -done - -skills_count=$(find "$SKILLS_DIR" -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ') -echo "checked ${total} symbol(s) across ${skills_count} skill(s), ${missing} missing" - -# A run that examined nothing is indistinguishable from a clean run — refuse to -# report success in that case. -if [ "$total" -eq 0 ]; then - echo "extracted no symbols — the check is broken, not the skills" >&2 - exit 2 -fi - -[ "$missing" -eq 0 ] || exit 1 diff --git a/tool/skills_review.sh b/tool/skills_review.sh index a3e42257..37f77dbf 100755 --- a/tool/skills_review.sh +++ b/tool/skills_review.sh @@ -2,8 +2,8 @@ # # Which shipped skills does this release's diff put in doubt, and why. # -# `check_skills.sh` answers "does every name still exist" — renames and -# deletions. It stays green when a symbol survives but changes MEANING, which is +# `check_skills.dart` answers "does this still compile" — renames, deletions +# and signature changes. It stays green when a symbol survives but changes MEANING, which is # the failure that actually bit us: getActiveStt(language:) went from a # load-time property of the recognizer to a per-transcription one, with no # rename anywhere. diff --git a/website/content/docs/models.md b/website/content/docs/models.md index 36c89e08..0b3fc7d5 100644 --- a/website/content/docs/models.md +++ b/website/content/docs/models.md @@ -380,7 +380,7 @@ translation rather than an error. Moonshine and Parakeet have no language token in their decoder prompt and **reject** the parameter with an `ArgumentError` rather than ignoring it; both -transcribe the language they hear (Parakeet CTC 0.6B is English-only). +are English-only. **Text-to-speech** From fc95044b50d76ee482cc37ef279beaa134c84b55 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Fri, 11 Sep 2026 09:48:19 +0200 Subject: [PATCH 06/21] =?UTF-8?q?docs(skills):=20second=20review=20round?= =?UTF-8?q?=20=E2=80=94=20core=20import,=20gate=20hole,=20platform=20setup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The compile gate injected `package:flutter_gemma/flutter_gemma.dart` into every generated file, and none of the seven skills imported it. The engine packages do not re-export core, so every example failed with "Undefined name 'FlutterGemma'" while the gate stayed green. An agent given only the skills found it. Without the injected import the gate reports 161 errors on the previous text and none on this one. - every setup block shows the core import and a `flutter pub add` line - platform-setup: macOS post_install build phase, Android INTERNET for release builds, SPM deployment target, Xcode capabilities instead of a loose entitlements file, no iOS-only kernel keys on macOS, contents - rag: absolute store path on native (a bare name resolves against the working directory, not writable on Android or iOS), real EmbeddingGemma URLs and the gated-token caveat, upsert/removeDocument/clear - inference: real Gemma 4 URL and size, withProgress, systemInstruction, audio input as a whole WAV file, "Session is closed", sampler set on the first session, clamp only on native .litertlm, the token ships in the app - function-calling: modelType needed only on web and ONNX; the loop's contract (onToolCall type, maxToolTurns ends quietly, exceptions rethrown) - speech: typed PCM contract, recording at 16 kHz, a data-chunk reader instead of "strip 44 bytes", whisper-base, the verbatim TTS language error - builtin-ai: desktop Chrome only, unavailable states are not final, TimeoutException, a fallback to a downloaded model that compiles - mediapipe, onnx: static frameworks back, web image rules, ONNX web installs by repo URL - .pubignore: ship web/opfs_helper.js; the skills rule goes last - core README, the rag dartdoc and two site pages: absolute store path --- packages/flutter_gemma/.pubignore | 23 +++- packages/flutter_gemma/README.md | 7 +- .../lib/core/api/flutter_gemma.dart | 6 +- .../skills/flutter-gemma-builtin-ai/SKILL.md | 81 ++++++++--- .../flutter-gemma-function-calling/SKILL.md | 40 +++--- .../skills/flutter-gemma-inference/SKILL.md | 128 +++++++++++++----- .../references/platform-setup.md | 99 ++++++++++++-- .../skills/flutter-gemma-mediapipe/SKILL.md | 50 +++++-- .../skills/flutter-gemma-onnx/SKILL.md | 39 ++++-- .../skills/flutter-gemma-rag/SKILL.md | 66 ++++++--- .../skills/flutter-gemma-speech/SKILL.md | 84 +++++++++--- tool/check_skills.dart | 9 +- website/content/docs/embeddings-and-rag.md | 7 +- website/content/docs/migration.md | 3 +- 14 files changed, 474 insertions(+), 168 deletions(-) diff --git a/packages/flutter_gemma/.pubignore b/packages/flutter_gemma/.pubignore index c231dcb8..d87cd76c 100644 --- a/packages/flutter_gemma/.pubignore +++ b/packages/flutter_gemma/.pubignore @@ -8,12 +8,6 @@ test_reports/ **/*.md !README.md !CHANGELOG.md -# Agent skills MUST ship inside the archive — `dart run skills@ get` resolves -# the package on disk from package_config.json and reads `skills/` from there. -# Without this exception the blanket `**/*.md` above silently strips every -# SKILL.md and the skills reach nobody, with no error anywhere. -!skills/ -!skills/** tools/ # Build artifacts @@ -39,6 +33,10 @@ web/*.js.map !web/rag/ # And keep cache_api.js (hand-written, runtime needs window.cacheHas/Put/Get) !web/cache_api.js +# And opfs_helper.js — defines window.flutterGemmaOPFS, which web streaming +# storage (WebStorageMode.streaming) binds to. Without this line it never +# reached pub, so the documented copy-from-the-package step had nothing to copy. +!web/opfs_helper.js # Media files (for README, not needed in package) docs/ @@ -139,4 +137,15 @@ example/web/test_vectorstore.html # chromedriver binary — maintainer tool for running web integration tests # (flutter drive -d chrome). ~16 MB, not part of the plugin. Without this # it leaks into the pub package and bloats it (8 MB → archive). -chromedriver/ \ No newline at end of file +chromedriver/ + +# Agent skills MUST ship inside the archive — `dart run skills@ get` resolves +# the package on disk from package_config.json and reads `skills/` from there. +# Without this exception the blanket `**/*.md` above silently strips every +# SKILL.md and the skills reach nobody, with no error anywhere. +# +# It is LAST on purpose: the last matching rule wins, and rules such as +# `assets/`, `docs/` and `test/` would otherwise drop a skill subdirectory with +# those names — `assets/` is part of the Agent Skills layout. +!skills/ +!skills/** diff --git a/packages/flutter_gemma/README.md b/packages/flutter_gemma/README.md index d3a42166..cbc1f21d 100644 --- a/packages/flutter_gemma/README.md +++ b/packages/flutter_gemma/README.md @@ -1694,8 +1694,11 @@ await FlutterGemma.installEmbedder() ) .install(); -// 2. Initialize the vector store (one shard per database path) -await FlutterGemmaPlugin.instance.initializeVectorStore('rag_store'); +// 2. Initialize the vector store (one shard per database path). On native pass +// an absolute path: a bare name resolves against the process working +// directory, which is not writable on Android or iOS. On web a name is enough. +final dir = await getApplicationDocumentsDirectory(); // package:path_provider +await FlutterGemmaPlugin.instance.initializeVectorStore('${dir.path}/rag_store'); // 3. Add documents — let the plugin compute embeddings for you for (final doc in docs) { diff --git a/packages/flutter_gemma/lib/core/api/flutter_gemma.dart b/packages/flutter_gemma/lib/core/api/flutter_gemma.dart index ca41b251..9406d0e7 100644 --- a/packages/flutter_gemma/lib/core/api/flutter_gemma.dart +++ b/packages/flutter_gemma/lib/core/api/flutter_gemma.dart @@ -1011,7 +1011,11 @@ class FlutterGemma { /// throws a clear "add a RAG package" error. /// /// ```dart - /// await FlutterGemma.rag.initialize('rag.db'); + /// // Native: an absolute path in a writable directory. A bare name resolves + /// // against the process working directory, which is not writable on + /// // Android or iOS. Web: a bare name is fine. + /// final dir = await getApplicationDocumentsDirectory(); // path_provider + /// await FlutterGemma.rag.initialize('${dir.path}/rag.db'); /// await FlutterGemma.rag.addDocument(id: '1', content: 'hello'); /// final hits = await FlutterGemma.rag.searchSimilar(query: 'hi'); /// await FlutterGemma.rag.removeDocument(id: '1'); diff --git a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md index e9e1f36d..0c18840d 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md @@ -1,44 +1,76 @@ --- name: flutter-gemma-builtin-ai -description: Use when running the device's own model with flutter_gemma_builtin_ai — Gemini Nano on Android or in desktop Chrome, Apple Foundation Models on iPhone, iPad and Mac — with nothing to download or bundle. Also use when BuiltInAiUnavailableException is thrown, availability reports "downloadable", the Android build fails the manifest merge on minSdk, or the model is missing in Chrome. For models you download yourself, use flutter-gemma-inference. +description: Use when running the device's own model with flutter_gemma_builtin_ai — Gemini Nano on Android or in desktop Chrome, Apple Foundation Models on iPhone, iPad and Mac — with nothing to download or bundle, or when falling back to a downloaded model where it is missing. Also use when BuiltInAiUnavailableException or a TimeoutException is thrown, availability reports "downloadable", the Android build fails the manifest merge on minSdk, or the model is missing in Chrome. For models the app downloads itself, use flutter-gemma-inference. --- # The built-in OS model ## Rules -1. The OS owns the weights, but the model is still installed — as an identity: `fileType: ModelFileType.builtIn` with `.fromBundled(...)`. Nothing is downloaded by the app. -2. Call `BuiltInAi.ensureReady()` before `getActiveModel()`, from a user action: on Android the first call downloads the model and can take minutes. +1. The OS owns the weights, but the model is still installed — as an identity: `fileType: ModelFileType.builtIn` with `.fromBundled(...)`. The app downloads nothing. +2. Call `BuiltInAi.ensureReady()` before `getActiveModel()`, from a user action: on Android the first call downloads the model and can take minutes. It throws `TimeoutException` after `timeout` — 10 minutes by default. 3. Catch `BuiltInAiUnavailableException` and fall back to a downloadable model. 4. Android apps need `minSdk 26`, or the manifest merge fails. 5. There is no Windows or Linux support. -## Setup +## Setup with a fallback + +```sh +flutter pub add flutter_gemma flutter_gemma_builtin_ai flutter_gemma_litertlm +``` ```dart +import 'dart:async'; + import 'package:flutter/foundation.dart'; +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_builtin_ai/flutter_gemma_builtin_ai.dart'; +import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; -await FlutterGemma.initialize(inferenceEngines: [BuiltInAiEngine()]); +await FlutterGemma.initialize( + inferenceEngines: [BuiltInAiEngine(), LiteRtLmEngine()], +); final spec = kIsWeb || defaultTargetPlatform == TargetPlatform.android ? BuiltInAiModels.geminiNano - : BuiltInAiModels.appleFoundationModels; - -await FlutterGemma.installModel( - modelType: ModelType.general, - fileType: ModelFileType.builtIn, -).fromBundled(spec.name).install(); - -try { - await BuiltInAi.ensureReady(onProgress: (percent) => print('$percent%')); - final model = await FlutterGemma.getActiveModel(maxTokens: 4096); -} on BuiltInAiUnavailableException catch (e) { - print('No built-in model here: $e — fall back to a downloadable one'); + : defaultTargetPlatform == TargetPlatform.iOS || + defaultTargetPlatform == TargetPlatform.macOS + ? BuiltInAiModels.appleFoundationModels + : null; // Windows and Linux have no built-in model + +Future downloadGemma() async { + await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, + ).fromNetwork( + // 2.6 GB — ask first. On web: gemma-4-E2B-it-web.litertlm + 'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm', + ).install(); + return FlutterGemma.getActiveModel(maxTokens: 1024); +} + +InferenceModel model; +if (spec == null) { + model = await downloadGemma(); +} else { + try { + await FlutterGemma.installModel( + modelType: ModelType.general, + fileType: ModelFileType.builtIn, + ).fromBundled(spec.name).install(); + await BuiltInAi.ensureReady(onProgress: (int percent) => print('$percent%')); + model = await FlutterGemma.getActiveModel(maxTokens: 4096); + } on BuiltInAiUnavailableException { + model = await downloadGemma(); + } on TimeoutException { + model = await downloadGemma(); + } } ``` -## Checking before you offer the feature +The latest install is the one `getActiveModel` loads, so the fallback replaces the built-in model. Sessions and chats then work as in the flutter-gemma-inference skill. + +## Checking before offering the feature ```dart final availability = await BuiltInAi.availability(); @@ -47,7 +79,14 @@ final usable = availability == BuiltInAiAvailability.available || availability == BuiltInAiAvailability.downloading; ``` -`downloadable` and `downloading` mean "not yet", not "no" — `ensureReady` finishes the job. The `unavailable*` states are final for this device: `unavailableDeviceUnsupported`, `unavailableOsTooOld`, `unavailableDisabled`, `unavailableOther`. +`downloadable` and `downloading` mean "not yet" — `ensureReady` finishes the job. The `unavailable*` states describe the device now, not forever: + +| State | Meaning | +| --- | --- | +| `unavailableDeviceUnsupported` | the hardware cannot run it | +| `unavailableOsTooOld` | an OS update would enable it | +| `unavailableDisabled` | the user can turn it on in system settings (Apple Intelligence on Apple devices) | +| `unavailableOther` | anything else, including a probe that timed out after 20 seconds — worth asking again later | ## Platforms @@ -55,7 +94,7 @@ final usable = availability == BuiltInAiAvailability.available || | --- | --- | --- | | Android | Gemini Nano (AICore) | Pixel 9+, Galaxy S25+; `minSdk 26` | | iOS, macOS | Apple Foundation Models | iPhone 15 Pro+ or an Apple Silicon Mac, Apple Intelligence turned on | -| Web | Gemini Nano (Chrome Prompt API) | desktop Chrome or Edge only — not mobile browsers, Firefox or Safari | +| Web | Gemini Nano (Chrome Prompt API) | desktop Chrome — not Edge (its built-in model is Phi-4-mini), mobile browsers, Firefox or Safari | Images work on Android. On Apple platforms they need OS 27 — on OS 26 an image throws. The web model is text-only. @@ -73,4 +112,4 @@ There is no script to add: the Prompt API is part of the browser. It has to be e ## Trade-offs -No choice of weights, no LoRA, and capabilities that vary by OS version. Use it when zero download and zero disk matter more than picking the model. +No choice of weights, no LoRA, and capabilities that vary by OS version. The right pick when zero download and zero disk matter more than choosing the model. diff --git a/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md index d1d69e01..fae75a1a 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-function-calling/SKILL.md @@ -5,19 +5,24 @@ description: Use when adding function calling (tool calling) to a flutter_gemma # Function calling with flutter_gemma +Packages, engine and model install are in the flutter-gemma-inference skill. This one starts from a loaded `InferenceModel`. + ## Rules -1. `createChat` needs three arguments: `tools`, `supportsFunctionCalls: true` and `modelType`. Without the flag the tools are dropped with a debug-only warning; without the type the call format cannot be derived. -2. Switch over all four `ModelResponse` subtypes. It is sealed — a switch that leaves out `ThinkingResponse` does not compile. -3. Return tool results as data, errors included. Never throw from a tool. -4. Prefer `generateChatResponseWithTools` to a hand-written loop. -5. Use a tool-capable model: Gemma 4, Gemma 3 1B, FunctionGemma, Phi-4 Mini, Qwen 2.5, Qwen3, DeepSeek R1. Gemma 3 270M and SmolLM cannot call tools. +1. `createChat` needs `tools` and `supportsFunctionCalls: true`. Without the flag the tools are dropped with a debug-only warning and no call is parsed. +2. Pass `modelType` on web and on ONNX. Native `.litertlm`, MediaPipe on Android and iOS, and built-in AI use the installed model's type when it is left out; the web engines and ONNX fall back to `gemmaIt`, and another model's calls then arrive as raw text. +3. Switch over all four `ModelResponse` subtypes. It is sealed — a switch that leaves out `ThinkingResponse` does not compile. +4. Return tool results as data, errors included. Never throw from a tool. +5. Prefer `generateChatResponseWithTools` to a hand-written loop. +6. Use a tool-capable model: Gemma 4, Gemma 3 1B, FunctionGemma, Phi-4 Mini, Qwen 2.5, Qwen3, DeepSeek R1. Gemma 3 270M and SmolLM cannot call tools. ## Declare a tool `parameters` is a JSON Schema object. The model matches the user's intent against `description`, so write it as an action and describe every parameter. ```dart +import 'package:flutter_gemma/flutter_gemma.dart'; + const changeColor = Tool( name: 'change_color', description: 'Change the app background colour.', @@ -34,7 +39,7 @@ const changeColor = Tool( ## Open the chat ```dart -final chat = await model.createChat( +final InferenceChat chat = await model.createChat( tools: myTools, supportsFunctionCalls: true, modelType: ModelType.gemma4, @@ -43,18 +48,22 @@ final chat = await model.createChat( ## The built-in loop -It calls your handler for each tool call, feeds the result back, and continues until the model answers in text or `maxToolTurns` is reached. +It calls the handler for each tool call, feeds the result back, and continues until the model answers in text or `maxToolTurns` is reached. ```dart await chat.addQueryChunk(Message(text: prompt, isUser: true)); +final answer = StringBuffer(); await for (final r in chat.generateChatResponseWithTools( - onToolCall: (call) => runTool(call.name, call.args), + onToolCall: (FunctionCallResponse call) => runTool(call.name, call.args), maxToolTurns: 8, + onMaxToolTurns: () => print('stopped after 8 tool turns'), )) { - if (r is TextResponse) stdout.write(r.token); + if (r is TextResponse) answer.write(r.token); } ``` +`onToolCall` receives a `FunctionCallResponse` with `name` and `args`, and returns the map the model reads back. The stream carries `TextResponse` and `ThinkingResponse`. Reaching `maxToolTurns` ends the stream without an error — `onMaxToolTurns` is the only signal. An exception from `onToolCall` is reported to the model, then rethrown on the stream. + ## Handling calls yourself ```dart @@ -98,7 +107,7 @@ The model can recover from an error it can read. An exception thrown out of a to **Model answers in prose** - Symptom: "I would change the colour to red" instead of a call. -- Fix: check `supportsFunctionCalls: true` and `modelType`, then check the model is tool-capable. +- Fix: check `supportsFunctionCalls: true`, pass `modelType` on web and ONNX, then check the model is tool-capable. **Raw markers in the text** - Symptom: `<|tool_call>` or `` appears in `TextResponse` tokens. @@ -106,13 +115,4 @@ The model can recover from an error it can read. An exception thrown out of a to ## Web -Function calling works on the `.litertlm` web engine. Close each chat before creating the next one: a chat left open makes the following one fail with `Invalid token at state N`. - -```dart -await chat.close(); -final next = await model.createChat( - tools: myTools, - supportsFunctionCalls: true, - modelType: ModelType.gemma4, -); -``` +Function calling works on the `.litertlm` web engine. Pass `modelType`, and close each chat before creating the next — the web engine holds one session at a time. diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md index 430ae42a..71daa9d8 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/SKILL.md @@ -1,41 +1,62 @@ --- name: flutter-gemma-inference -description: Use when adding on-device LLM inference to a Flutter app with flutter_gemma — offline chat, running Gemma, Qwen or Phi locally, streaming replies, image prompts — or setting up the default .litertlm engine on Android, iOS, macOS, Windows, Linux or web. Also use when a reply comes back empty, maxTokens does not shorten replies, getActiveModel throws "No inference engine can handle this model", or .litertlm fails to load on Android. For function calling, RAG, speech, .task files, ONNX or the OS built-in model, also use the matching flutter-gemma-* skill. +description: Use when adding on-device LLM inference to a Flutter app with flutter_gemma — offline chat, running Gemma, Qwen or Phi locally, downloading and installing a model from Hugging Face (gated repos included), streaming replies, a system prompt, image or audio prompts — or setting up the recommended .litertlm engine (ModelFileType.litertlm) on Android, iOS, macOS, Windows, Linux or web. Also use when a reply comes back empty, maxTokens does not shorten replies, FlutterGemma is an undefined name, getActiveModel throws "No inference engine can handle this model", a session throws "Session is closed", or .litertlm fails to load on Android. For .task or .bin models (ModelFileType.task), use flutter-gemma-mediapipe. --- # Running a model with flutter_gemma ## Rules -1. Register an engine in `FlutterGemma.initialize(inferenceEngines: [...])`. Core ships none. -2. Declare `fileType` on `installModel`. It defaults to `ModelFileType.task`, and the declaration — never the file name — picks the engine. -3. `maxTokens` is the context window. Cap the reply with `maxOutputTokens` on the session. -4. Pass `isUser: true` on every user `Message`. -5. Close every session, chat and model in a `finally`. -6. Never put a Hugging Face token in source. Read it with `String.fromEnvironment`. -7. On Android, set `minSdk 30` for anything built on `.litertlm` — inference, embeddings, speech. +1. Depend on `flutter_gemma` and an engine package, and import both. Engine packages do not re-export core. +2. Register the engine in `FlutterGemma.initialize(inferenceEngines: [...])`. Core ships none. +3. Declare `fileType` on `installModel`. It defaults to `ModelFileType.task`, and the declaration — never the file name — picks the engine. +4. `maxTokens` is the context window. Cap the reply with `maxOutputTokens` on the session or chat. +5. Pass `isUser: true` on every user `Message`. +6. Close a session or chat when its conversation ends. Keep the model while the feature is in use, and close it when the app no longer needs it. +7. Keep Hugging Face tokens out of source: read them with `String.fromEnvironment`. That keeps a token out of git, not out of the app — it is compiled into the binary, and on web into `main.dart.js`. A shipped app should download from a repo that needs no token. +8. On Android, set `minSdk 30` for anything built on `.litertlm` — inference, embeddings, speech. -## Setup — the default engine (.litertlm) +## Setup — the recommended engine (.litertlm) + +```sh +flutter pub add flutter_gemma flutter_gemma_litertlm +``` ```dart +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; -const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN'); - -await FlutterGemma.initialize( - inferenceEngines: [LiteRtLmEngine()], - huggingFaceToken: hfToken.isEmpty ? null : hfToken, // gated repos only -); +await FlutterGemma.initialize(inferenceEngines: [LiteRtLmEngine()]); await FlutterGemma.installModel( modelType: ModelType.gemma4, fileType: ModelFileType.litertlm, -).fromNetwork(url).install(); +) + .fromNetwork( + 'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it.litertlm', + ) + .withProgress((int percent) => print('downloading: $percent%')) + .install(); + +final InferenceModel model = await FlutterGemma.getActiveModel(maxTokens: 1024); +``` -final model = await FlutterGemma.getActiveModel(maxTokens: 1024); +Gemma 4 E2B is 2.6 GB and needs no token. On web use `gemma-4-E2B-it-web.litertlm` from the same repo (2.0 GB). + +`install()` skips the download when the file is already on disk, so calling it at every launch is safe. The latest install becomes the model `getActiveModel` loads. + +A gated repo needs a token, given once: + +```dart +const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN'); + +await FlutterGemma.initialize( + inferenceEngines: [LiteRtLmEngine()], + huggingFaceToken: hfToken.isEmpty ? null : hfToken, +); ``` -Build with `--dart-define=HUGGINGFACE_TOKEN=hf_...` when the model repo is gated. +Build with `--dart-define=HUGGINGFACE_TOKEN=hf_...`. When a Hugging Face repo publishes a deployment manifest, one call picks the variant and its tested runtime settings. The engine carries its own resolver, so registering `LiteRtLmEngine` is enough: @@ -50,10 +71,14 @@ final model = await FlutterGemma.getActiveModel(defaults: install.runtime); Other sources on the same builder: `.fromAsset(path)` for a model bundled in the app, `.fromFile(path)` for one already on disk, `.fromBundled(name)` for a platform-bundled resource. -`modelType` sets the chat template. Gemma 3 and Gemma 3n are `ModelType.gemmaIt` — there is no `gemma3`. The full set: `general`, `gemmaIt`, `gemma4`, `deepSeek`, `qwen`, `qwen3`, `llama`, `hammer`, `functionGemma`, `phi`. A wrong type still generates, with the wrong prompt format. +`modelType` tells flutter_gemma how the model writes tool calls and reasoning, and on some engines it also picks the prompt format. Gemma 3 and Gemma 3n are `ModelType.gemmaIt` — there is no `gemma3`. The full set: `general`, `gemmaIt`, `gemma4`, `deepSeek`, `qwen`, `qwen3`, `llama`, `hammer`, `functionGemma`, `phi`. A wrong type still generates text; tool calls and reasoning then arrive as raw text. ## Traps +**Core not imported** +- Symptom: `Undefined name 'FlutterGemma'`, `Undefined class 'InferenceModel'`, with only the engine package imported. +- Fix: `import 'package:flutter_gemma/flutter_gemma.dart';` as well. + **No engine registered** - Symptom: `StateError: No inference engine can handle this model (ModelFileType.litertlm). Add the engine package to pubspec.yaml and pass it in inferenceEngines: of FlutterGemma.initialize(...)` - Fix: add the engine package and register its provider — or fix `fileType` if the wrong engine is registered. @@ -65,7 +90,7 @@ Other sources on the same builder: `.fromAsset(path)` for a model bundled in the final model = await FlutterGemma.getActiveModel(maxTokens: 100); ``` -- Symptom: replies are as long as ever. On `.litertlm` the value is raised to 1024, the smallest context those models support, and only a debug-mode log says so. +- Symptom: replies are as long as ever. On native `.litertlm` the value is raised to 1024, the smallest context those models support, and only a debug-mode log says so. The web `.litertlm` engine does not take the value at all; on MediaPipe it is the real limit. - Fix: ```dart @@ -80,20 +105,25 @@ Use 4096 or more with images or audio — one image costs hundreds of tokens. - Fix: `Message(text: prompt, isUser: true)`. **Same reply every time** -- Symptom: identical output for identical input. `createSession` defaults to `topK: 1`, which is greedy decoding. -- Fix: pass `topK` (e.g. 40) and a `temperature`. +- Symptom: identical output for identical input. `createSession` and `createChat` default to `topK: 1`, which is greedy decoding. +- Fix: pass `topK` (e.g. 40) and a `temperature`. Set them on the first session after `getActiveModel` — on `.litertlm` the first session's sampler settings can stay in effect for later ones. + +**`Session is closed`** +- Symptom: `StateError: Session is closed` from a session or chat that is still in use. +- Cause: `createSession` and `createChat` fill one slot per model; creating another closes the one before. +- Fix: one conversation at a time, or `openSession` / `openChat` for several (below). On web the `.litertlm` engine holds a single session — close the current chat before creating the next. ## Generate ```dart -final session = await model.createSession( +final InferenceModelSession session = await model.createSession( temperature: 0.8, topK: 40, maxOutputTokens: 256, ); try { await session.addQueryChunk(Message(text: prompt, isUser: true)); - final reply = await session.getResponse(); + final String reply = await session.getResponse(); } finally { await session.close(); } @@ -102,32 +132,45 @@ try { Streaming: ```dart +final reply = StringBuffer(); await session.addQueryChunk(Message(text: prompt, isUser: true)); await for (final token in session.getResponseAsync()) { - stdout.write(token); + reply.write(token); // update the UI here } ``` -To stop early, call `await session.stopGeneration()`. Cancelling the stream subscription detaches Dart but does not stop native decoding on every engine. +To stop early, call `await session.stopGeneration()` — `chat.stopGeneration()` on a chat. Cancelling the stream subscription detaches Dart but does not stop native decoding on every engine. ## Multi-turn chat ```dart -final chat = await model.createChat(tokenBuffer: 256, maxOutputTokens: 512); +final InferenceChat chat = await model.createChat( + systemInstruction: 'You are a concise assistant.', + temperature: 0.8, + topK: 40, + maxOutputTokens: 512, +); try { await chat.addQueryChunk(Message(text: prompt, isUser: true)); - final response = await chat.generateChatResponse(); - if (response is TextResponse) print(response.token); + final reply = StringBuffer(); + await for (final r in chat.generateChatResponseAsync()) { + switch (r) { + case TextResponse(:final token): + reply.write(token); + case ThinkingResponse() || FunctionCallResponse() || ParallelFunctionCallResponse(): + break; + } + } } finally { await chat.close(); } ``` -`generateChatResponse()` returns a sealed `ModelResponse`: `TextResponse`, `FunctionCallResponse`, `ParallelFunctionCallResponse` or `ThinkingResponse`. +The chat keeps the history: add the next user message and generate again. `generateChatResponse()` returns the whole reply as one sealed `ModelResponse` — `TextResponse`, `FunctionCallResponse`, `ParallelFunctionCallResponse` or `ThinkingResponse` — and a `switch` over it must cover all four. ## Two conversations at once -`createSession` and `createChat` fill a single slot on the model: a second call replaces the first, and the two chats corrupt each other. For concurrent conversations use `openSession` / `openChat`, and close each one. +`createSession` and `createChat` fill a single slot on the model, so a second one closes the first. For concurrent conversations use `openSession` / `openChat`, and close each one. ```dart final summariser = await model.openChat(); @@ -143,17 +186,18 @@ try { ## Thinking models -Gemma 4, Qwen3 and DeepSeek R1 can emit reasoning. Pass `isThinking: true` to `createChat`. Reasoning arrives as `ThinkingResponse` only from `generateChatResponseAsync()`; `generateChatResponse()` strips it. Not available on web. +Gemma 4, Qwen3 and DeepSeek R1 can emit reasoning. Pass `isThinking: true` to `createChat`. Reasoning arrives as `ThinkingResponse` only from `generateChatResponseAsync()`; `generateChatResponse()` strips it. On web Gemma 4 has no thinking; Qwen3 and DeepSeek R1 reasoning is still separated out of the text. ```dart final chat = await model.createChat(isThinking: true, modelType: ModelType.qwen3); +final answer = StringBuffer(); await chat.addQueryChunk(Message(text: question, isUser: true)); await for (final r in chat.generateChatResponseAsync()) { switch (r) { case ThinkingResponse(:final content): print('reasoning: $content'); case TextResponse(:final token): - stdout.write(token); + answer.write(token); case FunctionCallResponse() || ParallelFunctionCallResponse(): break; } @@ -171,9 +215,21 @@ await chat.addQueryChunk( ); ``` +## Audio + +```dart +final model = await FlutterGemma.getActiveModel(maxTokens: 4096, supportAudio: true); +final chat = await model.createChat(supportAudio: true); +await chat.addQueryChunk( + Message(text: 'What is said in this recording?', isUser: true, audioBytes: bytes), +); +``` + +`audioBytes` is a whole WAV file — 16 kHz mono, header included. The speech package is the opposite: `transcribe` takes raw PCM with no header. Audio input needs Gemma 4 or Gemma 3n, on Android, iOS or desktop; the `.litertlm` web engine takes no audio. + ## The model is a singleton -`getActiveModel` returns one model per process. Calling it again with different runtime arguments rebuilds it and closes the previous one — a handle you still hold stops working. Load once at startup, then create and close sessions per interaction. +`getActiveModel` returns one model per process. Calling it again with different runtime arguments rebuilds it and closes the previous one — a handle still held stops working. Load it once, then create and close sessions per conversation. ## Backends @@ -191,8 +247,8 @@ print(model.activeBackend); // what actually loaded | `npu` | NPU, GPU, CPU | | `cpu` | CPU only | -Read `activeBackend` rather than assuming the requested one loaded. NPU needs a Snapdragon (Android) or Intel Lunar/Panther Lake (Windows). The iOS Simulator is CPU-only; web is GPU-only. +Read `activeBackend` rather than assuming the requested one loaded; the web `.litertlm` engine reports `null`. NPU needs a Snapdragon (Android) or Intel Lunar/Panther Lake (Windows) and a model compiled for that NPU. The iOS Simulator is CPU-only. On web, MediaPipe is GPU-only. ## Platform setup -Android needs `minSdk 30` and ships `arm64-v8a` only. iOS and macOS need Podfile and entitlement entries; web needs script tags in `web/index.html`. Read `references/platform-setup.md` before building for iOS, macOS or web — without those entries the model fails to load or the app runs out of memory. +Android needs `minSdk 30` and the internet permission in release builds, and ships `arm64-v8a` only. iOS needs Podfile or Xcode settings and memory entitlements; macOS needs entitlements and a Podfile build phase; web needs script tags in `web/index.html`. Read `references/platform-setup.md` before building for any of them — without those entries the model fails to load or the app runs out of memory. diff --git a/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md b/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md index 38e19d1a..4b4b5d89 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md +++ b/packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md @@ -3,6 +3,12 @@ Entries each platform needs before a model will load. Without them the app builds and then fails at model load, or is killed for memory. +- [Android](#android) +- [iOS](#ios) +- [macOS](#macos) +- [Windows and Linux](#windows-and-linux) +- [Web](#web) + ## Android `android/app/build.gradle.kts` (or `build.gradle`): @@ -19,20 +25,39 @@ android { speech. On API 29 the native library fails to load at runtime — the build does not catch it. MediaPipe `.task` models run on lower API levels. +`android/app/src/main/AndroidManifest.xml` needs the internet permission to +download a model. Flutter's template declares it only for debug and profile +builds, so without this line the release build cannot download: + +```xml + +``` + Only `arm64-v8a` is shipped for `.litertlm`. The OpenCL manifest entries the GPU backend needs are merged in by the plugin; nothing to add. ## iOS -`ios/Podfile`, declared once: +Minimum iOS 15.0 — 16.0 if the app includes `flutter_gemma_mediapipe`. + +With CocoaPods, in `ios/Podfile`, declared once: ```ruby platform :ios, '15.0' # '16.0' if the app includes flutter_gemma_mediapipe use_frameworks! :linkage => :static ``` -`ios/Runner/Runner.entitlements` — without these, large models are killed for -memory: +With Swift Package Manager — the default since Flutter 3.44 — there is no +Podfile. Set **iOS Deployment Target** on the Runner target in Xcode instead, or +the build fails with `requires minimum platform version 15.0`. +`flutter_gemma_mediapipe` has no `Package.swift`, so an app using it gets a +Podfile as well; set the platform there too. + +In Xcode, under **Signing & Capabilities**, add **Extended Virtual Addressing** +and **Increased Memory Limit**. That writes these keys to +`ios/Runner/Runner.entitlements` and links the file to the target — a file +edited by hand but not linked does nothing. Without them large models are +killed for memory: ```xml com.apple.developer.kernel.extended-virtual-addressing @@ -53,20 +78,67 @@ Add to both `macos/Runner/DebugProfile.entitlements` and com.apple.security.network.client -com.apple.developer.kernel.extended-virtual-addressing - -com.apple.developer.kernel.increased-memory-limit - ``` `disable-library-validation` lets the app load the bundled native frameworks; `network.client` lets it download the model. Add them to both files — the debug and release builds read different ones. +`.litertlm` on macOS also needs a build phase that copies the LiteRT-LM +companion libraries into the app. Paste this into `macos/Podfile`, replacing any +existing `post_install` block, then run `pod install`: + +```ruby +post_install do |installer| + installer.pods_project.targets.each do |target| + flutter_additional_macos_build_settings(target) + end + + installer.aggregate_targets.each do |aggregate_target| + aggregate_target.user_targets.each do |user_target| + phase_name = '[flutter_gemma] Setup LiteRT-LM macOS' + + unless user_target.name == 'Runner' + user_target.build_phases + .select { |p| p.respond_to?(:name) && p.name == phase_name } + .each { |p| user_target.build_phases.delete(p) } + next + end + + existing = user_target.shell_script_build_phases.find { |p| p.name == phase_name } + phase = existing || user_target.new_shell_script_build_phase(phase_name) + phase.input_paths = [ + '$(BUILT_PRODUCTS_DIR)/$(PRODUCT_NAME).app/Contents/Frameworks/LiteRtLm.framework/Versions/A/LiteRtLm', + ] + phase.output_paths = ['$(DERIVED_FILE_DIR)/flutter_gemma_litertlm_macos.stamp'] + phase.shell_script = <<~SHELL + set -e + STAGER="${HOME}/Library/Caches/flutter_gemma/native/macos_arm64/stage_macos_companions.sh" + if [ ! -f "${STAGER}" ]; then + echo "[flutter_gemma] ERROR: ${STAGER} not found." >&2 + echo " flutter_gemma_litertlm 1.6.2+ installs it there from its build hook." >&2 + echo " Upgrade the package, then: flutter clean && flutter pub get" >&2 + exit 1 + fi + sh "${STAGER}" "${BUILT_PRODUCTS_DIR}/${PRODUCT_NAME}.app/Contents/Frameworks" + mkdir -p "$(dirname "${SCRIPT_OUTPUT_FILE_0}")" + touch "${SCRIPT_OUTPUT_FILE_0}" + SHELL + end + end +end +``` + +Without it the build succeeds and the model fails to load at runtime. + ## Windows and Linux -Nothing to add. The native libraries — including the Windows GPU shader compiler -and NPU runtime — are bundled at build time. +Nothing to add to the project. The native libraries — including the Windows GPU +shader compiler and NPU runtime — are bundled at build time. + +- Windows: end users need the Microsoft Visual C++ Redistributable 2019 or later. +- Linux: building needs `clang cmake ninja-build libgtk-3-dev lld`. GPU needs the + vendor Vulkan driver; Mesa's `llvmpipe` software fallback cannot run Gemma 4. ## Web @@ -103,5 +175,10 @@ Storage mode, set in `FlutterGemma.initialize(webStorageMode: ...)`: | `streaming` | larger models — streams through OPFS | | `none` | no persistence; downloads every launch | -Web is GPU-only. The `.litertlm` web engine is text-only: no images, audio, -thinking or LoRA. +The `.litertlm` web engine loads the web build of a model — +`gemma-4-E2B-it-web.litertlm` (2.0 GB, so use `streaming`), not +`gemma-4-E2B-it.litertlm`. It is text-only: no images, audio or LoRA, and no +Gemma 4 thinking. + +A `--dart-define` token is compiled into `main.dart.js`, where every visitor can +read it. Serve web users a model from a repo that needs no token. diff --git a/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md index 8aa3fc37..12e88659 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-mediapipe/SKILL.md @@ -1,21 +1,27 @@ --- name: flutter-gemma-mediapipe -description: Use when running .task or .bin models (MediaPipe GenAI) with flutter_gemma_mediapipe on Android, iOS or web. Also use when CocoaPods rejects the iOS platform version, images are ignored in a MediaPipe chat, or maxOutputTokens has no effect. MediaPipe has no macOS, Windows or Linux support — use a .litertlm model there (flutter-gemma-inference). +description: Use when running .task or .bin models (MediaPipe GenAI, ModelFileType.task or ModelFileType.binary) with flutter_gemma_mediapipe on Android, iOS or web. Also use when CocoaPods rejects the iOS platform version, images are ignored in a MediaPipe chat, or maxOutputTokens has no effect. MediaPipe has no macOS, Windows or Linux support — use a .litertlm model there (flutter-gemma-inference). --- # The MediaPipe engine ## Rules -1. Declare `fileType: ModelFileType.task` for `.task` files and `ModelFileType.binary` for `.bin` files. -2. An app that includes this package needs iOS 16.0. -3. There is no desktop support. -4. `maxOutputTokens` is ignored. Stop generation with `session.stopGeneration()`. -5. `createChat` does not inherit image support from the model — pass `supportImage: true` to the chat as well. +1. Depend on `flutter_gemma` and `flutter_gemma_mediapipe`, and import both. The engine package does not re-export core. +2. Declare `fileType: ModelFileType.task` for `.task` files and `ModelFileType.binary` for `.bin` files. +3. An app that includes this package needs iOS 16.0. +4. There is no desktop support. +5. `maxTokens` is the real context limit — small values are not raised as they are on `.litertlm`. `maxOutputTokens` is ignored; stop generation with `session.stopGeneration()`. +6. On Android and iOS, `createChat` does not inherit image support from the model — pass `supportImage: true` to the chat as well. ## Setup +```sh +flutter pub add flutter_gemma flutter_gemma_mediapipe +``` + ```dart +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_mediapipe/flutter_gemma_mediapipe.dart'; await FlutterGemma.initialize(inferenceEngines: [MediaPipeEngine()]); @@ -24,17 +30,28 @@ await FlutterGemma.installModel( modelType: ModelType.gemmaIt, fileType: ModelFileType.task, ).fromNetwork(url).install(); + +final InferenceModel model = await FlutterGemma.getActiveModel(maxTokens: 1024); ``` -`ios/Podfile`, once: +Sessions, chats, streaming and the common traps work as in the flutter-gemma-inference skill. + +## iOS + +`ios/Podfile`, declared once: ```ruby platform :ios, '16.0' +use_frameworks! :linkage => :static ``` +With Swift Package Manager, also set **iOS Deployment Target** to 16.0 on the Runner target in Xcode. This package ships no Swift package manifest, so the app gets an `ios/Podfile` either way. + Core and the other engines build from iOS 15.0. If the app does not use `.task` models, leave this package out and stay on 15. -## Images +Large models also need **Extended Virtual Addressing** and **Increased Memory Limit**, added in Xcode under **Signing & Capabilities**, or the app is killed for memory. + +## Images and audio ```dart final model = await FlutterGemma.getActiveModel(maxTokens: 4096, supportImage: true); @@ -44,16 +61,19 @@ await chat.addQueryChunk( ); ``` -Without `supportImage: true` on the chat, the image is dropped and the model answers the text alone. Audio input is not supported on MediaPipe. +On Android and iOS, a chat without `supportImage: true` drops the image and the model answers the text alone. On web the chat follows the model: an image sent to a model loaded without `supportImage: true` throws `ArgumentError`. + +Audio input works on Android and iOS with a model that takes audio, such as Gemma 3n. ## Bounding output ```dart final session = await model.createSession(); await session.addQueryChunk(Message(text: prompt, isUser: true)); +final reply = StringBuffer(); var produced = 0; await for (final token in session.getResponseAsync()) { - stdout.write(token); + reply.write(token); if (++produced >= 200) { await session.stopGeneration(); break; @@ -64,7 +84,7 @@ await session.close(); ## Web -Load the MediaPipe runtime in `web/index.html` before Flutter starts: +Add to `web/index.html` ``, before Flutter boots: ```html + + ``` -Pin the version — an unpinned import takes whatever was published last. +Pin the version — an unpinned import takes whatever was published last. Copy `cache_api.js` and `opfs_helper.js` from the `flutter_gemma` package's `web/` directory into the app's `web/`; find it with `grep -A1 '"name": "flutter_gemma"' .dart_tool/package_config.json`. Web is GPU-only. Models over about 2 GB need OPFS streaming storage: @@ -85,8 +107,6 @@ await FlutterGemma.initialize( ); ``` -The storage modes need helper scripts in `web/` — see `references/platform-setup.md` in the flutter-gemma-inference skill. - ## Android -Text inference runs on `arm64-v8a`, `x86_64` and `armeabi-v7a`. No manifest changes are needed. +Text inference runs on `arm64-v8a`, `x86_64` and `armeabi-v7a`. The release build needs `` in `android/app/src/main/AndroidManifest.xml` to download a model — Flutter's template declares it only for debug and profile builds. diff --git a/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md index f494f885..a478017f 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-onnx/SKILL.md @@ -1,21 +1,27 @@ --- name: flutter-gemma-onnx -description: Use when running ONNX models with flutter_gemma_onnx — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. Not for .litertlm or .task models. +description: Use when running ONNX models with flutter_gemma_onnx (ModelFileType.onnx) — ORT-GenAI text generation (e.g. Phi-3.5-mini) or ONNX embeddings — on macOS arm64, Linux x64, Windows x64, Android arm64, iOS arm64, or in the browser through Transformers.js. Also use when an ONNX install is routed to the wrong engine, genai_config.json is missing, or getActiveModel throws "No inference engine can handle this model" on another platform. For .litertlm models use flutter-gemma-inference; for .task, flutter-gemma-mediapipe. --- # The ONNX engine ## Rules -1. Declare `fileType: ModelFileType.onnx`. Without it the install defaults to `task` and never reaches `OnnxEngine`. -2. An ORT-GenAI model is a directory — `genai_config.json`, the `.onnx` graph, its weights and a tokenizer. Install it with `fromHuggingFace(repo)`, which downloads the whole folder, or point `fromFile` at a local `genai_config.json`. A single-file download or a Flutter asset cannot produce it. -3. Native generation runs on macOS arm64, Linux x64, Windows x64, Android arm64 and iOS arm64. Anywhere else no engine accepts the model and `getActiveModel` throws `No inference engine can handle this model`. -4. Android needs `minSdk 24`. Phi-3.5-mini peaks near 3.7 GB of RAM — target 8 GB devices. -5. Text only: no images, no audio, no LoRA. +1. Depend on `flutter_gemma` and `flutter_gemma_onnx`, and import both. The engine package does not re-export core. +2. Declare `fileType: ModelFileType.onnx`. Without it the install defaults to `task` and never reaches `OnnxEngine`. +3. An ORT-GenAI model is a directory — `genai_config.json`, the `.onnx` graph, its weights and a tokenizer. Install it with `fromHuggingFace(repo)`, which downloads the whole folder, or point `fromFile` at a local `genai_config.json`. A single-file download or a Flutter asset cannot produce it. +4. Native generation runs on macOS arm64, Linux x64, Windows x64, Android arm64 and iOS arm64. Anywhere else no engine accepts the model and `getActiveModel` throws `No inference engine can handle this model`. +5. Android needs `minSdk 24`. Phi-3.5-mini peaks near 3.7 GB of RAM — target 8 GB devices. +6. Text only: no images, no audio, no LoRA. ## Setup +```sh +flutter pub add flutter_gemma flutter_gemma_onnx +``` + ```dart +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_onnx/flutter_gemma_onnx.dart'; await FlutterGemma.initialize( @@ -24,27 +30,38 @@ await FlutterGemma.initialize( ); await FlutterGemma.installModel( - modelType: ModelType.general, + modelType: ModelType.phi, fileType: ModelFileType.onnx, ).fromHuggingFace('microsoft/Phi-3.5-mini-instruct-onnx').install(); -final model = await FlutterGemma.getActiveModel(maxTokens: 4096); +final InferenceModel model = await FlutterGemma.getActiveModel(maxTokens: 4096); ``` A repo with several execution-provider folders resolves to its CPU/mobile folder automatically — the bundled runtime is CPU-only. -A bundle you ship yourself: +A bundle shipped with the app: ```dart await FlutterGemma.installModel( - modelType: ModelType.general, + modelType: ModelType.phi, fileType: ModelFileType.onnx, ).fromFile('$path/genai_config.json').install(); ``` +Sessions, chats and streaming work as in the flutter-gemma-inference skill. Pass `modelType` to `createChat` for function calling — ONNX falls back to `gemmaIt`. + ## Web -On web `OnnxEngine` runs the model through Transformers.js. The model is a Hugging Face repo id (e.g. `onnx-community/Qwen2.5-0.5B-Instruct`) that the browser downloads and caches on first use. `PreferredBackend.cpu` forces WASM; anything else tries WebGPU first. +On web `OnnxEngine` runs the model through Transformers.js. Install it by Hugging Face repo URL; the browser downloads and caches the files on first use: + +```dart +await FlutterGemma.installModel( + modelType: ModelType.qwen, + fileType: ModelFileType.onnx, +).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install(); +``` + +The repo must be in Transformers.js layout, as the `onnx-community` ones are. ORT-GenAI repos such as `microsoft/Phi-3.5-mini-instruct-onnx` do not run in the browser. `fromFile` and `fromAsset` throw `UnsupportedError` on web; `fromBundled('')` serves a model from the app's own origin. `PreferredBackend.cpu` forces WASM; anything else tries WebGPU first. Add to `web/index.html` ``, before Flutter boots — the first script for generation, the second for embeddings: diff --git a/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md index 4baff760..4e7614d6 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-rag/SKILL.md @@ -1,46 +1,61 @@ --- name: flutter-gemma-rag -description: Use when adding RAG, semantic search or text embeddings to a flutter_gemma app — searching the user's documents on-device, an embedding model plus a vector store (flutter_gemma_rag_sqlite or flutter_gemma_rag_qdrant). Also use when a metadata filter returns unfiltered results, retrieval quality is poor, addDocument throws about a missing embedding model, or the vector store throws UnimplementedError on web. +description: Use when adding RAG, semantic search or text embeddings to a flutter_gemma app — searching the user's documents on-device, an embedding model plus a vector store (flutter_gemma_rag_sqlite or flutter_gemma_rag_qdrant). Also use when a metadata filter returns unfiltered results, retrieval quality is poor, addDocument throws about a missing embedding model, the vector store fails to open on a phone, or it throws UnimplementedError on web. --- # On-device RAG with flutter_gemma ## Rules -1. Use the `FlutterGemma.rag` facade: `initialize`, `addDocument`, `searchSimilar`. It embeds documents and queries with the correct task types for you. -2. Declare every field you will filter on in `filterSchema:` at `initialize`. A filter on an undeclared field — or any filter with no schema — is silently ignored and returns unfiltered results. -3. Activate an embedding model with `getActiveEmbedder()` before `addDocument`. -4. `LiteRtEmbeddingBackend` comes from `flutter_gemma_litertlm`, not `flutter_gemma_embeddings`. -5. On web use `WebSqliteVectorStore`; `SqliteVectorStore` throws `UnimplementedError` there. `flutter_gemma_rag_qdrant` is native-only. -6. Android needs `minSdk 30`. +1. Use the `FlutterGemma.rag` facade: `initialize`, `addDocument`, `searchSimilar`. It embeds documents and queries with the correct task types. +2. Declare every field used in a filter in `filterSchema:` at `initialize`. A filter on an undeclared field — or any filter with no schema — is silently ignored and returns unfiltered results. +3. On native, give `rag.initialize` an absolute path in a writable directory. A bare name resolves against the process working directory, which is not writable on Android or iOS. +4. Activate an embedding model with `getActiveEmbedder()` before `addDocument`. +5. `LiteRtEmbeddingBackend` comes from `flutter_gemma_litertlm`, not `flutter_gemma_embeddings`. +6. On web use `WebSqliteVectorStore`; `SqliteVectorStore` throws `UnimplementedError` there. `flutter_gemma_rag_qdrant` is native-only. +7. Android needs `minSdk 30`. ## Setup +```sh +flutter pub add flutter_gemma flutter_gemma_litertlm flutter_gemma_rag_sqlite path_provider +``` + ```dart import 'package:flutter/foundation.dart'; +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_litertlm/flutter_gemma_litertlm.dart'; import 'package:flutter_gemma_rag_sqlite/flutter_gemma_rag_sqlite.dart'; +import 'package:path_provider/path_provider.dart'; + +const hfToken = String.fromEnvironment('HUGGINGFACE_TOKEN'); +const embeddingGemma = + 'https://huggingface.co/litert-community/embeddinggemma-300m/resolve/main'; await FlutterGemma.initialize( - inferenceEngines: [LiteRtLmEngine()], embeddingBackends: [LiteRtEmbeddingBackend()], vectorStore: kIsWeb ? WebSqliteVectorStore() : SqliteVectorStore(), filterSchema: const FilterSchema(fields: [ FilterField(name: 'lang', type: FilterFieldType.string), FilterField(name: 'year', type: FilterFieldType.number), ]), + huggingFaceToken: hfToken.isEmpty ? null : hfToken, ); await FlutterGemma.installEmbedder() - .modelFromNetwork(url) - .tokenizerFromNetwork(url) + .modelFromNetwork('$embeddingGemma/embeddinggemma-300M_seq512_mixed-precision.tflite') + .tokenizerFromNetwork('$embeddingGemma/sentencepiece.model') .install(); -await FlutterGemma.getActiveEmbedder(); +final EmbeddingModel embedder = await FlutterGemma.getActiveEmbedder(); -await FlutterGemma.rag.initialize('rag.db'); +await FlutterGemma.rag.initialize( + kIsWeb ? 'rag.db' : '${(await getApplicationDocumentsDirectory()).path}/rag.db', +); ``` -`rag.initialize` takes a database file for sqlite and a directory for qdrant. +EmbeddingGemma is a gated repo: the token's Hugging Face account must have accepted the Gemma licence, and the token ships inside the app — on web inside `main.dart.js`. `seq512` in the file name is the input window in tokens; `seq256`, `seq1024` and `seq2048` variants sit in the same repo. + +`rag.initialize` takes a database file for sqlite and a directory for qdrant. On native it persists across launches at that path. Add `inferenceEngines:` from the flutter-gemma-inference skill when the app also generates answers from the results. ## Index and search @@ -53,7 +68,7 @@ await FlutterGemma.rag.addDocument( metadata: jsonEncode({'lang': 'en', 'year': 2024}), ); -final hits = await FlutterGemma.rag.searchSimilar( +final List hits = await FlutterGemma.rag.searchSimilar( query: question, topK: 5, filter: const Filter( @@ -62,21 +77,23 @@ final hits = await FlutterGemma.rag.searchSimilar( ), ); for (final hit in hits) { - print('${hit.similarity.toStringAsFixed(2)} ${hit.content}'); + print('${hit.id} ${hit.similarity.toStringAsFixed(2)} ${hit.content}'); } ``` -`searchSimilar` takes the question as text and embeds it itself. Filter operators: `FieldEquals`, `FieldRange` (`gte`, `lte`), `FieldMatchAny`, combined with `must`, `should` and `mustNot`. +`searchSimilar` takes the question as text and embeds it itself. Each `RetrievalResult` has `id`, `content`, `similarity` and `metadata`. Filter operators: `FieldEquals`, `FieldRange` (`gte`, `lte`), `FieldMatchAny`, combined with `must`, `should` and `mustNot`. + +`addDocument` with an existing `id` replaces that document. `FlutterGemma.rag.removeDocument(id:)` deletes one; `FlutterGemma.rag.clear()` empties the store. ## Traps **Filter has no effect** - Symptom: results ignore the filter; no error. -- Fix: declare the field in `filterSchema`. Names must match `^[A-Za-z][A-Za-z0-9_]*$` and cannot be `id`, `embedding`, `content`, `metadata`, `distance` or `k`. At most 16 fields. +- Fix: declare the field in `filterSchema`. With `flutter_gemma_rag_sqlite`, names must match `^[A-Za-z][A-Za-z0-9_]*$` and cannot be `id`, `embedding`, `content`, `metadata`, `distance` or `k`; at most 16 fields. **Poor retrieval after embedding by hand** - Query and document embeddings are trained asymmetrically. `generateEmbedding` defaults to `TaskType.retrievalQuery`, so text embedded for indexing without a task type gets the query prefix. -- Fix: pass `TaskType.retrievalDocument` when indexing yourself: +- Fix: pass `TaskType.retrievalDocument` when indexing by hand: ```dart final vector = await embedder.generateEmbedding( @@ -91,7 +108,12 @@ await FlutterGemma.rag.addDocumentWithEmbedding( ``` **`addDocument` throws** -- Cause: no active embedding model. Call `FlutterGemma.getActiveEmbedder()` after installing one. +- Symptom: `No embedding model is active. addDocument(content:) and searchSimilar(query:) auto-embed text, which requires an embedding model.` +- Fix: install an embedder and call `FlutterGemma.getActiveEmbedder()` first. + +**Store fails to open on a phone** +- Cause: a bare name such as `'rag.db'` passed to `rag.initialize` on Android or iOS. +- Fix: an absolute path under `getApplicationDocumentsDirectory()`, as in Setup. ## Backend @@ -100,10 +122,12 @@ Leave the embedder on the default CPU backend. The GPU delegate does not produce ## Web - Copy `web/rag/sqlite3.wasm` from the `flutter_gemma_rag_sqlite` package into the app as `web/rag/sqlite3.wasm`. -- Web embeddings need `litert_embeddings.js` and `sentencepiece.js` from the `flutter_gemma_embeddings` package's `web/` directory, copied into the app's `web/`, plus `` in `web/index.html`. +- Web embeddings need `litert_embeddings.js` and `sentencepiece.js` from the `web/` directory of `flutter_gemma_embeddings` — a dependency of `flutter_gemma_litertlm`, so it is already resolved — copied into the app's `web/`, plus `` in `web/index.html`. Find a package's directory with `grep -A1 '"name": "flutter_gemma_rag_sqlite"' .dart_tool/package_config.json`. ## Chunking -Splitting documents, chunk size and overlap are yours to decide. A chunk longer than the embedding model's input window is truncated without an error. +Splitting documents, chunk size and overlap are the app's to decide. Keep each chunk within the embedding model's window — 512 tokens for `seq512`; a longer one is truncated without an error. + +For ONNX embedding models see the flutter-gemma-onnx skill. diff --git a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md index c8766b95..7af60c7f 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md @@ -1,23 +1,29 @@ --- name: flutter-gemma-speech -description: Use when adding speech to a flutter_gemma app — speech-to-text (transcribe a voice note, dictation, Whisper, moonshine, Parakeet), text-to-speech (Matcha, Qwen3-TTS, Inflect), or a push-to-talk voice assistant with VoiceSession. Also use when transcripts come back in English for non-English audio, synthesized audio plays at the wrong pitch, or getActiveTts throws a StateError about the language. For LLM text generation, use flutter-gemma-inference. +description: Use when adding speech to a flutter_gemma app — speech-to-text (transcribe a voice note, dictation, Whisper, moonshine, Parakeet), text-to-speech (Matcha, Qwen3-TTS, Inflect), or a push-to-talk voice assistant with VoiceSession. Also use when transcripts come back in English for non-English audio, a WAV file has to become 16 kHz PCM, synthesized audio plays at the wrong pitch, or getActiveTts throws a StateError about the language. For audio sent straight to Gemma in a chat, use flutter-gemma-inference. --- # Speech with flutter_gemma_speech ## Rules -1. Audio input is 16 kHz, mono, 16-bit little-endian PCM. Not a WAV file (strip its 44-byte header), not 44.1 or 48 kHz. Nothing is resampled for you. -2. Play synthesized audio at `synth.sampleRate`. It differs per model. -3. Only Whisper has a selectable output language. moonshine-tiny and Parakeet are English-only, and passing a language to them throws `ArgumentError`. -4. STT language: set a default with `getActiveStt(language:)` or override one call with `transcribe(pcm, language:)`. Nothing reloads. -5. TTS language: `close()` the synthesizer first. Asking a live synthesizer for another language throws `StateError`. -6. Android needs `minSdk 30`. There is no web support — the web backends throw `UnsupportedError`. -7. Close recognizers and synthesizers. +1. Depend on `flutter_gemma` and `flutter_gemma_speech`, and import both. The speech package does not re-export core. +2. `transcribe` takes raw PCM — 16 kHz, mono, 16-bit little-endian, as a `Uint8List` — and returns the text. Not a WAV file, not 44.1 or 48 kHz: nothing resamples or converts it. +3. Play synthesized audio at `synth.sampleRate`. It differs per model. +4. Only Whisper has a selectable output language. moonshine-tiny and Parakeet are English-only, and passing a language to them throws `ArgumentError`. +5. STT language: set a default with `getActiveStt(language:)` or override one call with `transcribe(pcm, language:)`. Nothing reloads. +6. TTS language: `close()` the synthesizer first. Asking a live synthesizer for another language throws `StateError`. +7. Android needs `minSdk 30`. There is no web support — the web backends throw `UnsupportedError`. +8. Close recognizers and synthesizers. ## Setup +```sh +flutter pub add flutter_gemma flutter_gemma_speech +``` + ```dart +import 'package:flutter_gemma/flutter_gemma.dart'; import 'package:flutter_gemma_speech/flutter_gemma_speech.dart'; await FlutterGemma.initialize( @@ -26,9 +32,11 @@ await FlutterGemma.initialize( ); ``` +Speech runs on the same native libraries as the `.litertlm` engine. Its build setup — Android `minSdk 30`, the Apple entries — is in `references/platform-setup.md` of the flutter-gemma-inference skill. + ## Speech-to-text -An STT model is two files — the model and its tokenizer — usually from different repos. +An STT model is two files — the model and its tokenizer — usually from different repos. `install()` skips files already on disk. ```dart await FlutterGemma.installStt() @@ -37,10 +45,10 @@ await FlutterGemma.installStt() .ofType(SttModelType.whisper) .install(); -final recognizer = await FlutterGemma.getActiveStt(language: 'de'); +final SpeechRecognizer recognizer = await FlutterGemma.getActiveStt(language: 'de'); try { - final german = await recognizer.transcribe(germanPcm); - final french = await recognizer.transcribe(frenchPcm, language: 'fr'); + final String german = await recognizer.transcribe(germanPcm); + final String french = await recognizer.transcribe(frenchPcm, language: 'fr'); } finally { await recognizer.close(); } @@ -52,7 +60,48 @@ try { | `whisper` | 99, selectable, default `'en'` | 30 s | | `parakeet` | English | 5 s, desktop only (2.35 GB) | -Longer audio has to be split by the caller. +Audio longer than the window has to be split by the caller. + +Whisper tiny is weak outside English. Whisper base is more accurate; install it the same way from `https://huggingface.co/litert-community/whisper-base/resolve/main/whisper_base_30s_i8.tflite` with the tokenizer `https://huggingface.co/openai/whisper-base/resolve/main/tokenizer.json`. + +## Getting 16 kHz mono PCM + +The package has no resampler and no WAV reader. Record in the right format from the start — with the `record` package: + +```dart +import 'package:record/record.dart'; + +const config = RecordConfig( + encoder: AudioEncoder.wav, + sampleRate: 16000, + numChannels: 1, +); +``` + +That produces a WAV file. Its header is not always 44 bytes — take the samples from the `data` chunk: + +```dart +import 'dart:typed_data'; + +/// The samples of a 16 kHz mono 16-bit WAV file, without its header. +Uint8List pcmFromWav(Uint8List wav) { + final view = ByteData.sublistView(wav); + var offset = 12; // after 'RIFF', the size and 'WAVE' + while (offset + 8 <= wav.length) { + final id = String.fromCharCodes(wav, offset, offset + 4); + final size = view.getUint32(offset + 4, Endian.little); + final start = offset + 8; + if (id == 'data') { + final end = start + size > wav.length ? wav.length : start + size; + return Uint8List.sublistView(wav, start, end); + } + offset = start + size + (size & 1); // chunks are padded to an even size + } + throw const FormatException('WAV file has no data chunk'); +} +``` + +A file recorded at another rate or channel count — 44.1 kHz stereo, say — has to be converted first: average the channels to mono, then resample with a low-pass filter. Dropping samples instead aliases and costs accuracy. ## Traps @@ -73,7 +122,7 @@ await FlutterGemma.installTts() .ofType(TtsModelType.matcha) .install(); -final synth = await FlutterGemma.getActiveTts(); +final SpeechSynthesizer synth = await FlutterGemma.getActiveTts(); try { final audio = await synth.synthesize('Hello world.'); // 16-bit PCM final rate = synth.sampleRate; // 22050 for Matcha @@ -98,11 +147,14 @@ await english.close(); final german = await FlutterGemma.getActiveTts(language: 'german'); ``` +Without the `close()`, the second call throws `StateError: Active TTS synthesizer was created for language 'english'; call close() before requesting 'german'.` + ## Voice assistant `VoiceSession` runs one push-to-talk turn: transcribe, generate, speak, with barge-in. It uses the recognizer's current language. ```dart +final reply = StringBuffer(); final voice = VoiceSession.fromChat( recognizer: await FlutterGemma.getActiveStt(language: 'de'), chat: chat, @@ -114,7 +166,7 @@ await for (final event in voice.runTurn(pcm16kMono)) { case VoiceTranscriptEvent(:final text): print('heard: $text'); case VoiceReplyTextEvent(:final chunk): - stdout.write(chunk); + reply.write(chunk); case VoiceReplyAudioEvent(:final sampleRate): print('audio at $sampleRate Hz'); case VoiceTurnInterruptedEvent(): @@ -127,4 +179,4 @@ await for (final event in voice.runTurn(pcm16kMono)) { } ``` -A chat created with tools also needs `onToolCall:` — without it `fromChat` throws. +`chat` is an `InferenceChat` from the flutter-gemma-inference skill. A chat created with tools also needs `onToolCall:` — without it `fromChat` throws. diff --git a/tool/check_skills.dart b/tool/check_skills.dart index e5ec285e..12f4f488 100644 --- a/tool/check_skills.dart +++ b/tool/check_skills.dart @@ -16,7 +16,11 @@ // - every ```dart fence, each as its own async function body. Imports written // in any fence of a skill are hoisted to that skill's file, so a snippet // using a satellite class must show the import that provides it — which is -// the thing an agent needs to know anyway. +// the thing an agent needs to know anyway. The preamble adds only +// `dart:typed_data`, for the placeholders. Core in particular is NOT +// injected: the engine packages do not re-export it, and an earlier version +// of this script that imported it here was green while every skill taught +// code failing with "Undefined name 'FlutterGemma'". // - every inline `TypeName` in the prose, as a typed declaration. // - every inline `TypeName.member`, as an expression. An instance member // referenced this way reports `static_access_to_instance_member`, which the @@ -47,10 +51,7 @@ const _hostPackage = 'packages/flutter_gemma/example'; const _preamble = ''' // GENERATED by tool/check_skills.dart — deleted after each run. Do not edit. // ignore_for_file: type=lint, unused_import, unused_local_variable, unused_element, dead_code, unused_catch_clause, dead_code_on_catch_subtype -import 'dart:async'; -import 'dart:io'; import 'dart:typed_data'; -import 'package:flutter_gemma/flutter_gemma.dart'; '''; /// Inputs a snippet may use without declaring them. Data and already-built diff --git a/website/content/docs/embeddings-and-rag.md b/website/content/docs/embeddings-and-rag.md index 2f8179e1..dc2f3b4b 100644 --- a/website/content/docs/embeddings-and-rag.md +++ b/website/content/docs/embeddings-and-rag.md @@ -86,8 +86,11 @@ import 'package:flutter_gemma/flutter_gemma.dart'; // 1. Install an embedding model (any of Gecko / EmbeddingGemma) — see above. -// 2. Initialize the vector store (one shard per database path) -await FlutterGemma.rag.initialize('rag_store'); +// 2. Initialize the vector store (one shard per database path). On native pass +// an absolute path: a bare name resolves against the process working +// directory, which is not writable on Android or iOS. On web a name is enough. +final dir = await getApplicationDocumentsDirectory(); // package:path_provider +await FlutterGemma.rag.initialize('${dir.path}/rag_store'); // 3. Add documents — let flutter_gemma compute embeddings for you for (final doc in docs) { diff --git a/website/content/docs/migration.md b/website/content/docs/migration.md index 194b0942..4329b25f 100644 --- a/website/content/docs/migration.md +++ b/website/content/docs/migration.md @@ -269,7 +269,8 @@ await FlutterGemma.installEmbedder() .modelFromNetwork(modelUrl, token: token) .tokenizerFromNetwork(tokenizerUrl, token: token) .install(); -await FlutterGemma.rag.initialize('rag_store'); +final dir = await getApplicationDocumentsDirectory(); // native; on web pass a bare name +await FlutterGemma.rag.initialize('${dir.path}/rag_store'); await FlutterGemma.rag.addDocument(/* ... */); final hits = await FlutterGemma.rag.searchSimilar(query: query, topK: 5); ``` From 38a9192aa63c689a4074a7b6b408988d0e54e346 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Fri, 11 Sep 2026 09:50:52 +0200 Subject: [PATCH 07/21] docs(readme): install the agent skills with --all Run from the example app, `dart run skills@ get` installed nothing. With no agent directory it cannot detect the agent and stops; given one with `--agent`, it lists the seven skills and asks for `--skill` or `--all`. `--all` installs all seven plus references/platform-setup.md. --- packages/flutter_gemma/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/flutter_gemma/README.md b/packages/flutter_gemma/README.md index cbc1f21d..fe0fb2f2 100644 --- a/packages/flutter_gemma/README.md +++ b/packages/flutter_gemma/README.md @@ -61,10 +61,10 @@ There is an example of using: `flutter_gemma` ships [agent skills](https://dart.dev/blog/skills-cli-1-0-bundle-and-distribute-ai-agent-skills-for-your-packages) — short instruction files your coding assistant reads so it uses this API correctly the first time: ```bash -dart run skills@ get +dart run skills@ get --all ``` -That scans your dependencies, offers the skills each one bundles, and installs them where your agent looks (Claude Code, Codex, Cursor, Antigravity, Cline, Copilot, OpenCode are all supported). +That scans your dependencies and installs every skill they bundle where your agent looks — Claude Code, Codex, Cursor, Antigravity, Cline, Copilot and OpenCode are supported. If it reports that it could not detect your agent, name it with `--agent claude` (or `codex`, `cursor`, …). What they cover: registering an engine (core ships none), routing by the declared `ModelFileType` rather than the filename, and the two defaults that fail quietly — `maxTokens` is the context window and not the reply length, and `Message.isUser` defaults to `false`. From 29d611a6befb8e9297a0e06a820a7af1f473c86d Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Fri, 11 Sep 2026 11:30:44 +0200 Subject: [PATCH 08/21] =?UTF-8?q?docs(site):=20AI=20Assistant=20Skills=20?= =?UTF-8?q?=E2=80=94=20landing=20card=20and=20docs=20page?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A feature card in the landing grid and /docs/ai-assistants under Features: the install command, where each assistant's skills land (from the skills CLI's own agent table), how the skills load, the seven skills, and the defaults they spell out. Named apart from "Agent Skills", which is flutter_gemma_agent — skills the on-device model runs — and the page says so up front. Shell blocks use a bare fence: the site's highlighter knows only dart, and a bash fence fails the page with a null check in Highlighter. --- website/content/docs/ai-assistants.md | 97 ++++++++++++++++++++++ website/lib/landing/sections/features.dart | 6 ++ website/lib/main.server.dart | 1 + 3 files changed, 104 insertions(+) create mode 100644 website/content/docs/ai-assistants.md diff --git a/website/content/docs/ai-assistants.md b/website/content/docs/ai-assistants.md new file mode 100644 index 00000000..a9562739 --- /dev/null +++ b/website/content/docs/ai-assistants.md @@ -0,0 +1,97 @@ +--- +title: AI Assistant Skills +description: flutter_gemma ships agent skills for coding assistants — Claude Code, Codex, Cursor, Copilot and others learn the API from skills bundled in the package, installed with one dart run skills@ get. +image: https://fluttergemma.dev/images/og-image.png +--- + +A coding assistant that has never seen flutter_gemma writes plausible code that +fails. It installs a `.litertlm` model without declaring its file type and lands +on the wrong engine, reads `maxTokens` as the reply length, or leaves out +`isUser: true` and gets an empty reply. + +flutter_gemma ships **agent skills** — instruction files in the open +[Agent Skills](https://agentskills.io) format — inside the package itself. +Dart's [`skills` CLI](https://dart.dev/blog/skills-cli-1-0-bundle-and-distribute-ai-agent-skills-for-your-packages) +copies them into your project, where your assistant picks them up. + + +Not to be confused with Agent Skills — the +flutter_gemma_agent package, which gives the on-device model +skills to run. The skills on this page teach your coding assistant to +use flutter_gemma. + + +## Install + +From your app's root, once `flutter_gemma` 1.8.1 or later is a dependency: + +``` +dart run skills@ get --all +``` + +The CLI scans your dependencies, finds the skills they bundle, and installs +them for the assistant it detects in the project: + +| Assistant | `--agent` | Installed to | +|-----------|-----------|--------------| +| Claude Code | `claude` | `.claude/skills/` | +| Codex, Antigravity | `codex`, `antigravity` | `.agents/skills/` | +| Cursor | `cursor` | `.cursor/skills/` | +| GitHub Copilot | `copilot` | `.github/skills/` | +| Cline | `cline` | `.cline/skills/` | +| OpenCode | `opencode` | `.opencode/skills/` | +| any other | `generic` | `.agents/skills/` | + +Detection looks for the assistant's directory in the project root. When there is +none yet, the CLI stops with *Could not auto-detect agent* — name the assistant: + +``` +dart run skills@ get --all --agent claude +``` + +Copilot is never auto-detected, because `.github/` serves many other purposes, +so it always needs `--agent copilot`. `--all` installs everything without +asking; `--skill ` installs a single skill. The CLI records what it +installed in `.config/dart_skills/skills_config.json`. + +Run the same command after upgrading flutter_gemma: it updates the installed +skills to the ones the new version ships. + +## How your assistant uses them + +Each skill is a folder with a `SKILL.md`: a description of when it applies, then +rules, working code, and the traps that fail without an error. The assistant +reads only the descriptions up front and loads a skill when a task matches — +"add an offline chat with Gemma" pulls in the inference skill, "search my notes +by meaning" the RAG one. In Claude Code they install as skills the model invokes +on its own; there is no command to remember. + +## The skills + +| Skill | Covers | +|-------|--------| +| `flutter-gemma-inference` | engines, installing a model from Hugging Face, sessions and chats, streaming, system prompts, images and audio, backends — and the platform setup for Android, iOS, macOS, Windows, Linux and web | +| `flutter-gemma-function-calling` | declaring tools, `FunctionCallResponse`, the built-in tool loop, returning errors as results | +| `flutter-gemma-rag` | embedding models, `flutter_gemma_rag_sqlite` and `flutter_gemma_rag_qdrant`, metadata filters and their schema | +| `flutter-gemma-speech` | Whisper, moonshine and Parakeet STT; Matcha, Qwen3 and Inflect TTS; 16 kHz PCM; `VoiceSession` | +| `flutter-gemma-mediapipe` | `.task` and `.bin` models on Android, iOS and web | +| `flutter-gemma-onnx` | ORT-GenAI generation and ONNX embeddings, native and through Transformers.js | +| `flutter-gemma-builtin-ai` | Gemini Nano and Apple Foundation Models, availability, falling back to a downloaded model | + +## What they prevent + +The skills spell out the defaults that fail quietly, each with its fix: + +- `maxTokens` is the context window, not the reply length — cap a reply with `maxOutputTokens`. +- `Message.isUser` defaults to `false`; a prompt without it gets an empty reply. +- The declared `fileType`, not the file name, picks the engine, and it defaults to `.task`. +- A metadata filter on a field missing from `filterSchema` is ignored, and search returns unfiltered results. +- Speech-to-text takes raw 16 kHz mono PCM. A WAV file or 48 kHz audio is not rejected — it is transcribed wrong. +- The engine packages do not re-export core: import `package:flutter_gemma/flutter_gemma.dart` as well. + +## Kept in step with the code + +Every code block in the skills is compiled against the real packages before each +release, and a release that changes an API a skill describes gets that skill +re-read and updated. The skills ship inside the package, so upgrading +flutter_gemma and re-running the CLI keeps your assistant current. diff --git a/website/lib/landing/sections/features.dart b/website/lib/landing/sections/features.dart index 16d70a8f..730f6d0e 100644 --- a/website/lib/landing/sections/features.dart +++ b/website/lib/landing/sections/features.dart @@ -77,6 +77,12 @@ const _features = [ desc: 'Core + 11 opt-in packages — agent skills, speech, RAG & more; ship only what you use', accent: Brand.green, ), + _FeatureData( + icon: '🧑‍💻', + title: 'AI Assistant Skills', + desc: 'Claude Code, Codex, Cursor & Copilot learn the flutter_gemma API from skills shipped in the package', + accent: Brand.blue, + ), ]; /// Feature grid. diff --git a/website/lib/main.server.dart b/website/lib/main.server.dart index 4469d0e3..7e053aff 100644 --- a/website/lib/main.server.dart +++ b/website/lib/main.server.dart @@ -156,6 +156,7 @@ void main() { SidebarLink(text: 'Agent Skills', href: '/docs/agent'), SidebarLink(text: 'Speech', href: '/docs/speech'), SidebarLink(text: 'Embeddings & RAG', href: '/docs/embeddings-and-rag'), + SidebarLink(text: 'AI Assistant Skills', href: '/docs/ai-assistants'), ], ), SidebarGroup( From 59f380eaa365d580f7f4145980a9a3c45999f1b6 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 00:36:32 +0200 Subject: [PATCH 09/21] docs(builtin-ai): Edge runs Phi-4-mini; web download needs a user gesture The site page and the skill said the web arm runs Gemini Nano in "desktop Chrome/Chromium-Edge". Edge has no Gemini Nano. Measured on stable Edge 151 on macOS with the "Prompt API for on-device language model" flag: availability() answers "downloadable", Phi-4-mini downloads in about five minutes, and then prompt, streaming, abort and measureContextUsage all work with the options flutter_gemma sends; the context window is 9216 tokens. Edge Dev 154-155 exposes LanguageModel but cannot run the model (MSEdgeExplainers#1392). Chrome and Edge both reject the first create() while the model is downloadable unless the user has interacted with the page, so web ensureReady() has to be called from a tap handler. Neither page said so. --- .../skills/flutter-gemma-builtin-ai/SKILL.md | 8 +++-- website/content/docs/builtin-ai.md | 31 ++++++++++++++----- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md index 0c18840d..624d9eea 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-builtin-ai/SKILL.md @@ -1,6 +1,6 @@ --- name: flutter-gemma-builtin-ai -description: Use when running the device's own model with flutter_gemma_builtin_ai — Gemini Nano on Android or in desktop Chrome, Apple Foundation Models on iPhone, iPad and Mac — with nothing to download or bundle, or when falling back to a downloaded model where it is missing. Also use when BuiltInAiUnavailableException or a TimeoutException is thrown, availability reports "downloadable", the Android build fails the manifest merge on minSdk, or the model is missing in Chrome. For models the app downloads itself, use flutter-gemma-inference. +description: Use when running the device's own model with flutter_gemma_builtin_ai — Gemini Nano on Android or in desktop Chrome, Phi-4-mini in Microsoft Edge, Apple Foundation Models on iPhone, iPad and Mac — with nothing to download or bundle, or when falling back to a downloaded model where it is missing. Also use when BuiltInAiUnavailableException or a TimeoutException is thrown, availability reports "downloadable", web throws NotAllowedError about a user gesture, the Android build fails the manifest merge on minSdk, or the model is missing in Chrome. For models the app downloads itself, use flutter-gemma-inference. --- # The built-in OS model @@ -8,7 +8,7 @@ description: Use when running the device's own model with flutter_gemma_builtin_ ## Rules 1. The OS owns the weights, but the model is still installed — as an identity: `fileType: ModelFileType.builtIn` with `.fromBundled(...)`. The app downloads nothing. -2. Call `BuiltInAi.ensureReady()` before `getActiveModel()`, from a user action: on Android the first call downloads the model and can take minutes. It throws `TimeoutException` after `timeout` — 10 minutes by default. +2. Call `BuiltInAi.ensureReady()` before `getActiveModel()`, from a user action: the first call downloads the model and can take minutes, and on web the browser will not start that download without a user gesture ("NotAllowedError: Requires a user gesture"). Call it straight from the tap handler, with no other `await` before it. It throws `TimeoutException` after `timeout` — 10 minutes by default. 3. Catch `BuiltInAiUnavailableException` and fall back to a downloadable model. 4. Android apps need `minSdk 26`, or the manifest merge fails. 5. There is no Windows or Linux support. @@ -94,7 +94,8 @@ final usable = availability == BuiltInAiAvailability.available || | --- | --- | --- | | Android | Gemini Nano (AICore) | Pixel 9+, Galaxy S25+; `minSdk 26` | | iOS, macOS | Apple Foundation Models | iPhone 15 Pro+ or an Apple Silicon Mac, Apple Intelligence turned on | -| Web | Gemini Nano (Chrome Prompt API) | desktop Chrome — not Edge (its built-in model is Phi-4-mini), mobile browsers, Firefox or Safari | +| Web | Gemini Nano (Chrome Prompt API) | desktop Chrome — not mobile browsers, Firefox or Safari | +| Web | Phi-4-mini (Edge Prompt API) | Microsoft Edge with the Prompt API flag on; Edge Dev 154–155 exposes the API but cannot run the model | Images work on Android. On Apple platforms they need OS 27 — on OS 26 an image throws. The web model is text-only. @@ -109,6 +110,7 @@ There is no script to add: the Prompt API is part of the browser. It has to be e ``` - Local development — enable `chrome://flags/#prompt-api-for-gemini-nano` and restart Chrome. +- Microsoft Edge — enable `edge://flags` → "Prompt API for on-device language model" and restart. Both browsers use `BuiltInAiModels.geminiNano`: the spec names the API, and the browser picks the model. ## Trade-offs diff --git a/website/content/docs/builtin-ai.md b/website/content/docs/builtin-ai.md index f1115af8..7dc6b2d5 100644 --- a/website/content/docs/builtin-ai.md +++ b/website/content/docs/builtin-ai.md @@ -1,6 +1,6 @@ --- title: Built-in AI -description: Run the device's own OS/browser AI as an engine — Gemini Nano (Android + Web) and Apple Foundation Models (iOS/macOS) — with no model to download, plus the availability-probe → open-model fallback pattern. +description: Run the device's own OS/browser AI as an engine — Gemini Nano (Android + Chrome), Phi-4-mini (Edge) and Apple Foundation Models (iOS/macOS) — with no model to download, plus the availability-probe → open-model fallback pattern. image: https://fluttergemma.dev/images/og-image.png --- @@ -17,10 +17,11 @@ want, and the platform owns the weights. |----------|----------------|---------|-----------------| | Android | **Gemini Nano** | ML Kit GenAI / AICore | Pixel 9+, Galaxy S25+ (`minSdk 26`) | | iOS / macOS | **Apple Foundation Models** (Apple Intelligence) | FoundationModels framework | iOS 26+ / macOS 26+ on iPhone 15 Pro+, Apple Silicon Macs — Apple Intelligence enabled | -| Web | **Gemini Nano** | Chrome **Prompt API** (`self.LanguageModel`) | Desktop Chrome / Chromium-Edge only | +| Web | **Gemini Nano** in Chrome, **Phi-4-mini** in Edge | **Prompt API** (`self.LanguageModel`) | Desktop Chrome; Microsoft Edge with a flag (see [Web setup](#web-setup)) | -> **Note:** the Chrome Prompt API *is* Gemini Nano — the browser runs the same -> on-device model, exposed through a JS API. **Windows and Linux have no OS +> **Note:** in Chrome the Prompt API *is* Gemini Nano — the browser runs the same +> on-device model, exposed through a JS API. Edge implements the same API with +> Microsoft's own model, Phi-4-mini: the same calls, a different model. **Windows and Linux have no OS > built-in model** (no ML Kit, no Apple Foundation Models, no browser Prompt API > in a Flutter desktop app) — there `availability()` reports > `unavailableDeviceUnsupported`, and you fall back to a downloaded model @@ -76,7 +77,7 @@ model list. `BuiltInAi.availability()` reports whether the OS model is ready. `BuiltInAi.ensureReady()` makes sure the feature is on — and drives the on-device -download the first time it is used (Android), reporting progress: +download the first time it is used (Android and web), reporting progress: ```dart final status = await BuiltInAi.availability(); @@ -88,6 +89,13 @@ await BuiltInAi.ensureReady( // Throws BuiltInAiUnavailableException for any unavailable* status. ``` +**On web, call `ensureReady()` from a user gesture** — straight from a button's +tap handler, with no other `await` in front of it. While the model still has to +be downloaded, the browser refuses to start the session that downloads it unless +the user has interacted with the page (`NotAllowedError: Requires a user gesture +when availability is "downloading" or "downloadable"`). Chrome and Edge both +enforce this. + ## The fallback pattern The point of a pluggable engine: **use the built-in model when the device @@ -156,6 +164,9 @@ final response = await session.getResponse(); older OS. Either way the count comes from core's `text.length / 4` estimate. - **Web is text-only in this release** (image/audio dropped with a one-time log). +- **Edge:** measured on Edge 151 (macOS) with Phi-4-mini — streaming, stopping + and `measureContextUsage` work through the same calls, and the context window + is 9216 tokens. Function calling on Phi-4-mini has not been tested. ## Web setup @@ -169,8 +180,14 @@ There is **no CDN `` in `web/index.html`. +- Web embeddings need four module files side by side in the app's `web/`: `litert_embeddings.js` and `sentencepiece.js` from `flutter_gemma_embeddings/web/`, plus `litert.js` and `tensorflow.js` from `flutter_gemma_litertlm/web/` — the first one imports the other three by relative path, so three files alone give a 404 and an embedder that never initialises. +- They also need the LiteRT WASM runtime at `web/wasm/`, which no package ships: build it once from the core package (`cd /web/rag && npm install && npm run build`) and copy `dist/wasm` into the app's `web/`. The Dart side loads it from `/wasm/` and nowhere else. +- In `web/index.html`, before Flutter boots: `` first — it is not a module, and the embedding runtime calls its cache helpers during init — then ``. Find a package's directory with `grep -A1 '"name": "flutter_gemma_rag_sqlite"' .dart_tool/package_config.json`. diff --git a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md index 40f890b6..e849f901 100644 --- a/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md +++ b/packages/flutter_gemma/skills/flutter-gemma-speech/SKILL.md @@ -56,13 +56,13 @@ try { | `SttModelType` | Languages | Window | | --- | --- | --- | -| `moonshine` | English | 5 s | -| `whisper` | 99, selectable, default `'en'` | 30 s | -| `parakeet` | English | 5 s, desktop only (2.35 GB) | +| `SttModelType.moonshine` | English | 5 s | +| `SttModelType.whisper` | 99, selectable, default `'en'` | 30 s | +| `SttModelType.parakeet` | English | 5 s; 2.35 GB, so desktop in practice — nothing refuses it on a phone | -Audio longer than the window has to be split by the caller. +**Audio longer than the window is silently truncated**, not rejected: it is zero-padded when shorter and cut when longer, so a 40-second clip on Whisper returns the first 30 seconds with no error. Split long recordings yourself. -Whisper tiny is weak outside English. Whisper base is more accurate; install it the same way from `https://huggingface.co/litert-community/whisper-base/resolve/main/whisper_base_30s_i8.tflite` with the tokenizer `https://huggingface.co/openai/whisper-base/resolve/main/tokenizer.json`. +Whisper tiny is weak outside English. Whisper base int8 is the next size up in the catalog; install it the same way from `https://huggingface.co/litert-community/whisper-base/resolve/main/whisper_base_30s_i8.tflite` with the tokenizer `https://huggingface.co/openai/whisper-base/resolve/main/tokenizer.json`. ## Getting 16 kHz mono PCM @@ -133,13 +133,13 @@ try { | `TtsModelType` | Languages | | --- | --- | -| `matcha` | fixed by the installed bundle | -| `qwen3` | `chinese`, `english`, `german`, `italian`, `portuguese`, `spanish`, `japanese`, `korean`, `french`, `russian`, or `auto` | -| `inflect` | English | +| `TtsModelType.matcha` | English, fixed by the installed bundle — it ignores `language:` | +| `TtsModelType.qwen3` | `chinese`, `english`, `german`, `italian`, `portuguese`, `spanish`, `japanese`, `korean`, `french`, `russian`, or `auto` | +| `TtsModelType.inflect` | English | -`supertonic` and `kokoro` are in the enum but throw `UnimplementedError` — do not use them. +`TtsModelType.supertonic` and `TtsModelType.kokoro` are in the enum but throw `UnimplementedError` — do not use them. -Switching the Qwen3 language — full lowercase names, not ISO codes: +Switching the Qwen3 language — full lowercase names, not ISO codes, and only with the Qwen3 bundle installed (Matcha still throws the same `StateError` but the language changes nothing): ```dart final english = await FlutterGemma.getActiveTts(language: 'english'); @@ -153,6 +153,8 @@ Without the `close()`, the second call throws `StateError: Active TTS synthesize `VoiceSession` runs one push-to-talk turn: transcribe, generate, speak, with barge-in. It uses the recognizer's current language. +Wrap the loop in `try`/`catch`: a failed stage — transcribe, generate or synthesize — arrives as a **stream error**, not as an event. `VoiceErrorEvent` is reserved in this release and never emitted; the `case` is only there because the switch must be exhaustive. To barge in, call `await voice.interrupt()` — cancelling the subscription is not a portable stop. + ```dart final reply = StringBuffer(); final voice = VoiceSession.fromChat( diff --git a/tool/check_macos_podfile_snippet.sh b/tool/check_macos_podfile_snippet.sh index c7abb1a4..07791074 100755 --- a/tool/check_macos_podfile_snippet.sh +++ b/tool/check_macos_podfile_snippet.sh @@ -72,7 +72,10 @@ extract_fenced() { { fence = 0 } ' "$1" } -for f in packages/flutter_gemma/README.md website/content/docs/desktop.md; do +# The skill's copy is the one a coding agent pastes into someone's project, so +# it is held to the same byte-identity as the rest. +for f in packages/flutter_gemma/README.md website/content/docs/desktop.md \ + packages/flutter_gemma/skills/flutter-gemma-inference/references/platform-setup.md; do if [ ! -f "$f" ]; then echo "MISSING: $f" >&2 exit 1 From 4999d824f517f0fe60ba58e30eb31e9c5b8514d7 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 16:14:42 +0200 Subject: [PATCH 15/21] docs(release): skills are fixed in the PR that changes the API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 12d checked the skills at release time, which is the last moment they could be wrong for weeks. Hard rule 4 moves the check to where the change is: any PR touching packages/*/lib, a native build file, a pinned CDN version or a pubspec floor runs the router and both gates, and reads the skills the router flags. CI runs the two gates on every PR already; what it cannot catch is a symbol that survives while its meaning moves, which is why the reading step stays human. 12d remains as the release backstop. skills_review.sh accepted a ref this checkout could not resolve and answered "no source changes" — a typo or an unfetched tag green-lit skipping every skill. It now refuses with exit 2. The checklist's "5 copies" of the macOS post_install snippet is 23 today, including the copy inside the inference skill. --- .claude/skills/release/SKILL.md | 28 +++++++++++++++++++++++++--- tool/skills_review.sh | 13 +++++++++++++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 89e8f1f2..32143a7d 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -34,6 +34,25 @@ silently do the other thing. 3. **Reproduce this Definition-of-Done in your reply and mark every item** (done / N/A + reason) before you publish. Do not publish off memory of the skill — walk it as a literal checklist against the actual repo state. +4. **The skills are fixed in the PR that changes the API, not at release time.** + `packages/flutter_gemma/skills/**` ships inside core and is read by other + people's coding agents, so a stale sentence there becomes confident, broken + code in someone else's app. On any PR touching `packages/*/lib/**`, a native + build file (`android/`, `ios/`, `darwin/`, `macos/`, `windows/`, `hook/`), a + pinned CDN version or a `pubspec.yaml` floor: + + ```bash + bash tool/skills_review.sh origin/main # which skills the diff puts in doubt + dart tool/check_skills.dart # compiles every block; exit 0 required + dart run skills_lint@0.5.1 # file-level rules; exit 0 required + ``` + + The two gates also run in CI (`skills` job), so a rename is caught without + you. What CI cannot catch is a symbol that survives while its MEANING moves — + `getActiveStt(language:)` went from "the language this recognizer was built + with" to "the default for its transcriptions" with no rename anywhere. That + is what reading the flagged skills is for. Step 12d is the release backstop, + not the first time this happens. ### Definition of Done (paste it; check 1a–12b before Step 10 publish; 12c is verified after merge) @@ -44,8 +63,10 @@ silently do the other thing. [ ] 5b manifest gate RUN and printed "N platform(s) compared" — N == number of tarballs [ ] 1e core public API changed? → upgrade-genkit (realign + version), else N/A [ ] 1f shared code duplicated across satellites patched everywhere (grep the pattern) -[ ] 1f-bis tool/check_macos_podfile_snippet.sh passes (5 copies of the macOS - post_install snippet identical) — RUN it, do not eyeball +[ ] 1f-bis tool/check_macos_podfile_snippet.sh passes (every copy of the macOS + post_install snippet byte-identical — 23 today: three example Podfiles, + the codelab step apps, README, desktop.md and the inference skill's + references/platform-setup.md) — RUN it, do not eyeball [ ] 1g each changed satellite's flutter_gemma: floor >= the core version it now needs [ ] 2 versions bumped: pubspec + podspec (if any) + CLAUDE.md Current-Version line [ ] 7 CHANGELOG: one short line per package, every published package @@ -54,7 +75,8 @@ silently do the other thing. [ ] 12b new/changed public API + behavior documented (README + website) ← SAME PR [ ] 12d skills/: `skills_review.sh ` run, every flagged skill READ, updated where the prose drifted, `dart tool/check_skills.dart` green and - `dart run skills_lint@0.5.1` green + `dart run skills_lint@0.5.1` green — backstop: rule 4 means the PRs in + this release already did it [ ] 12c after merge: firebase-hosting-merge run == success (not just triggered) ``` diff --git a/tool/skills_review.sh b/tool/skills_review.sh index 37f77dbf..e30cfef1 100755 --- a/tool/skills_review.sh +++ b/tool/skills_review.sh @@ -36,6 +36,19 @@ if [ ! -d "$SKILLS_DIR" ]; then exit 2 fi +# A ref this checkout cannot resolve — a typo, or a tag never fetched into a +# fresh clone or worktree — used to read as "nothing changed": `git diff` wrote +# to stderr, the diff file stayed empty, and the script printed the all-clear +# and exited 0. That green-lights skipping every skill, which is the one answer +# this script must never give by accident. +for r in "$FROM" "$TO"; do + if ! git rev-parse --verify -q "$r^{commit}" >/dev/null; then + echo "unknown ref: $r" >&2 + echo " fetch it first, or pass one this checkout has (git tag, git branch -a)" >&2 + exit 2 + fi +done + # Added/removed source lines only. A symbol that merely sits near a change is # not evidence; a symbol on a +/- line is. DIFF=$(mktemp) From ac7b4e125cc8906e7d167a4d655e9392cefb9fc4 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 16:25:45 +0200 Subject: [PATCH 16/21] docs(site): the landing snippet, and three sections that said the opposite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The landing's Quick start called FlutterGemma.initialize without awaiting it — engine registration and the active-model restore happen after an await inside — and installed a .litertlm model without fileType, which routes it to MediaPipe. Both are traps our own pages warn about. - getting-started said getActiveModel "discards the new arguments" and hands back the cached model. Since 1.6.1 it compares every runtime parameter, CLOSES the cached model and builds a new one, so the danger is the opposite of what was written: the old handle is dead, not stale. - embeddings-and-rag said a filter on an undeclared field "matches nothing" and returns zero hits. Both stores drop the condition, so the search returns what filter: null returns — the symptom is too many results, never zero. - thinking-mode listed SmolLM3 3B and Phi-4 Mini Reasoning as supported. They run as ModelType.general, which has no reasoning parser: no ThinkingResponse, and the tags are not stripped either. Its web row said thinking is unsupported; the Qwen3/DeepSeek split is pure Dart and runs everywhere, while Gemma 4 needs the native channel. models.md's rows and footnotes follow. --- website/content/docs/embeddings-and-rag.md | 20 ++++++------ website/content/docs/getting-started.md | 18 +++++++---- website/content/docs/models.md | 14 +++++--- website/content/docs/thinking-mode.md | 32 ++++++++++++------- website/lib/landing/sections/quick_start.dart | 9 +++--- 5 files changed, 59 insertions(+), 34 deletions(-) diff --git a/website/content/docs/embeddings-and-rag.md b/website/content/docs/embeddings-and-rag.md index f2dd6d06..4372d1ca 100644 --- a/website/content/docs/embeddings-and-rag.md +++ b/website/content/docs/embeddings-and-rag.md @@ -166,9 +166,12 @@ implementations must declare `flush()`. Both stores honor `Filter` on **all platforms**, and both need the filterable fields declared up front in a `FilterSchema` (see below). qdrant-edge promotes exactly the declared fields to payload keys at write time; sqlite-vec creates -them as columns at table-creation time. On either store a filter on an -undeclared field is a no-op — it matches nothing and never throws, so a missing -declaration looks like "no results" rather than an error. +them as columns at table-creation time. On either store a condition on an +undeclared field is **dropped**, as if it had never been written: the search +returns the same hits as `filter: null`, and a filter mixing declared and +undeclared fields narrows only by the declared ones. A missing declaration +therefore looks like "my filter had no effect" — too many results, never an +error and never zero. ### Declaring filter fields @@ -217,12 +220,11 @@ A `Filter` over the declared fields is then applied inside the store; a filter referencing an **undeclared** field is silently ignored (no-op, never throws). -Declare a `FilterSchema` on **both** stores. qdrant-edge promotes only the fields -named in the schema to payload keys — an undeclared field is absent from the -payload, so a `Filter` on it matches nothing and the search returns zero hits -rather than an error. The difference between the two stores is not "schema -optional": it is that sqlite-vec needs the schema at table-creation time, while -qdrant promotes at write time. +Declare a `FilterSchema` on **both** stores. Neither store filters on a field it +was not told about: the condition is dropped and the search comes back +unfiltered, with no error and no log in a release build. The difference between +the two is not "schema optional": sqlite-vec needs the schema at table-creation +time, while qdrant promotes at write time. ## Platform support diff --git a/website/content/docs/getting-started.md b/website/content/docs/getting-started.md index 110c27af..4524b83c 100644 --- a/website/content/docs/getting-started.md +++ b/website/content/docs/getting-started.md @@ -144,19 +144,25 @@ final chat = await model.createChat( - **`.litertlm` on every platform, native and web**: passed as a real system turn. - **MediaPipe `.task` on web**: prepended to the first user message as a fallback. -### Runtime parameters are fixed at first load +### Changing a runtime parameter reloads the model -`getActiveModel` returns a **cached singleton**. A second call for the same -active model hands back the same object and **discards** the new arguments — so -this does not produce two differently-sized models: +`getActiveModel` caches one model per active spec, but it compares every runtime +parameter — `maxTokens`, `preferredBackend`, `supportImage`, +`maxConcurrentSessions` and the rest. A call that differs in any of them +**closes the cached model and builds a new one**, logging which parameter forced +the rebuild: ```dart final quick = await FlutterGemma.getActiveModel(maxTokens: 512); final deep = await FlutterGemma.getActiveModel(maxTokens: 4096); -// deep is quick. Its context window is whatever the FIRST call set. +// deep is a NEW model, and `quick` has been closed — using it now throws +// StateError('Session is closed'). Drop the old handle. ``` -To change `maxTokens` or the backend, close the model and load it again: +So don't hold a handle across a parameter change, and don't vary the arguments +on a hot path: each change costs a full weight reload. To serve several +conversations from one loaded model, keep the arguments identical and open +several sessions instead (below). Closing it yourself first is equivalent: ```dart await quick.close(); diff --git a/website/content/docs/models.md b/website/content/docs/models.md index 0b3fc7d5..d396d8e6 100644 --- a/website/content/docs/models.md +++ b/website/content/docs/models.md @@ -87,7 +87,7 @@ MediaPipe `.task` build. | **SmolVLM2 500M** | Compact vision-language model | ❌ | ❌ | ✅ | Multilingual | 0.36GB | | **LLaVA-OneVision 0.5B** | Compact vision-language model | ❌ | ❌ | ✅ | Multilingual | 0.83GB | | **Phi-4 Mini** | Advanced reasoning and instruction following | ✅ | ❌ | ❌ | Multilingual | 3.9GB | -| **Phi-4 Mini Reasoning** | Step-by-step reasoning | ❌ | ✅ | ❌ | Multilingual | 2.8GB | +| **Phi-4 Mini Reasoning** | Step-by-step reasoning | ❌ | ⚠️ ‡ | ❌ | Multilingual | 2.8GB | | **DeepSeek R1** | High-performance reasoning and code generation | ✅ | ✅ | ❌ | Multilingual | 1.7GB | | **Qwen3 0.6B** | Compact multilingual chat with function calling | ✅ | ✅ | ❌ | Multilingual | 586MB | | **Qwen 2.5** | Strong multilingual chat and instruction following | ✅ | ❌ | ❌ | Multilingual | 0.5-1.6GB | @@ -96,11 +96,17 @@ MediaPipe `.task` build. | **FunctionGemma 270M** | Specialized for function calling on-device | ✅ | ❌ | ❌ | Multilingual | 284MB | | **SmolLM 135M** | Ultra-compact, resource-constrained devices | ❌ | ❌ | ❌ | English | 135MB | | **LFM2.5 230M** | Smallest entry; no HF token needed | ❌ | ❌ | ❌ | Multilingual | 168MB | -| **SmolLM3 3B** | Multilingual small LLM with reasoning mode | ❌ | ✅ | ❌ | Multilingual | 2.0GB | +| **SmolLM3 3B** | Multilingual small LLM with reasoning mode | ❌ | ⚠️ ‡ | ❌ | Multilingual | 2.0GB | | **TranslateGemma 4B** † | Single-shot 55-language translation | ❌ | ❌ | ❌ | 55 languages | 2-4GB | -¹ Gemma 4 **Thinking Mode** works on Android, iOS, and Desktop only — **not on -Web** (the MediaPipe web engine does not support the `extraContext` thinking path). +¹ Gemma 4 **Thinking Mode** needs the native `extraContext` channel: Android, iOS +and Desktop only. Qwen3 and DeepSeek R1 reasoning is split out of the text by +flutter_gemma itself, so it works on every platform including Web. + +‡ **Reasons, but emits no `ThinkingResponse`.** These models run as +`ModelType.general`, which has no reasoning parser — their thinking blocks +arrive inside the answer as ordinary text and are not stripped. See +[Thinking Mode](/docs/thinking-mode). † **TranslateGemma is CPU-only for now.** Google hasn't released a diff --git a/website/content/docs/thinking-mode.md b/website/content/docs/thinking-mode.md index e9e1b8b8..a39cf06f 100644 --- a/website/content/docs/thinking-mode.md +++ b/website/content/docs/thinking-mode.md @@ -10,13 +10,20 @@ answer. ## Supported models -- **Gemma 4** (E2B, E4B) -- **DeepSeek R1** -- **Qwen3 0.6B** — generates thinking by default; tags are stripped when `isThinking: false`. -- **SmolLM3 3B** — multilingual small LLM with a reasoning mode. -- **Phi-4 Mini Reasoning** — Phi-4 Mini tuned for step-by-step reasoning. +- **Gemma 4** (E2B, E4B) — `ModelType.gemma4` +- **DeepSeek R1** — `ModelType.deepSeek` +- **Qwen3 0.6B** — `ModelType.qwen3`; generates thinking by default, tags are stripped when `isThinking: false`. -Enable it with `isThinking: true` on the matching `ModelType`. +Enable it with `isThinking: true` and the matching `ModelType`. + + +The reasoning channel is parsed per `ModelType`, and `ModelType.general` has no +parser at all. Models that reason but run as `general` — **SmolLM3 3B**, +**Phi-4 Mini Reasoning** — emit no `ThinkingResponse`, and their thinking tags +are not stripped either: the raw blocks arrive inside the answer as ordinary +`TextResponse` tokens. Strip them yourself, or don't advertise a thinking UI for +those models. + ## Handling thinking responses @@ -49,13 +56,16 @@ final thinkingMessage = Message.thinking(text: "Let me analyze this problem...") | Android | ✅ Full | | iOS | ✅ Full | | Desktop (macOS/Windows/Linux) | ✅ Full | -| Web | ❌ Not supported | +| Web | ⚠️ Qwen3 / DeepSeek R1 only | -Thinking mode is **not supported on Web yet**. MediaPipe `.task` web has no -`extraContext` hook, and the web `.litertlm` path (`@litert-lm/core`) does not -wire the `extraContext` thinking channel. Thinking mode for Gemma 4 is available -on Android, iOS, and Desktop only. +On web, Qwen3 and DeepSeek R1 reasoning **is** separated out of the token +stream: that split is pure Dart and runs on every platform. Gemma 4's thinking +is different — it needs the native `extraContext` channel. MediaPipe `.task` web +has no such hook and warns that `enableThinking` is ignored; the web `.litertlm` +path does pass `extra_context` to `@litert-lm/core`, but it has never been +verified end to end, so treat Gemma 4 thinking on web as unsupported until it +is. ## Advanced: ModelThinkingFilter diff --git a/website/lib/landing/sections/quick_start.dart b/website/lib/landing/sections/quick_start.dart index d0874632..bd557350 100644 --- a/website/lib/landing/sections/quick_start.dart +++ b/website/lib/landing/sections/quick_start.dart @@ -3,15 +3,16 @@ import 'package:jaspr/jaspr.dart'; import '../../theme/brand.dart'; -const _code = '''FlutterGemma.initialize( +const _code = '''await FlutterGemma.initialize( inferenceEngines: [LiteRtLmEngine(), MediaPipeEngine()], embeddingBackends: [LiteRtEmbeddingBackend()], vectorStore: QdrantVectorStore(), ); -await FlutterGemma.installModel(modelType: ModelType.gemma4) - .fromNetwork('https://.../gemma-4-E2B-it.litertlm') - .install(); +await FlutterGemma.installModel( + modelType: ModelType.gemma4, + fileType: ModelFileType.litertlm, // the declared type picks the engine +).fromNetwork('https://.../gemma-4-E2B-it.litertlm').install(); final model = await FlutterGemma.getActiveModel(maxTokens: 2048); final chat = await model.createChat(); From 2b88ef9e7c81a94698c856f264679540c6ebfa1c Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 16:27:42 +0200 Subject: [PATCH 17/21] =?UTF-8?q?docs(site):=20engine=20pages=20=E2=80=94?= =?UTF-8?q?=20ONNX=20web=20URL,=20web=20tool=20calls,=20Phi,=20LFM2.5,=20g?= =?UTF-8?q?ating?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - onnx.md installed the web model with a bare repo id; the resolver parses a Hugging Face URL and throws ArgumentError on anything else. packages.md described the same thing as "a repo id". - troubleshooting.md said function calling is unavailable on the web .litertlm runtime. It has worked since core 1.7.3 fixed createChat dropping tools, the web arm serializes them for Gemma 4, and a device suite covers it. Its thinking row now distinguishes the Dart-side tag split (every platform) from Gemma 4's native channel (unverified on web). - models.md routed Phi-4 Mini to ModelType.general; ModelType.phi exists and is the one that parses Phi's tool-call markers. LFM2.5 230M was marked as not running on desktop, but the example ships a desktop build verified on macOS, Windows and Linux. - installation.md listed Gemma 4 as gated — the litert-community builds answer 200 anonymously. Its Android GPU snippet asked apps to declare the OpenCL entries the plugin already merges in, and omitted libvndksupport.so, which is the one #324 was about: following the page produced a worse manifest than doing nothing. - desktop.md and DESKTOP_SUPPORT.md pointed at an AI Edge Model Garden URL that 404s. --- packages/flutter_gemma/DESKTOP_SUPPORT.md | 2 +- website/content/docs/desktop.md | 2 +- website/content/docs/installation.md | 18 ++++++++++-------- website/content/docs/models.md | 4 ++-- website/content/docs/onnx.md | 2 +- website/content/docs/packages.md | 6 +++--- website/content/docs/troubleshooting.md | 10 ++++++---- 7 files changed, 24 insertions(+), 20 deletions(-) diff --git a/packages/flutter_gemma/DESKTOP_SUPPORT.md b/packages/flutter_gemma/DESKTOP_SUPPORT.md index d480af3d..5c7490f4 100644 --- a/packages/flutter_gemma/DESKTOP_SUPPORT.md +++ b/packages/flutter_gemma/DESKTOP_SUPPORT.md @@ -60,7 +60,7 @@ loading sequence differs per platform (handled in `litert_lm_client.dart`). > > Desktop accepts only LiteRT-LM `.litertlm` files. MediaPipe `.bin` / `.task` > models used on web won't load on desktop. See -> [AI Edge Model Garden](https://ai.google.dev/edge/litert/models) for compatible models. +> [litert-community on Hugging Face](https://huggingface.co/litert-community) for compatible models. --- diff --git a/website/content/docs/desktop.md b/website/content/docs/desktop.md index 5b89aea6..427511b8 100644 --- a/website/content/docs/desktop.md +++ b/website/content/docs/desktop.md @@ -51,7 +51,7 @@ library loading sequence differs per platform. **Model format (LiteRT-LM engine):** the LiteRT-LM engine on desktop accepts only `.litertlm` files. MediaPipe `.bin` / `.task` models used on web won't load on -desktop. See the [AI Edge Model Garden](https://ai.google.dev/edge/litert/models) +desktop. See [litert-community on Hugging Face](https://huggingface.co/litert-community) for compatible models. (The [ONNX engine](/docs/onnx) uses its own `.onnx` model directories instead.) diff --git a/website/content/docs/installation.md b/website/content/docs/installation.md index 844c9766..39332b72 100644 --- a/website/content/docs/installation.md +++ b/website/content/docs/installation.md @@ -238,14 +238,15 @@ host fails with `Could not find method kotlin()`. Add KGP to the host's root `buildscript`/`plugins {}`. A normal `flutter build` app needs nothing — Flutter's own Gradle plugin carries KGP. -**GPU (any engine):** if you want to run on the GPU, add OpenCL support to the -manifest. Required by both inference engines. CPU-only? Skip this step. Add the -following above `` in `AndroidManifest.xml`: +**GPU (any engine): nothing to add.** `flutter_gemma`'s own manifest declares the +OpenCL namespace entries and the manifest merger folds them into your app. These +are what your merged manifest must contain if you pin or audit it — note +`libvndksupport.so`: without it the OpenCL ICD load is denied on Android 12+, the +engine falls back to WebGPU, and some Mali drivers hard-freeze (#324). ``` - + + ``` @@ -451,10 +452,11 @@ void main() async { ### Which models require authentication? -**Gated (auth required):** Gemma 4, Gemma3n (E2B, E4B), Gemma 3 1B, Gemma 3 270M, +**Gated (auth required):** Gemma3n (E2B, E4B), Gemma 3 1B, Gemma 3 270M, EmbeddingGemma. -**Public (no auth):** DeepSeek, Qwen3, Qwen 2.5, SmolLM, Phi-4, FastVLM. +**Public (no auth):** Gemma 4 (the litert-community builds), DeepSeek, Qwen3, +Qwen 2.5, SmolLM, LFM2.5, Phi-4, FastVLM. To use a gated repo: visit the model page → "Request Access" button. diff --git a/website/content/docs/models.md b/website/content/docs/models.md index d396d8e6..2c497e63 100644 --- a/website/content/docs/models.md +++ b/website/content/docs/models.md @@ -132,8 +132,8 @@ When installing models, specify the correct `ModelType`: | **DeepSeek** | `ModelType.deepSeek` | DeepSeek R1 | | **Qwen 2.5** | `ModelType.qwen` | Qwen 2.5 1.5B, Qwen 2.5 0.5B | | **Qwen 3** | `ModelType.qwen3` | Qwen3 0.6B | +| **Phi-4** | `ModelType.phi` | Phi-4 Mini (parses Phi's own tool-call markers) | | **FunctionGemma** | `ModelType.functionGemma` | FunctionGemma 270M IT | -| **Phi** | `ModelType.general` | Phi-4 Mini | | **General** | `ModelType.general` | FastVLM 0.5B, SmolLM 135M, LFM2.5 230M, SmolLM3 3B, Phi-4 Mini Reasoning, Qwen2-VL 2B, SmolVLM2 500M, LLaVA-OneVision 0.5B | @@ -177,7 +177,7 @@ await FlutterGemma.installModel(modelType: ModelType.general) | [Qwen 2.5 1.5B](https://huggingface.co/litert-community/Qwen2.5-1.5B-Instruct) | 1.6GB | ✅ | ✅ | ❌ | | [Qwen 2.5 0.5B](https://huggingface.co/litert-community/Qwen2.5-0.5B-Instruct) | 0.5GB | ❌ | ✅ | ❌ | | [SmolLM 135M](https://huggingface.co/litert-community/SmolLM-135M-Instruct) | 135MB | ❌ | ✅ | ❌ | -| [LFM2.5 230M](https://huggingface.co/litert-community/LFM2.5-230M) | 168MB | ❌ | ✅ | ❌ | +| [LFM2.5 230M](https://huggingface.co/litert-community/LFM2.5-230M) | 168MB | ✅ | ✅ | ❌ | | [SmolLM3 3B](https://huggingface.co/litert-community/SmolLM3-3B) | 2.0GB | ✅ | ✅ | ❌ | | [Phi-4 Mini](https://huggingface.co/litert-community/Phi-4-mini-instruct) | 3.9GB | ✅ | ✅ | ✅ | | [Phi-4 Mini Reasoning](https://huggingface.co/litert-community/Phi-4-mini-reasoning) | 2.8GB | ✅ | ✅ | ❌ | diff --git a/website/content/docs/onnx.md b/website/content/docs/onnx.md index a581a942..dfd58663 100644 --- a/website/content/docs/onnx.md +++ b/website/content/docs/onnx.md @@ -102,7 +102,7 @@ bytes: await FlutterGemma.installModel( modelType: ModelType.general, fileType: ModelFileType.onnx, -).fromNetwork('onnx-community/Qwen2.5-0.5B-Instruct').install(); +).fromNetwork('https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct').install(); ``` From here the code is identical to any other engine: diff --git a/website/content/docs/packages.md b/website/content/docs/packages.md index 9cd00bd2..2c60916c 100644 --- a/website/content/docs/packages.md +++ b/website/content/docs/packages.md @@ -107,9 +107,9 @@ On native, an ORT-GenAI model installs as a **directory**, not a single file: + tokenizer files. `fromHuggingFace(repo)` downloads the whole bundle (the ONNX resolver picks a CPU execution-provider folder), or ship it yourself as an asset / local directory and install with `fromFile(genai_config.json)`. On **Web** this -doesn't apply — the model is a -Hugging Face repo id (e.g. `onnx-community/Qwen2.5-0.5B-Instruct`), not a -directory, and install is fileless: `ModelFileType.onnx` just marks it active, +doesn't apply — the model is identified by its +Hugging Face repo URL (`https://huggingface.co/onnx-community/Qwen2.5-0.5B-Instruct`), +from which the repo id is derived, not by a directory, and install is fileless: `ModelFileType.onnx` just marks it active, and Transformers.js downloads and caches the repo itself. diff --git a/website/content/docs/troubleshooting.md b/website/content/docs/troubleshooting.md index 3bc2c00c..599a140d 100644 --- a/website/content/docs/troubleshooting.md +++ b/website/content/docs/troubleshooting.md @@ -103,15 +103,17 @@ package (WebGPU + WASM). It is an **early preview** and a subset of the native path. MediaPipe `.task` on web is unaffected and remains fully supported. **Works on web `.litertlm`:** text generation (sync + streaming), multi-turn chat -with history, system instruction, concurrent sessions (serialized), large models -via OPFS streaming, GPU only. +with history, system instruction, **function calling / tool calls** (Gemma 4), +concurrent sessions (serialized), large models via OPFS streaming, GPU only. **Not supported on web `.litertlm` yet (mobile/desktop only):** - ❌ **Vision / image input** — image inputs are dropped with a debug warning. - ❌ **Audio input** — no Audio executor config in the JS API. -- ❌ **Thinking mode** — `extraContext` thinking channel is not wired on web. -- ❌ **Function calling / tool calls** — not available on the web runtime. +- ⚠️ **Thinking mode** — Qwen3 and DeepSeek R1 reasoning is still split out of the + token stream (that part is pure Dart). Gemma 4 thinking needs the native + `extraContext` channel: the web engine passes it to `@litert-lm/core`, but it + has never been verified end to end — treat it as unsupported until it is. - ❌ **LoRA weights** — `loraPath` throws `UnsupportedError`. From ca03f2ecc2eb87b2e106aa06acdb259bb8a70e75 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 16:37:59 +0200 Subject: [PATCH 18/21] =?UTF-8?q?docs(site):=20feature=20pages=20=E2=80=94?= =?UTF-8?q?=20the=20flags,=20the=20loops=20and=20the=20policies=20as=20cod?= =?UTF-8?q?ed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit multimodal: supportImage has to be set on getActiveModel AND createChat — MediaPipe does not inherit it and drops the image silently — and the audio section never showed how to send audio (Message.withAudio, a whole WAV file, the opposite of the speech package's raw PCM). speech: the language section followed a moonshine install, where language: throws, so it now installs Whisper first; a bare getActiveStt() clears the default; audio past the model's window is silently truncated; Qwen3's language is fixed when the synthesizer is created, not chosen at synth time, it is 10 languages plus auto, and the sample no longer asks for English while synthesizing French; the voice loop needs an inference engine the page had told readers not to add. genkit: the pubspec omitted genkit itself while every snippet imports it; .litertlm needs its file type in both installModel and the plugin config; the embedder always uses the retrievalQuery prefix, which is wrong for indexing; toolChoice precedence was stated backwards; fallback fires only on transient failures, not on any failure before the first token. agent: the web footnote said the loop is disabled there — nothing gates on the platform, it is unverified; core-library desugaring is unconditional, not tied to the schedule_notification intent. migration: the qdrant break is 1.3.0, not 2.0, and both the re-index and the RAG snippet now call flush(), without which a qdrant index dies with the process. --- website/content/docs/agent.md | 16 +++++++---- website/content/docs/genkit.md | 43 ++++++++++++++++++++++++------ website/content/docs/migration.md | 8 +++--- website/content/docs/multimodal.md | 22 ++++++++++++++- website/content/docs/speech.md | 42 +++++++++++++++++++++++------ 5 files changed, 106 insertions(+), 25 deletions(-) diff --git a/website/content/docs/agent.md b/website/content/docs/agent.md index 10b977c0..05344de7 100644 --- a/website/content/docs/agent.md +++ b/website/content/docs/agent.md @@ -34,7 +34,7 @@ dependencies: flutter_gemma_litertlm: ^1.6.3 # an inference engine (LiteRtLmEngine) ``` -The agent is **not supported on Web** yet — see the note below. +The agent is **unverified on Web** — nothing disables it, but it has never been driven in a browser. See the note below. ## The four skill mechanisms @@ -52,8 +52,10 @@ execution mechanisms: [WebView2 Runtime](https://developer.microsoft.com/microsoft-edge/webview2/) (pre-installed on Windows 11). ² Linux has no embeddable webview, so JS skills return an `ErrorResult`; text / native-intent / MCP skills work on Linux. -³ The agent is not supported on Web yet — the browser LLM runtimes don't reliably -emit tool calls, so the agent loop is disabled there (see the note below). +³ Unverified on Web rather than disabled: nothing in the package gates on the +platform, and the web `.litertlm` path does emit well-formed tool calls. What is +missing is a run of the agent itself (see the note below). Native-intent skills +are stubbed on web by design. JS skills run in a headless, sandboxed webview. To grant a secure context (so skills using `crypto.subtle` and other secure-context Web APIs work), the package @@ -204,8 +206,12 @@ Most skills need no platform setup. For the platform-specific bits: NSCalendarsUsageDescription Create calendar events from the agent. ``` -- **Android** — `schedule_notification` requires core-library desugaring in - `android/app/build.gradle(.kts)`: +- **Android** — `flutter_gemma_agent` depends on `flutter_local_notifications`, + which requires core-library desugaring in `android/app/build.gradle(.kts)`. + This is unconditional: an app that never uses the `schedule_notification` + intent still fails to build without it. Kotlin DSL below; in Groovy the lines + are `coreLibraryDesugaringEnabled true` and + `coreLibraryDesugaring 'com.android.tools:desugar_jdk_libs:2.1.4'`. ``` android { compileOptions { isCoreLibraryDesugaringEnabled = true } } dependencies { coreLibraryDesugaring("com.android.tools:desugar_jdk_libs:2.1.4") } diff --git a/website/content/docs/genkit.md b/website/content/docs/genkit.md index b83f878e..8aa69e0f 100644 --- a/website/content/docs/genkit.md +++ b/website/content/docs/genkit.md @@ -20,6 +20,7 @@ with the on-device model exactly as it would with any cloud provider. ``` dependencies: + genkit: ^0.16.0 # the framework itself — every snippet below uses it genkit_flutter_gemma: ^0.6.1 flutter_gemma: ^1.8.2 # Add the inference engine(s) you need: @@ -52,6 +53,10 @@ await FlutterGemma.installModel(modelType: ModelType.gemmaIt) .fromAsset('assets/gemma-3-1b-it-int4.task') .install(); +// For a .litertlm model declare the type in BOTH places — installModel( +// fileType: ModelFileType.litertlm) and FlutterGemmaModelConfig(fileType: ...). +// Both default to ModelFileType.task, which routes the model to MediaPipe. + // 3. Create a Genkit instance with the plugin. final ai = Genkit(plugins: [ GenkitFlutterGemmaPlugin( @@ -102,6 +107,16 @@ final embeddings = await ai.embed( ); ``` + +The Genkit embedder always embeds with flutter_gemma's default +`TaskType.retrievalQuery` prefix — `FlutterGemmaEmbedConfig` has no `taskType` +option. For RAG **indexing**, where documents must be embedded with +`TaskType.retrievalDocument`, call +`FlutterGemma.getActiveEmbedder().generateEmbeddings(..., taskType: ...)` +directly. Mixing the two prefixes is the cross-prefix drift that #264 fixed at +the core level. See [Embeddings & RAG](/docs/embeddings-and-rag). + + ### Configuration options Pass `FlutterGemmaModelOptions` to tune inference: @@ -124,8 +139,11 @@ final response = await ai.generate( ); ``` -`toolChoice` maps to Genkit 0.15's native top-level `toolChoice`: `'auto'` -lets the model decide, `'required'` forces a tool call, `'none'` forbids one. +Prefer Genkit's standard top-level parameter — `ai.generate(toolChoice: 'none')` +— which takes **precedence** over the `toolChoice` config field above (kept as a +legacy fallback). Either way: `'auto'` lets the model decide, `'required'` forces +a tool call, `'none'` forbids one. An unrecognized value throws +`INVALID_ARGUMENT` rather than quietly falling back to `'auto'`. The plugin does **not** manage model installation. Call @@ -298,12 +316,21 @@ hybridModel( ); ``` -### Streaming and fallback - -Fallback during streaming happens **only before the first token**. If a branch -fails before emitting any output, the next branch is tried transparently. Once -the first token has streamed, a later failure propagates as an error — a -partially delivered response cannot be silently re-routed. +### Error policy and fallback + +Fallback is error-driven: the strategy picks an order, and the next branch is +tried only when the current one **throws**, and only on a transient failure — +any non-`GenkitException` error (network, timeout, OOM), or a `GenkitException` +with `UNAVAILABLE`, `DEADLINE_EXCEEDED`, `RESOURCE_EXHAUSTED` or `INTERNAL`. +Permanent errors — `INVALID_ARGUMENT`, `PERMISSION_DENIED`, `UNAUTHENTICATED`, +`FAILED_PRECONDITION`, `NOT_FOUND` — propagate immediately, since they would +fail the same way on every branch. A `GenkitException` thrown without an +explicit status defaults to `INTERNAL`, so it *is* retried. + +During **streaming** the same policy applies plus a hard cut-off: fallback is +possible only before the first token. Once a branch has emitted a chunk, any +later failure propagates — a partially delivered response cannot be silently +re-routed. ### Escalate on a quality check with `cascadeModel` diff --git a/website/content/docs/migration.md b/website/content/docs/migration.md index ce46631a..76a6958e 100644 --- a/website/content/docs/migration.md +++ b/website/content/docs/migration.md @@ -170,12 +170,12 @@ data change, not an API change: your `addDocument` / `searchSimilar` calls are unchanged, but the documents already on the device are not. -An upgraded app finds no documents where its corpus used to be. 2.0 refuses +An upgraded app finds no documents where its corpus used to be. 1.3.0 refuses loudly rather than starting empty — `initialize()` throws a `QdrantLegacyStoreException` naming the old store — so this shows up the first time the store opens, not as silently unanswered questions later. -Remove the old store's files once, then re-index. 2.0 will not do it for you: +Remove the old store's files once, then re-index. 1.3.0 will not do it for you: it never deletes data it cannot read, and the three entries a 1.x shard owns (`edge_config.json`, `wal/`, `segments/`) may sit beside files of your own. @@ -190,7 +190,8 @@ try { // file APIs you already use for `path`, then initialize() again. rethrow; } -// ...then re-add your documents. +// ...then re-add your documents, and flush() — on qdrant, points live in the +// shard's in-RAM segment until then, so a background kill loses the re-index. ``` @@ -272,6 +273,7 @@ await FlutterGemma.installEmbedder() final dir = await getApplicationDocumentsDirectory(); // native; on web pass a bare name await FlutterGemma.rag.initialize('${dir.path}/rag_store'); await FlutterGemma.rag.addDocument(/* ... */); +await FlutterGemma.rag.flush(); // qdrant: required, or the index dies with the process final hits = await FlutterGemma.rag.searchSimilar(query: query, topK: 5); ``` diff --git a/website/content/docs/multimodal.md b/website/content/docs/multimodal.md index e027c178..8b76e08e 100644 --- a/website/content/docs/multimodal.md +++ b/website/content/docs/multimodal.md @@ -43,6 +43,12 @@ allow it — pass `preferredVisionBackend: PreferredBackend.gpu` to ### Sending an image +The flag goes in **two** places: on `getActiveModel` it loads the vision encoder, +on `createChat` the session declares it will send images. `.litertlm` inherits the +model's flag, but MediaPipe `.task` does not — a chat created without it **drops +the image and answers about the text alone**, with no error and no log in a +release build. Set it in both places either way. + ```dart // Text + Image final message = Message.withImages( @@ -54,7 +60,7 @@ final message = Message.withImages( // Image only final imageMessage = Message.imagesOnly(imageBytes: [imageBytes], isUser: true); -final chat = await model.createChat(); +final chat = await model.createChat(supportImage: true); await chat.addQueryChunk(message); final response = await chat.generateChatResponse(); @@ -95,6 +101,20 @@ final model = await FlutterGemma.getActiveModel( ); ``` +Sending audio is `Message.withAudio` — and `audioBytes` is a whole WAV file, +16 kHz mono, header included (the opposite of `flutter_gemma_speech`, whose +`transcribe` takes raw PCM): + +```dart +final chat = await model.createChat(supportAudio: true); +await chat.addQueryChunk(Message.withAudio( + text: 'What is said in this recording?', + audioBytes: wavBytes, + isUser: true, +)); +final response = await chat.generateChatResponse(); +``` + Audio input only works with `.litertlm` models that include the audio adapter. MediaPipe `.task` models on web do not support audio. On macOS, Gemma 3n audio diff --git a/website/content/docs/speech.md b/website/content/docs/speech.md index b079ea0e..c2676a58 100644 --- a/website/content/docs/speech.md +++ b/website/content/docs/speech.md @@ -81,6 +81,17 @@ await recognizer.close(); Whisper's shipped checkpoints are multilingual, and the output language is one token in the decoder's seed prompt. Set a default, or override a single call: +The install above is moonshine, which has no language token — `language:` throws +`ArgumentError` on it. Install Whisper first: + +```dart +await FlutterGemma.installStt() + .modelFromNetwork('https://huggingface.co/litert-community/whisper-tiny/resolve/main/whisper_tiny_30s_f32.tflite') + .tokenizerFromNetwork('https://huggingface.co/openai/whisper-tiny/resolve/main/tokenizer.json') + .ofType(SttModelType.whisper) + .install(); +``` + ```dart final recognizer = await FlutterGemma.getActiveStt(language: 'de'); final german = await recognizer.transcribe(germanPcm); @@ -91,7 +102,8 @@ final french = await recognizer.transcribe(frenchPcm, language: 'fr'); `getActiveStt` returns a process-wide singleton, and calling it again with a new `language` retargets that recognizer rather than rebuilding it — so you never -need to `close()` just to change language. +need to `close()` just to change language. Calling it with **no** `language` +clears the default you set earlier, back to the checkpoint's own `'en'`. The code is Whisper's own, without the delimiters (`'en'`, `'de'`, `'uk'`), and it defaults to `'en'`. It changes what the model WRITES, not what it hears: @@ -112,6 +124,11 @@ recorder for 16 kHz / mono / 16-bit PCM. Starting from a WAV file, locate its or padding chunks, and on iOS a multi-kilobyte `FLLR` block, so a fixed skip feeds header bytes in as audio. +Each model also has a fixed window — 5 s for moonshine and Parakeet, 30 s for +Whisper. Shorter audio is zero-padded; **longer audio is silently truncated**, so +a 40-second clip on Whisper returns the first 30 seconds with no error. Split +long recordings yourself. + ## Text-to-speech (Matcha) TTS is opt-in the same way: pass `LiteRtTtsBackend()` to `initialize()`, then @@ -143,10 +160,13 @@ low-float-bit divergence on x86_64. ### Qwen3-TTS (multilingual) A second TTS family is selectable via `TtsModelType.qwen3` — an autoregressive -codec-LM (`litert-community/Qwen3-TTS-12Hz-0.6B-Base`, ~1.9 GB) that speaks 11 -languages, chosen at synth time with the new `language:` param. Same -install/synthesize API as Matcha, but heavier: CPU-only, RTF≈3 (~3 s of compute -per 1 s of audio), and needs a 6 GB-RAM-class device. +codec-LM (`litert-community/Qwen3-TTS-12Hz-0.6B-Base`, ~1.9 GB) that speaks 10 +languages plus `'auto'` detection, picked with `getActiveTts(language:)`. Unlike +STT, the language is fixed when the synthesizer is created — `synthesize()` takes +no language — and `getActiveTts` returns a singleton, so asking it for a different +language throws `StateError` until you `close()` the current one. Same +install/synthesize API as Matcha, but heavier: CPU-only, 24 kHz output, RTF≈3 +(~3 s of compute per 1 s of audio), and needs a 6 GB-RAM-class device. ```dart await FlutterGemma.installTts() @@ -154,10 +174,11 @@ await FlutterGemma.installTts() .ofType(TtsModelType.qwen3) .install(); -// language is one of qwen3SupportedLanguages (11) or 'auto'; ignored by Matcha. -final synth = await FlutterGemma.getActiveTts(language: 'english'); +// language is a full lowercase name from qwen3SupportedLanguages ('english', +// 'german', … 10 of them) or 'auto' — not an ISO code, and ignored by Matcha. +final synth = await FlutterGemma.getActiveTts(language: 'french'); final pcm = await synth.synthesize('Bonjour le monde.'); // Uint8List, 16-bit PCM -await synth.close(); +await synth.close(); // close before asking for another language ``` ### Inflect-Nano-v2 (fast) @@ -186,6 +207,11 @@ one with `VoiceSession.fromChat`, which wraps an `InferenceChat`. A tools-free chat is the plain speech-to-speech loop; pass `onToolCall` to run function calls inside a spoken turn (see [Tool calling](#tool-calling-in-the-voice-loop) below). +The loop needs an LLM as well as the two speech models: add +`flutter_gemma_litertlm`, register `inferenceEngines: [LiteRtLmEngine()]` in +`initialize`, and install a `.litertlm` model (see [LiteRT-LM](/docs/litertlm)) — +otherwise `getActiveModel` throws `StateError('No active inference model set')`. + ```dart final recognizer = await FlutterGemma.getActiveStt(); final synthesizer = await FlutterGemma.getActiveTts(); From a3416f8c4a8c87b62c4a5c74b2be1c78b86e8826 Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 16:57:51 +0200 Subject: [PATCH 19/21] docs(codelabs): the API floor .litertlm actually needs, and pins that resolve MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit minSdk 30 in every codelab app that loads libLiteRtLm.so, and in the two codelabs that walk a reader through the Android setup. The floor is not a build error: Flutter's template 24 merges fine, the APK installs, and the failure arrives as a dlopen error at the first model load, because the library calls pthread_cond_clockwait/sem_clockwait — API-30 Bionic. The Inference Engines codelab told the reader to set 26 for ML Kit GenAI, which would have lowered the floor its own starter arrives with. Pins moved to what pub.dev publishes (flutter_gemma 1.8.1, flutter_gemma_litertlm 1.6.3); the hybrid starter's commented-out lines, the ones a reader uncomments, were three cycles behind at 0.15.1/0.3.1. SDK floors raised to the ones the packages declare (Dart 3.12, Flutter 3.44) — the hybrid apps still asked for 3.10/3.27. Web availability: with the Prompt API flag on and the hardware floor unmet, Chrome answers 'unavailable' with no reason, which _mapStatus buckets as unavailableOther, not unavailableDeviceUnsupported — that one means the API is absent entirely. --- .../complete/android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../complete/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_02_download/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../step_03_chat/android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_03_chat/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_04_streaming/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../complete/android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../complete/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../step_00_starter/pubspec.yaml | 17 ++++++++++------- .../step_01_cloud_ai/pubspec.yaml | 4 ++-- .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_02_local_ai/pubspec.yaml | 8 ++++---- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_03_hybrid/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_04_smart_routing/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_05_embeddings/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../complete/android/app/build.gradle.kts | 8 +++++--- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../complete/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_01_starter/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 8 +++++--- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_02_two_engines/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 8 +++++--- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_03_pick_at_startup/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../complete/android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../complete/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_01_starter/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_02_vision/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../android/app/build.gradle.kts | 5 ++++- .../Flutter/GeneratedPluginRegistrant.swift | 2 ++ .../step_03_audio/pubspec.yaml | 4 ++-- .../flutter/generated_plugin_registrant.cc | 3 +++ .../windows/flutter/generated_plugins.cmake | 1 + .../getting-started-flutter-gemma/index.md | 19 +++++++++++++++++-- .../hybrid-ai-flutter-genkit/index.md | 16 ++++++++++++++-- .../inference-engines-flutter-gemma/index.md | 18 ++++++++++++------ .../multimodal-flutter-gemma/index.md | 5 +++-- website/content/docs/genkit.md | 2 +- 92 files changed, 268 insertions(+), 81 deletions(-) diff --git a/codelabs/getting-started-flutter-gemma/complete/android/app/build.gradle.kts b/codelabs/getting-started-flutter-gemma/complete/android/app/build.gradle.kts index 4a724f64..eca94cce 100644 --- a/codelabs/getting-started-flutter-gemma/complete/android/app/build.gradle.kts +++ b/codelabs/getting-started-flutter-gemma/complete/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.quickstart" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/getting-started-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/getting-started-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/getting-started-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/getting-started-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/getting-started-flutter-gemma/complete/pubspec.yaml b/codelabs/getting-started-flutter-gemma/complete/pubspec.yaml index 324dd18b..44218589 100644 --- a/codelabs/getting-started-flutter-gemma/complete/pubspec.yaml +++ b/codelabs/getting-started-flutter-gemma/complete/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc b/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugins.cmake b/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugins.cmake +++ b/codelabs/getting-started-flutter-gemma/complete/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/getting-started-flutter-gemma/step_02_download/android/app/build.gradle.kts b/codelabs/getting-started-flutter-gemma/step_02_download/android/app/build.gradle.kts index 4a724f64..eca94cce 100644 --- a/codelabs/getting-started-flutter-gemma/step_02_download/android/app/build.gradle.kts +++ b/codelabs/getting-started-flutter-gemma/step_02_download/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.quickstart" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/getting-started-flutter-gemma/step_02_download/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/getting-started-flutter-gemma/step_02_download/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/getting-started-flutter-gemma/step_02_download/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/getting-started-flutter-gemma/step_02_download/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/getting-started-flutter-gemma/step_02_download/pubspec.yaml b/codelabs/getting-started-flutter-gemma/step_02_download/pubspec.yaml index 6e7eae03..90014568 100644 --- a/codelabs/getting-started-flutter-gemma/step_02_download/pubspec.yaml +++ b/codelabs/getting-started-flutter-gemma/step_02_download/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugin_registrant.cc b/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugins.cmake b/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugins.cmake +++ b/codelabs/getting-started-flutter-gemma/step_02_download/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/getting-started-flutter-gemma/step_03_chat/android/app/build.gradle.kts b/codelabs/getting-started-flutter-gemma/step_03_chat/android/app/build.gradle.kts index 4a724f64..eca94cce 100644 --- a/codelabs/getting-started-flutter-gemma/step_03_chat/android/app/build.gradle.kts +++ b/codelabs/getting-started-flutter-gemma/step_03_chat/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.quickstart" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/getting-started-flutter-gemma/step_03_chat/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/getting-started-flutter-gemma/step_03_chat/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/getting-started-flutter-gemma/step_03_chat/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/getting-started-flutter-gemma/step_03_chat/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/getting-started-flutter-gemma/step_03_chat/pubspec.yaml b/codelabs/getting-started-flutter-gemma/step_03_chat/pubspec.yaml index 6e7eae03..90014568 100644 --- a/codelabs/getting-started-flutter-gemma/step_03_chat/pubspec.yaml +++ b/codelabs/getting-started-flutter-gemma/step_03_chat/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugin_registrant.cc b/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugins.cmake b/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugins.cmake +++ b/codelabs/getting-started-flutter-gemma/step_03_chat/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/getting-started-flutter-gemma/step_04_streaming/android/app/build.gradle.kts b/codelabs/getting-started-flutter-gemma/step_04_streaming/android/app/build.gradle.kts index 4a724f64..eca94cce 100644 --- a/codelabs/getting-started-flutter-gemma/step_04_streaming/android/app/build.gradle.kts +++ b/codelabs/getting-started-flutter-gemma/step_04_streaming/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.quickstart" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/getting-started-flutter-gemma/step_04_streaming/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/getting-started-flutter-gemma/step_04_streaming/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/getting-started-flutter-gemma/step_04_streaming/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/getting-started-flutter-gemma/step_04_streaming/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/getting-started-flutter-gemma/step_04_streaming/pubspec.yaml b/codelabs/getting-started-flutter-gemma/step_04_streaming/pubspec.yaml index 6e7eae03..90014568 100644 --- a/codelabs/getting-started-flutter-gemma/step_04_streaming/pubspec.yaml +++ b/codelabs/getting-started-flutter-gemma/step_04_streaming/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugin_registrant.cc b/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugins.cmake b/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugins.cmake +++ b/codelabs/getting-started-flutter-gemma/step_04_streaming/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/hybrid-ai-flutter-genkit/complete/android/app/build.gradle.kts b/codelabs/hybrid-ai-flutter-genkit/complete/android/app/build.gradle.kts index d6b2a906..f6f7c520 100644 --- a/codelabs/hybrid-ai-flutter-genkit/complete/android/app/build.gradle.kts +++ b/codelabs/hybrid-ai-flutter-genkit/complete/android/app/build.gradle.kts @@ -23,7 +23,10 @@ android { applicationId = "dev.flutterberlin.workshop_flutter_gemma_hybrid_ai" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion versionCode = flutter.versionCode versionName = flutter.versionName diff --git a/codelabs/hybrid-ai-flutter-genkit/complete/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/hybrid-ai-flutter-genkit/complete/macos/Flutter/GeneratedPluginRegistrant.swift index 1a668cbd..f50d0aeb 100644 --- a/codelabs/hybrid-ai-flutter-genkit/complete/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/hybrid-ai-flutter-genkit/complete/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/hybrid-ai-flutter-genkit/complete/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/complete/pubspec.yaml index db9aa606..92f51fa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/complete/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/complete/pubspec.yaml @@ -18,10 +18,10 @@ dependencies: # On-device AI via genkit_flutter_gemma genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # LiteRT-LM engine (.litertlm inference) + LiteRT embedding backend — # flutter_gemma 1.x registers no engines by default; opt in here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 # Hybrid on-device ↔ cloud routing genkit_hybrid: ^0.2.1 diff --git a/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugin_registrant.cc b/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugin_registrant.cc index bdc9b64b..20486aa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugin_registrant.cc @@ -6,10 +6,13 @@ #include "generated_plugin_registrant.h" +#include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugins.cmake b/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugins.cmake index 812d3d4c..265de1b5 100644 --- a/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugins.cmake +++ b/codelabs/hybrid-ai-flutter-genkit/complete/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma ) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_00_starter/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_00_starter/pubspec.yaml index c9d25393..6b6ec4c7 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_00_starter/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_00_starter/pubspec.yaml @@ -4,8 +4,8 @@ publish_to: 'none' version: 1.0.0+1 environment: - sdk: '>=3.10.0 <4.0.0' - flutter: '>=3.27.0' + sdk: '>=3.12.0 <4.0.0' + flutter: '>=3.44.0' dependencies: flutter: @@ -13,12 +13,15 @@ dependencies: cupertino_icons: ^1.0.8 # Step 2: Uncomment to add Cloud AI - # genkit: ^0.13.0 - # genkit_google_genai: ^0.2.7 + # genkit: ^0.16.0 + # genkit_google_genai: ^0.3.1 - # Step 3: Uncomment to add On-device AI - # genkit_flutter_gemma: ^0.3.1 - # flutter_gemma: ^0.15.1 + # Step 3: Uncomment to add On-device AI (LiteRT-LM engine) + # genkit_flutter_gemma: ^0.6.0 + # flutter_gemma: ^1.8.1 + # flutter_gemma 1.x registers no engine by default — opt into LiteRT-LM + # (.litertlm inference) here. + # flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/hybrid-ai-flutter-genkit/step_01_cloud_ai/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_01_cloud_ai/pubspec.yaml index 3df9cd19..197903c6 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_01_cloud_ai/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_01_cloud_ai/pubspec.yaml @@ -4,8 +4,8 @@ publish_to: 'none' version: 1.0.0+1 environment: - sdk: '>=3.10.0 <4.0.0' - flutter: '>=3.27.0' + sdk: '>=3.12.0 <4.0.0' + flutter: '>=3.44.0' dependencies: flutter: diff --git a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/android/app/build.gradle.kts b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/android/app/build.gradle.kts index c456a54d..3884559f 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/android/app/build.gradle.kts +++ b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/android/app/build.gradle.kts @@ -23,7 +23,10 @@ android { applicationId = "dev.flutterberlin.workshop_flutter_gemma_hybrid_ai" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion versionCode = flutter.versionCode versionName = flutter.versionName diff --git a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/pubspec.yaml index d2b9dc22..581664cc 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/pubspec.yaml @@ -4,8 +4,8 @@ publish_to: 'none' version: 1.0.0+1 environment: - sdk: '>=3.10.0 <4.0.0' - flutter: '>=3.27.0' + sdk: '>=3.12.0 <4.0.0' + flutter: '>=3.44.0' dependencies: flutter: @@ -18,10 +18,10 @@ dependencies: # Step 3: On-device AI via genkit_flutter_gemma genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # flutter_gemma 1.x registers no engine by default — opt into LiteRT-LM # (.litertlm inference) here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugin_registrant.cc b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugins.cmake b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugins.cmake +++ b/codelabs/hybrid-ai-flutter-genkit/step_02_local_ai/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/android/app/build.gradle.kts b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/android/app/build.gradle.kts index d6b2a906..f6f7c520 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/android/app/build.gradle.kts +++ b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/android/app/build.gradle.kts @@ -23,7 +23,10 @@ android { applicationId = "dev.flutterberlin.workshop_flutter_gemma_hybrid_ai" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion versionCode = flutter.versionCode versionName = flutter.versionName diff --git a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/pubspec.yaml index 507c582a..5a334fd4 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/pubspec.yaml @@ -18,10 +18,10 @@ dependencies: # On-device AI via genkit_flutter_gemma genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # LiteRT-LM engine (.litertlm inference) + LiteRT embedding backend — # flutter_gemma 1.x registers no engines by default; opt in here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 # Hybrid on-device ↔ cloud routing genkit_hybrid: ^0.2.1 diff --git a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugin_registrant.cc b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugins.cmake b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugins.cmake +++ b/codelabs/hybrid-ai-flutter-genkit/step_03_hybrid/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/android/app/build.gradle.kts b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/android/app/build.gradle.kts index d6b2a906..f6f7c520 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/android/app/build.gradle.kts +++ b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/android/app/build.gradle.kts @@ -23,7 +23,10 @@ android { applicationId = "dev.flutterberlin.workshop_flutter_gemma_hybrid_ai" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion versionCode = flutter.versionCode versionName = flutter.versionName diff --git a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/macos/Flutter/GeneratedPluginRegistrant.swift index 1a668cbd..f50d0aeb 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/pubspec.yaml index db9aa606..92f51fa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/pubspec.yaml @@ -18,10 +18,10 @@ dependencies: # On-device AI via genkit_flutter_gemma genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # LiteRT-LM engine (.litertlm inference) + LiteRT embedding backend — # flutter_gemma 1.x registers no engines by default; opt in here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 # Hybrid on-device ↔ cloud routing genkit_hybrid: ^0.2.1 diff --git a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugin_registrant.cc b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugin_registrant.cc index bdc9b64b..20486aa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugin_registrant.cc @@ -6,10 +6,13 @@ #include "generated_plugin_registrant.h" +#include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugins.cmake b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugins.cmake index 812d3d4c..265de1b5 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugins.cmake +++ b/codelabs/hybrid-ai-flutter-genkit/step_04_smart_routing/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma ) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/android/app/build.gradle.kts b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/android/app/build.gradle.kts index d6b2a906..f6f7c520 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/android/app/build.gradle.kts +++ b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/android/app/build.gradle.kts @@ -23,7 +23,10 @@ android { applicationId = "dev.flutterberlin.workshop_flutter_gemma_hybrid_ai" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion versionCode = flutter.versionCode versionName = flutter.versionName diff --git a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/macos/Flutter/GeneratedPluginRegistrant.swift index 1a668cbd..f50d0aeb 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/pubspec.yaml b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/pubspec.yaml index db9aa606..92f51fa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/pubspec.yaml +++ b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/pubspec.yaml @@ -18,10 +18,10 @@ dependencies: # On-device AI via genkit_flutter_gemma genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # LiteRT-LM engine (.litertlm inference) + LiteRT embedding backend — # flutter_gemma 1.x registers no engines by default; opt in here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 # Hybrid on-device ↔ cloud routing genkit_hybrid: ^0.2.1 diff --git a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugin_registrant.cc b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugin_registrant.cc index bdc9b64b..20486aa2 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugin_registrant.cc @@ -6,10 +6,13 @@ #include "generated_plugin_registrant.h" +#include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugins.cmake b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugins.cmake index 812d3d4c..265de1b5 100644 --- a/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugins.cmake +++ b/codelabs/hybrid-ai-flutter-genkit/step_05_embeddings/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma ) diff --git a/codelabs/inference-engines-flutter-gemma/complete/android/app/build.gradle.kts b/codelabs/inference-engines-flutter-gemma/complete/android/app/build.gradle.kts index e815b07f..caa0c8c7 100644 --- a/codelabs/inference-engines-flutter-gemma/complete/android/app/build.gradle.kts +++ b/codelabs/inference-engines-flutter-gemma/complete/android/app/build.gradle.kts @@ -19,9 +19,11 @@ android { applicationId = "dev.fluttergemma.engines" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26; - // the manifest merger rejects an app below it. - minSdk = 26 + // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26 and + // the manifest merger rejects an app below it; libLiteRtLm.so needs API 30+ + // Bionic (pthread_cond_clockwait, sem_clockwait) on top of that, so 30 is the + // floor for an app that registers both engines. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/inference-engines-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/inference-engines-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift index cfa44f15..b3bb4f9f 100644 --- a/codelabs/inference-engines-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/inference-engines-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import flutter_gemma_builtin_ai import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) FlutterGemmaBuiltInAiPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaBuiltInAiPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/inference-engines-flutter-gemma/complete/pubspec.yaml b/codelabs/inference-engines-flutter-gemma/complete/pubspec.yaml index 8d9aa613..6ed0517c 100644 --- a/codelabs/inference-engines-flutter-gemma/complete/pubspec.yaml +++ b/codelabs/inference-engines-flutter-gemma/complete/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # The OS built-in engine: Gemini Nano (Android, Chrome), Apple # Foundation Models (iOS, macOS). No file to download — the OS or the # browser owns the weights. Windows and Linux have no built-in arm. diff --git a/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc b/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugins.cmake b/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugins.cmake +++ b/codelabs/inference-engines-flutter-gemma/complete/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/inference-engines-flutter-gemma/step_01_starter/android/app/build.gradle.kts b/codelabs/inference-engines-flutter-gemma/step_01_starter/android/app/build.gradle.kts index effb2060..9477909e 100644 --- a/codelabs/inference-engines-flutter-gemma/step_01_starter/android/app/build.gradle.kts +++ b/codelabs/inference-engines-flutter-gemma/step_01_starter/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.engines" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/inference-engines-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/inference-engines-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/inference-engines-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/inference-engines-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/inference-engines-flutter-gemma/step_01_starter/pubspec.yaml b/codelabs/inference-engines-flutter-gemma/step_01_starter/pubspec.yaml index 324dd18b..44218589 100644 --- a/codelabs/inference-engines-flutter-gemma/step_01_starter/pubspec.yaml +++ b/codelabs/inference-engines-flutter-gemma/step_01_starter/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc b/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake b/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake +++ b/codelabs/inference-engines-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/android/app/build.gradle.kts b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/android/app/build.gradle.kts index e815b07f..caa0c8c7 100644 --- a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/android/app/build.gradle.kts +++ b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/android/app/build.gradle.kts @@ -19,9 +19,11 @@ android { applicationId = "dev.fluttergemma.engines" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26; - // the manifest merger rejects an app below it. - minSdk = 26 + // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26 and + // the manifest merger rejects an app below it; libLiteRtLm.so needs API 30+ + // Bionic (pthread_cond_clockwait, sem_clockwait) on top of that, so 30 is the + // floor for an app that registers both engines. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/macos/Flutter/GeneratedPluginRegistrant.swift index cfa44f15..b3bb4f9f 100644 --- a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import flutter_gemma_builtin_ai import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) FlutterGemmaBuiltInAiPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaBuiltInAiPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/pubspec.yaml b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/pubspec.yaml index 8d9aa613..6ed0517c 100644 --- a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/pubspec.yaml +++ b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # The OS built-in engine: Gemini Nano (Android, Chrome), Apple # Foundation Models (iOS, macOS). No file to download — the OS or the # browser owns the weights. Windows and Linux have no built-in arm. diff --git a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugin_registrant.cc b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugins.cmake b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugins.cmake +++ b/codelabs/inference-engines-flutter-gemma/step_02_two_engines/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/android/app/build.gradle.kts b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/android/app/build.gradle.kts index e815b07f..caa0c8c7 100644 --- a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/android/app/build.gradle.kts +++ b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/android/app/build.gradle.kts @@ -19,9 +19,11 @@ android { applicationId = "dev.fluttergemma.engines" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26; - // the manifest merger rejects an app below it. - minSdk = 26 + // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26 and + // the manifest merger rejects an app below it; libLiteRtLm.so needs API 30+ + // Bionic (pthread_cond_clockwait, sem_clockwait) on top of that, so 30 is the + // floor for an app that registers both engines. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/macos/Flutter/GeneratedPluginRegistrant.swift index cfa44f15..b3bb4f9f 100644 --- a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import flutter_gemma_builtin_ai import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) FlutterGemmaBuiltInAiPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaBuiltInAiPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/pubspec.yaml b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/pubspec.yaml index 8d9aa613..6ed0517c 100644 --- a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/pubspec.yaml +++ b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # The OS built-in engine: Gemini Nano (Android, Chrome), Apple # Foundation Models (iOS, macOS). No file to download — the OS or the # browser owns the weights. Windows and Linux have no built-in arm. diff --git a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugin_registrant.cc b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugins.cmake b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugins.cmake +++ b/codelabs/inference-engines-flutter-gemma/step_03_pick_at_startup/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/multimodal-flutter-gemma/complete/android/app/build.gradle.kts b/codelabs/multimodal-flutter-gemma/complete/android/app/build.gradle.kts index 43915ad4..ad7acaad 100644 --- a/codelabs/multimodal-flutter-gemma/complete/android/app/build.gradle.kts +++ b/codelabs/multimodal-flutter-gemma/complete/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.multimodal" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/multimodal-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/multimodal-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift index 5172c542..c68ce0d5 100644 --- a/codelabs/multimodal-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/multimodal-flutter-gemma/complete/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,6 +5,7 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler @@ -12,6 +13,7 @@ import record_macos import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/multimodal-flutter-gemma/complete/pubspec.yaml b/codelabs/multimodal-flutter-gemma/complete/pubspec.yaml index 5da20b5a..af075fe9 100644 --- a/codelabs/multimodal-flutter-gemma/complete/pubspec.yaml +++ b/codelabs/multimodal-flutter-gemma/complete/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # Picks a photo on all six platforms: a gallery on Android and iOS, a file # dialog (via file_selector) on macOS, Windows, Linux and the web. diff --git a/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc b/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc index e1be5753..78c359b5 100644 --- a/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugin_registrant.cc @@ -6,11 +6,14 @@ #include "generated_plugin_registrant.h" +#include #include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugins.cmake b/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugins.cmake index 9d7135d2..a12621d0 100644 --- a/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugins.cmake +++ b/codelabs/multimodal-flutter-gemma/complete/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma record_windows diff --git a/codelabs/multimodal-flutter-gemma/step_01_starter/android/app/build.gradle.kts b/codelabs/multimodal-flutter-gemma/step_01_starter/android/app/build.gradle.kts index 43915ad4..ad7acaad 100644 --- a/codelabs/multimodal-flutter-gemma/step_01_starter/android/app/build.gradle.kts +++ b/codelabs/multimodal-flutter-gemma/step_01_starter/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.multimodal" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/multimodal-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/multimodal-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift index 0a85e4cb..0be86a9b 100644 --- a/codelabs/multimodal-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/multimodal-flutter-gemma/step_01_starter/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,11 +5,13 @@ import FlutterMacOS import Foundation +import connectivity_plus import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) SharedPreferencesPlugin.register(with: registry.registrar(forPlugin: "SharedPreferencesPlugin")) diff --git a/codelabs/multimodal-flutter-gemma/step_01_starter/pubspec.yaml b/codelabs/multimodal-flutter-gemma/step_01_starter/pubspec.yaml index 324dd18b..44218589 100644 --- a/codelabs/multimodal-flutter-gemma/step_01_starter/pubspec.yaml +++ b/codelabs/multimodal-flutter-gemma/step_01_starter/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 dev_dependencies: flutter_test: diff --git a/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc b/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc index 0d9da181..38c2332a 100644 --- a/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugin_registrant.cc @@ -6,9 +6,12 @@ #include "generated_plugin_registrant.h" +#include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FlutterGemmaPluginRegisterWithRegistrar( registry->GetRegistrarForPlugin("FlutterGemmaPlugin")); } diff --git a/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake b/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake index 71fc21b2..8dcbac20 100644 --- a/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake +++ b/codelabs/multimodal-flutter-gemma/step_01_starter/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus flutter_gemma ) diff --git a/codelabs/multimodal-flutter-gemma/step_02_vision/android/app/build.gradle.kts b/codelabs/multimodal-flutter-gemma/step_02_vision/android/app/build.gradle.kts index 43915ad4..ad7acaad 100644 --- a/codelabs/multimodal-flutter-gemma/step_02_vision/android/app/build.gradle.kts +++ b/codelabs/multimodal-flutter-gemma/step_02_vision/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.multimodal" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/multimodal-flutter-gemma/step_02_vision/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/multimodal-flutter-gemma/step_02_vision/macos/Flutter/GeneratedPluginRegistrant.swift index 1a668cbd..f50d0aeb 100644 --- a/codelabs/multimodal-flutter-gemma/step_02_vision/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/multimodal-flutter-gemma/step_02_vision/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,12 +5,14 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/multimodal-flutter-gemma/step_02_vision/pubspec.yaml b/codelabs/multimodal-flutter-gemma/step_02_vision/pubspec.yaml index 2d6d1584..b9c96fc0 100644 --- a/codelabs/multimodal-flutter-gemma/step_02_vision/pubspec.yaml +++ b/codelabs/multimodal-flutter-gemma/step_02_vision/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # Picks a photo on all six platforms: a gallery on Android and iOS, a file # dialog (via file_selector) on macOS, Windows, Linux and the web. diff --git a/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugin_registrant.cc b/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugin_registrant.cc index bdc9b64b..20486aa2 100644 --- a/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugin_registrant.cc @@ -6,10 +6,13 @@ #include "generated_plugin_registrant.h" +#include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugins.cmake b/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugins.cmake index 812d3d4c..265de1b5 100644 --- a/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugins.cmake +++ b/codelabs/multimodal-flutter-gemma/step_02_vision/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma ) diff --git a/codelabs/multimodal-flutter-gemma/step_03_audio/android/app/build.gradle.kts b/codelabs/multimodal-flutter-gemma/step_03_audio/android/app/build.gradle.kts index 43915ad4..ad7acaad 100644 --- a/codelabs/multimodal-flutter-gemma/step_03_audio/android/app/build.gradle.kts +++ b/codelabs/multimodal-flutter-gemma/step_03_audio/android/app/build.gradle.kts @@ -19,7 +19,10 @@ android { applicationId = "dev.fluttergemma.multimodal" // You can update the following values to match your application needs. // For more information, see: https://flutter.dev/to/review-gradle-config. - minSdk = flutter.minSdkVersion + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 targetSdk = flutter.targetSdkVersion // Uses the version code from pubspec.yaml. When using split APKs, 1000 * ABI_VERSION // is added automatically by Flutter. (https://developer.android.com/studio/build/configure-apk-splits#configure-APK-versions) diff --git a/codelabs/multimodal-flutter-gemma/step_03_audio/macos/Flutter/GeneratedPluginRegistrant.swift b/codelabs/multimodal-flutter-gemma/step_03_audio/macos/Flutter/GeneratedPluginRegistrant.swift index 5172c542..c68ce0d5 100644 --- a/codelabs/multimodal-flutter-gemma/step_03_audio/macos/Flutter/GeneratedPluginRegistrant.swift +++ b/codelabs/multimodal-flutter-gemma/step_03_audio/macos/Flutter/GeneratedPluginRegistrant.swift @@ -5,6 +5,7 @@ import FlutterMacOS import Foundation +import connectivity_plus import file_selector_macos import flutter_gemma import large_file_handler @@ -12,6 +13,7 @@ import record_macos import shared_preferences_foundation func RegisterGeneratedPlugins(registry: FlutterPluginRegistry) { + ConnectivityPlusPlugin.register(with: registry.registrar(forPlugin: "ConnectivityPlusPlugin")) FileSelectorPlugin.register(with: registry.registrar(forPlugin: "FileSelectorPlugin")) FlutterGemmaPlugin.register(with: registry.registrar(forPlugin: "FlutterGemmaPlugin")) LargeFileHandlerPlugin.register(with: registry.registrar(forPlugin: "LargeFileHandlerPlugin")) diff --git a/codelabs/multimodal-flutter-gemma/step_03_audio/pubspec.yaml b/codelabs/multimodal-flutter-gemma/step_03_audio/pubspec.yaml index 5da20b5a..af075fe9 100644 --- a/codelabs/multimodal-flutter-gemma/step_03_audio/pubspec.yaml +++ b/codelabs/multimodal-flutter-gemma/step_03_audio/pubspec.yaml @@ -36,10 +36,10 @@ dependencies: cupertino_icons: ^1.0.8 # The engine-agnostic core: registry, install/runtime API, chat. - flutter_gemma: ^1.7.1 + flutter_gemma: ^1.8.1 # The .litertlm inference engine. Engines are opt-in — core # registers none, so this package must be added explicitly. - flutter_gemma_litertlm: ^1.6.2 + flutter_gemma_litertlm: ^1.6.3 # Picks a photo on all six platforms: a gallery on Android and iOS, a file # dialog (via file_selector) on macOS, Windows, Linux and the web. diff --git a/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugin_registrant.cc b/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugin_registrant.cc index e1be5753..78c359b5 100644 --- a/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugin_registrant.cc +++ b/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugin_registrant.cc @@ -6,11 +6,14 @@ #include "generated_plugin_registrant.h" +#include #include #include #include void RegisterPlugins(flutter::PluginRegistry* registry) { + ConnectivityPlusWindowsPluginRegisterWithRegistrar( + registry->GetRegistrarForPlugin("ConnectivityPlusWindowsPlugin")); FileSelectorWindowsRegisterWithRegistrar( registry->GetRegistrarForPlugin("FileSelectorWindows")); FlutterGemmaPluginRegisterWithRegistrar( diff --git a/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugins.cmake b/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugins.cmake index 9d7135d2..a12621d0 100644 --- a/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugins.cmake +++ b/codelabs/multimodal-flutter-gemma/step_03_audio/windows/flutter/generated_plugins.cmake @@ -3,6 +3,7 @@ # list(APPEND FLUTTER_PLUGIN_LIST + connectivity_plus file_selector_windows flutter_gemma record_windows diff --git a/website/codelabs/getting-started-flutter-gemma/index.md b/website/codelabs/getting-started-flutter-gemma/index.md index 127fbc55..58f83cdb 100644 --- a/website/codelabs/getting-started-flutter-gemma/index.md +++ b/website/codelabs/getting-started-flutter-gemma/index.md @@ -116,14 +116,29 @@ each drags in native binaries you would otherwise ship for nothing. Less than you would expect on any of the six, and nothing at all on two of them. Read the subsection for the platform you are running on and skip the others. -**Android** — one line, because downloading the model is an ordinary HTTPS -request: +**Android** — two things. First, the internet permission, because downloading +the model is an ordinary HTTPS request: ```xml ``` +Second, the API floor, in `android/app/build.gradle.kts`: + +```kotlin +defaultConfig { + // … + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 +``` + +This one is easy to miss, because nothing rejects the build: Flutter's template +floor (24) merges fine, the APK installs, and the failure arrives later as a +`dlopen` error the first time you load a model. The step apps are already at 30. + You do **not** need to declare the OpenCL libraries the GPU backend uses. The plugin's own manifest declares them and the manifest merger folds them into your app. diff --git a/website/codelabs/hybrid-ai-flutter-genkit/index.md b/website/codelabs/hybrid-ai-flutter-genkit/index.md index 55abd4bc..8ac13e25 100644 --- a/website/codelabs/hybrid-ai-flutter-genkit/index.md +++ b/website/codelabs/hybrid-ai-flutter-genkit/index.md @@ -308,6 +308,18 @@ are ordinary HTTPS requests (`android/app/src/main/AndroidManifest.xml`): ``` +…and the API floor in `android/app/build.gradle.kts`, because the on-device +half of the app loads `libLiteRtLm.so`: + +```kotlin +defaultConfig { + // … + // libLiteRtLm.so needs API 30+ Bionic (pthread_cond_clockwait, + // sem_clockwait). Below 30 the app installs and then fails at the first + // model load with a dlopen error. + minSdk = 30 +``` + **iOS** — a deployment target of 15.0 or newer, and three memory entitlements in `ios/Runner/Runner.entitlements`: @@ -393,10 +405,10 @@ Add `genkit_flutter_gemma` and `flutter_gemma`: ```yaml # Step 3: On-device AI (LiteRT-LM engine) genkit_flutter_gemma: ^0.6.0 - flutter_gemma: ^1.7.0 + flutter_gemma: ^1.8.1 # flutter_gemma 1.x registers no engine by default — opt into LiteRT-LM # (.litertlm inference) here. - flutter_gemma_litertlm: ^1.6.1 + flutter_gemma_litertlm: ^1.6.3 ``` Run `flutter pub get`. diff --git a/website/codelabs/inference-engines-flutter-gemma/index.md b/website/codelabs/inference-engines-flutter-gemma/index.md index 7bc0fc54..e6e560df 100644 --- a/website/codelabs/inference-engines-flutter-gemma/index.md +++ b/website/codelabs/inference-engines-flutter-gemma/index.md @@ -56,7 +56,7 @@ point. * Chrome has a hardware floor for its copy of Nano that the flag does not lift. `flutter_gemma_builtin_ai` states it as **~22 GB of free disk and a GPU with more than 4 GB of VRAM**, or a CPU-only path on a machine with 16 GB of RAM. - Under it the probe answers `unavailableDeviceUnsupported` with the flag + Under it the probe answers `unavailableOther` with the flag switched on — which reads like a setup mistake and is not one * **Windows and Linux have no built-in arm at all**, and that is not a gap in your setup: the app is designed to notice and take the downloaded model @@ -145,17 +145,23 @@ all. ### One platform change Gemini Nano's Android SDK requires API 26. The package declares that, and the -manifest merger refuses an app that sets less, so raise the app's floor in -`android/app/build.gradle.kts`: +manifest merger refuses an app that sets less. The app you brought from Getting +Started is already at **30** — LiteRT-LM's own floor, and above this one — so +there is nothing to change here. Leave it where it is: ```kotlin defaultConfig { // … - // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26; - // the manifest merger rejects an app below it. - minSdk = 26 + // flutter_gemma_builtin_ai (ML Kit GenAI / AICore) declares minSdk 26 and + // the manifest merger rejects an app below it; libLiteRtLm.so needs API 30+ + // Bionic (pthread_cond_clockwait, sem_clockwait) on top of that, so 30 is the + // floor for an app that registers both engines. + minSdk = 30 ``` +If you are adding built-in AI to an app that has *no* LiteRT-LM engine in it, +26 is enough. + iOS needs nothing beyond what Getting Started already set up — the iOS 15.0 deployment target and the three memory entitlements. `flutter_gemma_builtin_ai` declares an iOS 15.0 floor of its own, so a project still pinned at Flutter's diff --git a/website/codelabs/multimodal-flutter-gemma/index.md b/website/codelabs/multimodal-flutter-gemma/index.md index b957c75a..b0e44ec8 100644 --- a/website/codelabs/multimodal-flutter-gemma/index.md +++ b/website/codelabs/multimodal-flutter-gemma/index.md @@ -803,8 +803,9 @@ is text-only. What a learner should expect there: the app loads, the model downloads, text chat works, and both attachment buttons are disabled with a line naming the runtime that refused. That is the app working correctly. (Full vision on the web today means MediaPipe `.task` models and the -`flutter_gemma_mediapipe` package — a different engine, and the subject of the -[Inference Engines codelab](/codelabs/inference-engines-flutter-gemma).) +`flutter_gemma_mediapipe` package — a different engine, documented in +[MediaPipe](/docs/mediapipe); the Inference Engines codelab pairs LiteRT-LM with +built-in AI instead, so it is not the place to look for this one.) **The iOS Simulator** is the case the two questions do not cover, and the reason is not that Dart cannot see it — `device_info_plus` exposes diff --git a/website/content/docs/genkit.md b/website/content/docs/genkit.md index 8aa69e0f..c863a536 100644 --- a/website/content/docs/genkit.md +++ b/website/content/docs/genkit.md @@ -21,7 +21,7 @@ with the on-device model exactly as it would with any cloud provider. ``` dependencies: genkit: ^0.16.0 # the framework itself — every snippet below uses it - genkit_flutter_gemma: ^0.6.1 + genkit_flutter_gemma: ^0.6.0 flutter_gemma: ^1.8.2 # Add the inference engine(s) you need: flutter_gemma_litertlm: ^1.6.3 # .litertlm models (mobile + desktop) + LiteRtEmbeddingBackend From 1c642e58d0ba207313ccecf703592cb415f6196c Mon Sep 17 00:00:00 2001 From: Sasha Denisov Date: Sun, 13 Sep 2026 17:46:13 +0200 Subject: [PATCH 20/21] docs: point every package and docs page at the skill that covers it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The speech README claimed the package bundles a skill. It does not: the only skills/ directory in the repo is core's, and only core's .pubignore ships it. The skill reaches a speech user because speech depends on core and the CLI scans the dependency graph — so the README now says that instead. Its command was bare `skills@ get`, the only one in the project without --all. Without --all or --skill the CLI goes to _promptForSkillsToInstall, an interactive picker over every skill in the graph: not one speech skill, and not something to hand a reader in a copy-paste block. The seven satellites that map to a skill now carry the same corrected section, and the seven docs pages that own a topic name their skill, so a reader wanting their assistant to get one API right is not told to install all seven and hope. Nothing pointed anywhere before this. flutter_gemma_agent is the one package where "skills" means the opposite thing — assets the on-device model runs. Both its README and /docs/agent now say so, and that `get --all` never installs them: they live under assets/skills/, and skill_scanner.dart only scans /skills. Two claims on the Package Skills page corrected against what ships: the built-in AI row omitted the web arm (Nano in Chrome, Phi-4-mini in Edge), and --skill was described as installing a single skill when it is an addMultiOption. --- packages/flutter_gemma_agent/README.md | 5 +++++ packages/flutter_gemma_builtin_ai/README.md | 8 ++++++++ packages/flutter_gemma_embeddings/README.md | 8 ++++++++ packages/flutter_gemma_litertlm/README.md | 8 ++++++++ packages/flutter_gemma_mediapipe/README.md | 8 ++++++++ packages/flutter_gemma_onnx/README.md | 8 ++++++++ packages/flutter_gemma_rag_qdrant/README.md | 8 ++++++++ packages/flutter_gemma_rag_sqlite/README.md | 8 ++++++++ packages/flutter_gemma_speech/README.md | 4 ++-- website/content/docs/agent.md | 8 ++++++++ website/content/docs/builtin-ai.md | 2 ++ website/content/docs/embeddings-and-rag.md | 2 ++ website/content/docs/function-calling.md | 2 ++ website/content/docs/getting-started.md | 2 ++ website/content/docs/mediapipe.md | 2 ++ website/content/docs/onnx.md | 2 ++ website/content/docs/package-skills.md | 6 +++--- website/content/docs/speech.md | 2 ++ 18 files changed, 88 insertions(+), 5 deletions(-) diff --git a/packages/flutter_gemma_agent/README.md b/packages/flutter_gemma_agent/README.md index 685cf17a..a4864fe9 100644 --- a/packages/flutter_gemma_agent/README.md +++ b/packages/flutter_gemma_agent/README.md @@ -70,6 +70,11 @@ WebView), verified on hardware. On web the skill runs in a sandboxed `