Add flush(), so a qdrant index survives the process (#492) - #506
Closed
DenisovAV wants to merge 8 commits into
Closed
Add flush(), so a qdrant index survives the process (#492)#506DenisovAV wants to merge 8 commits into
DenisovAV wants to merge 8 commits into
Conversation
Fifth codelab. Four step apps under codelabs/function-calling-flutter-gemma/ plus a fifth step that is not an app, and the text at website/codelabs/function-calling-flutter-gemma/index.md. step_01_starter is Getting Started's finished app, byte-identical in lib/ and test/ — a third mirror pair, added to check_codelabs.sh's MIRRORS table and proved to fail closed. step_02_one_tool declares one Tool on FunctionGemma 270M and writes the call/response loop out by hand, so the learner sees the three questions it actually asks — and sees why a call the app declines to run still has to be answered: the SDK commits a call to history the moment it yields it. step_03_the_loop deletes that and calls generateChatResponseWithTools, with maxToolTurns as the stop condition and onMaxToolTurns as the signal. What the SDK adds over the hand-written version is the three exits it did not have — cancellation, a throwing tool, a stream that errors mid-turn — each of which balances the history before it returns. step_04_finetune is data and commands, not an app: a tools.json and 72 prompt-to-tool-call rows for a litetune run over the same three tools, ending in a .litertlm the app opens through fromFile. Not one line of Dart changes, which is the point. It has no pubspec.yaml, so the gate does not discover it; codelabs/README.md now says so, since it previously claimed every step directory is an app. complete pays 2.59 GB for Gemma 4 E2B and spends it on the two things a 270M model cannot do: reason out loud, and be told it must call something. Three tools dispatched by name, toolChoice and isThinking as session settings that rebuild the chat, and a model list that finds a tuned model again on the next launch through listInstalledModels + getModelPath. Tools are the clock, a platform reading and multiplication — answers a learner can check by eye with the network off, which is the whole argument for running this on device. Identity dev.fluttergemma.functioncalling, distinct from the other four. All six platforms in every app.
Measured on macOS. The model this codelab opens with generates nothing but
`<pad>` repeated to the token limit when the engine is built for the GPU
backend — no exception, no warning, a chat that looks alive and returns
filler. Steps 2 and 3 shipped that way and would not have worked for anyone
who ran them.
On CPU the same weights ask for the tool correctly:
[fg] calls: [multiply({a: 6, b: 7})]
So both steps now ask for `PreferredBackend.cpu`, and maxTokens drops 4096 →
1024, which is what this checkpoint is built for and the value the working run
used. A 270M model does not need a GPU; the codelab says so and says why.
Not a codelab bug: the example app configures the same model with
PreferredBackend.gpu, so it has the same problem off Android. Worth an issue
against the plugin — a silent `<pad>` is the failure mode nobody diagnoses.
…lly do Review of #496 found the function-calling codelab asserting five things the source contradicts. - `ToolChoice.none` does not stop Gemma 4 seeing the declarations: the litertlm `createChat` forwards `tools:` regardless, and `none` only switches off the SDK's swallow — so a call under it streams raw tool_calls JSON into the bubble. Say that instead. - `ToolChoice.required` is a no-op on Gemma 4 too: the "you must call" text lives on the Dart-injection path passthrough models skip, and `tools_json` carries no tool_choice. `supportsRequiredToolChoice` is now false for both models, so the app's own notice tells the truth at the point of use. - Step 4's base checkpoint is HF-gated with manual approval. Scope the "no token needed" promise to the app, and put "request access first" where someone decides whether to start the step. - Web function calling is known-broken from source, not untested: the browser runtime never overrides `createChat`, so the session gets no declarations. - Drop the stale "so 4096" argument left under a `maxTokens: 1024` call. Plus four minors: litetune's command numbering disambiguated from the codelab's steps, step_02's stream-error hole disclosed where it happens, its one unguarded setState guarded, and two claims softened to what was measured.
…acOS I measured FunctionGemma answering `<pad>` on the GPU backend on macOS and then forced PreferredBackend.cpu on every platform. The measurement was real; the conclusion drawn from it was not. On Android the model runs on the GPU — which is how the plugin's own example configures it — so a macOS-only result was pessimising the platform most learners use. The backend is now asked for only where the failure was seen, and the text and the comment say so in as many words, including "do not read this as FunctionGemma needs CPU".
I measured FunctionGemma answering <pad> on the GPU and blamed first the backend, then macOS. Both were wrong. The cause is the published .litertlm: it was converted before litetune set prefer_activation_type=fp32, and without that key the GPU path floods <pad> on Android and on macOS Metal alike. A re-conversion with a current litetune scores the same on GPU as on CPU and runs about 1.5x faster. So the CPU request goes back to every platform — it is a workaround for one downloadable file — and the text says which file, why, and that Step 4's re-conversion is what removes the need for it. That also gives the fine-tuning step a payoff a learner can see before changing a single training row. Measured by the owner on Metal: without the key 40/40 rows returned pad and nothing else; with it, 0.875 exact and 40/40 tool names, the same as CPU.
step_01_starter was a byte-identical mirror of the Getting Started codelab's finished app, so it downloaded Gemma 3 1B — 0.5 GB behind a Hugging Face licence gate — and Step 2 immediately replaced it with FunctionGemma 270M and never opened it again. A learner paid half a gigabyte and a licence request for a model this codelab does not use. A starter does not have to be another codelab's finished app. This one is now Step 2 minus the tool machinery: the same 284 MB ungated FunctionGemma, a plain streaming chat, no token plumbing anywhere. main.dart, model.dart and download_page.dart are byte for byte Step 2's, so the diff between the two steps is exactly tools.dart plus the loop in chat_page.dart. The mirror pair is dropped from MIRRORS in tool/check_codelabs.sh; the other two rows and the fail-closed behaviour are untouched, and the check still reports drift on a perturbed pair. The codelab text drops the Getting Started prerequisite, says the app needs no Hugging Face token in any step, and states the download as 284 MB for Steps 1-3 plus 2.59 GB for Gemma 4 in Step 5.
The package never called EdgeShard.flush(). Points added through addDocument stayed in the shard's in-RAM segment, so an index built in one session was gone in the next and the corpus was embedded again from scratch on every launch. The reporter measured it on two Android devices and named the missing piece: the segment's id-tracker files are the ones that never reach disk, which is why the next open fails with "Failed to load ID tracker mappings". close() persisted already — it unloads the shard, and unload writes too — but that does not help the case this is about. An Android app the system kills in the background never gets to close anything, and until now there was no way to persist without giving up the store. flush() joins the VectorStoreRepository contract rather than only the qdrant class, with a no-op default body: the SQLite stores are already durable (sqlite3 autocommits, and the web VFS writes through to OPFS or IndexedDB), so their implementations say so at their own site instead of leaving the caller to guess which backends need the call. Every implementation uses `implements`, not `extends`, so the default is documentation rather than inheritance — each of the seven declares it. The qdrant flush runs on the lifecycle lane next to close and clear, since a flush overlapping a close would reach a shard the other call had unloaded, and it is quiet on a store that was never initialized: callers flush from lifecycle callbacks they cannot make conditional. Tested against a real shard rather than a mock: 20 documents written, then the id-tracker files asserted absent before flush and present after. Mutation- checked by dropping the flush call — that test, and only that test, fails. The dependency floor moves to flutter_gemma 1.7.3 in both RAG packages: their `@override flush()` needs the method to exist in the contract.
# Conflicts: # packages/flutter_gemma/CHANGELOG.md # packages/flutter_gemma/pubspec.yaml
Owner
Author
|
Superseded by #507. This branch was cut from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #492 (first finding).
flutter_gemma_rag_qdrantnever calledEdgeShard.flush(). Points added throughaddDocumentstayed in the shard's in-RAM segment, so an index built in one session was gone in the next and the corpus was embedded again from scratch on every launch.@think2execute measured it on two Android devices and named the missing piece precisely — the next open fails with
Failed to load ID tracker mappings. That turned out to be exactly the right place to look.What was actually missing
Writing 20 documents and looking at the segment directory before and after a flush:
flush()mutable_id_tracker.mappingsmutable_id_tracker.versionsEverything else — vector storage, payload storage,
segment.json— is on disk well before this. The id tracker is the part that is not, and it is the part the failing open asks for.Why
close()was not already enoughIt persists, because it unloads the shard and unload writes too. But that only covers a store the app closes cleanly. An Android app the system kills in the background never gets there, and until now there was no way to persist without giving up the store.
Scope
flush()joins theVectorStoreRepositorycontract, not just the qdrant class, with a no-op default body. The SQLite stores are already durable —sqlite3autocommits, and the web VFS writes through to OPFS or IndexedDB — so their implementations say so at their own site rather than leaving a caller to guess which backends need the call.All seven implementations use
implementsrather thanextends, so the default body is documentation rather than inheritance; each one declares the method.The qdrant flush runs on the lifecycle lane beside
closeandclear— a flush overlapping a close would reach a shard the other call had already unloaded — and is quiet on a store that was never initialized, since callers flush from lifecycle callbacks they cannot make conditional.Verified
qdrant_lifecycle_test.dart, against a real shard rather than a mockflush()call fails the id-tracker test, and only that oneflutter_gemma_rag_qdrant113/113,flutter_gemma_rag_sqlite121/121,flutter_gemma701/701flutter analyze packages/— no errors or warningsThe dependency floor moves to
flutter_gemma: ^1.7.3in both RAG packages, since their@override flush()needs the method to exist in the contract.Not addressed
The second finding in #492 — one Samsung device failing to reopen a shard that is intact and loads on Windows — is untouched here. It looks like it belongs to
qdrant_edgeor its Rust layer rather than to this package, and it needs its own reproduction.