fix(dedup): request buffered detector responses and surface scan progress - #741
Open
bmdy5 wants to merge 1 commit into
Open
fix(dedup): request buffered detector responses and surface scan progress#741bmdy5 wants to merge 1 commit into
bmdy5 wants to merge 1 commit into
Conversation
…ress Structured duplicate-detection calls use streamChat with streaming enabled, which can return an empty body on some providers and leave the scan hanging with no feedback. Request a non-streaming (buffered) response for the detector so a complete JSON payload always arrives, and report phase progress (loading / embedding / scanning batches) back to the maintenance UI with a live progress bar. Fixes an issue where long scans appeared frozen and could time out without producing candidate groups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The duplicate-detection scan asks the LLM for a structured JSON result (candidate duplicate groups), but the request was sent through the streaming chat path. On providers that return an empty body or cut the stream when the model only produces a final JSON payload, the scan hangs with no visible feedback and eventually times out without candidates.
This PR makes the structured detector request non-streaming (buffered) so a complete response always arrives, and adds phase progress reporting so long scans no longer look frozen:
runDuplicateDetectionnow accepts anonProgresscallback and reports 3 phases:loading(reading pages) /embedding(per-page) /scanning(per batch).Changes
src/lib/dedup-runner.tssrc/lib/dedup_embedding.tssignalinCandidateOptions)src/components/settings/sections/maintenance-section.tsxsrc/i18n/{en,zh,it,ru}.jsonsrc/lib/dedup-runner.test.tsstreamingEnabled: falseVerification
npm run typecheck— passesvitest runon dedup-related suites — 88 tests passRisks / non-goals
Review focus
dedup-runner.ts(buildDedupLlmCall(..., streamingEnabled: false)) — confirm no streaming-dependent code downstream.Rollback
Revert this commit; behavior returns to the previous streaming detector path.