Skip to content

fix(dedup): request buffered detector responses and surface scan progress - #741

Open
bmdy5 wants to merge 1 commit into
nashsu:mainfrom
bmdy5:fix/dedup-scan-buffered-response
Open

fix(dedup): request buffered detector responses and surface scan progress#741
bmdy5 wants to merge 1 commit into
nashsu:mainfrom
bmdy5:fix/dedup-scan-buffered-response

Conversation

@bmdy5

@bmdy5 bmdy5 commented Sep 9, 2026

Copy link
Copy Markdown

What & why

The duplicate-detection scan asks the LLM for a structured JSON result (candidate duplicate groups), but the request was sent through the streaming chat path. On providers that return an empty body or cut the stream when the model only produces a final JSON payload, the scan hangs with no visible feedback and eventually times out without candidates.

This PR makes the structured detector request non-streaming (buffered) so a complete response always arrives, and adds phase progress reporting so long scans no longer look frozen:

  • runDuplicateDetection now accepts an onProgress callback and reports 3 phases: loading (reading pages) / embedding (per-page) / scanning (per batch).
  • Batch-loop failures now include the failing batch number in the error.
  • Maintenance UI renders a live progress bar + percentage during embedding/scanning phases.

Changes

File Change
src/lib/dedup-runner.ts buffered detector request; progress types & callbacks; batch error context
src/lib/dedup_embedding.ts per-page embedding progress callback (re-added missing signal in CandidateOptions)
src/components/settings/sections/maintenance-section.tsx live progress bar for scan phases
src/i18n/{en,zh,it,ru}.json new progress strings (fixed stray double-comma in it/ru)
src/lib/dedup-runner.test.ts test asserting detector request uses streamingEnabled: false

Verification

  • npm run typecheck — passes
  • vitest run on dedup-related suites — 88 tests pass

Risks / non-goals

  • Only the structured detector call is switched to buffered; streaming chat UX is untouched.
  • Progress is best-effort UI feedback; no behavioral change to merge logic.
  • Does not change dedup thresholds or grouping logic.

Review focus

  • The buffered-request change in dedup-runner.ts (buildDedupLlmCall(..., streamingEnabled: false)) — confirm no streaming-dependent code downstream.
  • Progress plumbing is intentionally minimal.

Rollback

Revert this commit; behavior returns to the previous streaming detector path.

…ress

Structured duplicate-detection calls use streamChat with streaming
enabled, which can return an empty body on some providers and leave
the scan hanging with no feedback. Request a non-streaming (buffered)
response for the detector so a complete JSON payload always arrives,
and report phase progress (loading / embedding / scanning batches)
back to the maintenance UI with a live progress bar.

Fixes an issue where long scans appeared frozen and could time out
without producing candidate groups.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant