Skip to content

Resumable sync with per-entity error journal - #100

Merged
vboctor merged 2 commits into
mainfrom
sync-error-handling
Apr 26, 2026
Merged

vboctor merged 2 commits into
mainfrom
sync-error-handling

Conversation

@vboctor

@vboctor vboctor commented Apr 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • New `sync_errors` journal table: per-entity fetch failures are recorded and skipped, not aborting the run; retried (up to 3 attempts) on the next sync.
  • Cursor persisted after every batch — interrupted full syncs resume from the last batch instead of restarting.
  • MantisHub initial sync uses a `snapshotCeiling` pinned at run start (`filter_updated_before`) so DESC pagination stays stable across resumes; list endpoint uses `select=id,updated_at`; `PAGE_SIZE=25` everywhere; per-issue detail fetches run 5-way concurrent.
  • MantisHub users phase aborts after 10 fetch failures and emits shell entities for the remaining users from data already collected during the issues phase. User failures are never journaled.
  • `startSyncJob` was double-calling `ensureSyncJob` and silently exiting because the guard saw the synchronously-seeded entry as already active — fixed by claiming the seeded entry by status.
  • UI: error counter shown in progress card; `counter-deleted` is grey, new `counter-failed` is red. Status messages drop the collection-name and inline counter redundancy.

Test plan

  • `bun run ci` passes locally (markdown lint, typecheck, all test suites, UI build, worker build)
  • Full sync of a large MantisHub project completes; cursor advances per batch and resume works after a kill
  • Inject a 5xx for a single issue and confirm it lands in `sync_errors` (visible as "N failed" in the UI) without aborting
  • Re-run sync and confirm the journaled entry retries; after 3 failed attempts, retries stop
  • Users-phase abort triggers when most user fetches return 403; shell entities are emitted for the rest
  • Click "Sync" — UI no longer hangs on "starting"; status moves through "syncing" → progress messages

🤖 Generated with Claude Code

vboctor and others added 2 commits April 25, 2026 23:36
Per-entity fetch failures now go to a sync_errors journal instead of
aborting the run. The cursor is persisted after every batch so an
interrupted sync resumes from where it stopped, and journal entries are
retried on the next sync (up to 3 attempts each). MantisHub initial
sync uses a snapshotCeiling pinned at run start so DESC pagination
stays stable across resumes; full-sync list pages are 100, incremental
batches are 25. The users phase aborts after 10 fetch failures and
emits shell entities for the remaining users from data already
collected during the issues phase.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The parallel-jobs refactor seeded an active job synchronously in the HTTP
handler and then called ensureSyncJob again from startSyncJob, where the
already-active guard returned null and the sync silently never ran —
"starting" hung until the 60s retention sweep. startSyncJob now picks up
the seeded entry from the map and claims it iff status === "starting".

MantisHub: list endpoint now uses select=id,updated_at (the per-issue
detail call returns the full payload anyway), single PAGE_SIZE=25 for
both full and incremental, and per-issue detail fetches run 5-way
concurrent for a meaningful RTT-bound speedup. Status messages drop the
collection-name prefix and inline counters since both are visible in the
progress card. Counter-deleted is neutral grey now; counter-failed gets
the red.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vboctor
vboctor merged commit 015a64d into main Apr 26, 2026
1 check passed
@vboctor
vboctor deleted the sync-error-handling branch April 26, 2026 17:14
@vboctor vboctor self-assigned this Apr 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant