fix(syncer): stop backfill from regressing latest_message_id - #86
Merged
steipete merged 2 commits intoJun 10, 2026
Merged
Conversation
syncBackfillPages persisted the newest id of each backfilled page to channel:<id>:latest_message_id. During a resumed --full sync the backfill region is by construction older than the stored head, so every page write regressed the pointer; when the per-channel timeout then deferred the channel, the next pass re-crawled the whole span between the backfill cursor and the true head before resuming backfill. Resume cost grew roughly quadratically with channel size. Thread the already-known head into syncBackfillPages as a floor and only checkpoint the latest pointer when it actually advances — i.e. when backfill starts at the channel head on a first --full run, the one case the in-loop write existed for. Fixes openclaw#85 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scope channel latest-message checkpoint advancement to canonical decimal Discord snowflakes and make the database write monotonic. Keep empty-channel checkpoint markers insert-only so a concurrent tail event cannot be cleared. Add regressions for resumed large backfills, live tail older/newer events, DB-boundary concurrent writes, digit-boundary ordering, empty cursors, and malformed cursor handling. Co-authored-by: Hannes Rudolph <hrudolph@gmail.com>
Contributor
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(syncer): stop backfill from regressing latest_message_id This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #85.
Problem
syncBackfillPagespersisted the newest id of each backfilled page tochannel:<id>:latest_message_id. During a resumed--fullsync the backfill region is by construction older than the stored head, so every page write regressed the pointer. When the per-channel timeout then deferred the channel, the next pass's forward sync re-crawled the entire span between the backfill cursor and the channel's true head (all duplicateINSERT OR IGNOREwork) before backfill got whatever remained of the timeout budget — so resume cost grew roughly quadratically with channel size. On a ~200k-message channel, net-new rows per 5-minute pass collapsed from ~28k to ~500.Fix
syncFullChannelHistoryalready knows the current head (state.Latestmaxed with the forward-sync result), so it now passes that intosyncBackfillPagesas a floor. The in-loopchannelLatestScopecheckpoint is only written when the backfill page's newest id actually advances past the floor — which only happens in the one case the write existed for: a first-run--fullwhose backfill starts at the channel head (no stored latest yet). With a stored head in place, backfill now only advanceschannel:<id>:backfill_before_id, and the pointer can never regress.Test
TestSyncFullBackfillDoesNotRegressLatestPointerreproduces the issue: a half-backfilled channel (latest=250, backfill cursor=200) whose backfill is interrupted bycontext.DeadlineExceededmid-run. Without the fix the latest pointer regresses to 199 (the newest id in the backfilled page); with the fix it stays at 250, and the resumed pass fetches only the remaining 99 backfill messages instead of re-crawling the head span.Verified:
go build ./...,go vet ./...,go test -count=1 ./..., andgo test -race ./internal/syncer/all pass; the new test fails againstmainwithexpected: "250" / actual: "199".🤖 Generated with Claude Code