Summary
When a `sync --full` of a large channel is interrupted by the per-channel timeout (`channel message crawl deferred` / `context deadline exceeded`), each retry makes less and less net progress — eventually almost none. The cause is that the backfill loop overwrites the channel's `latest_message_id` sync state with message ids from the backfill region (old messages), so the next pass's forward sync re-crawls everything between that stale pointer and the channel's true head before any backfill resumes, consuming most of the channel timeout budget on duplicate fetches.
Version: 0.10.0 (darwin_arm64 release binary).
Reproduction
- Run `discrawl sync --full` against a guild with one channel large enough that its crawl exceeds the 5m channel timeout (ours: ~200k messages).
- The channel defers with `context deadline exceeded`; re-run `discrawl sync --channels --full` repeatedly to resume.
- Watch `messages=` per run stay constant (~28k fetched per 5m) while net-new rows in the `messages` table collapse — in our case from ~28k/pass to ~500/pass by the time the cursor was a year behind the channel head.
Observed state after a few passes (channel head is actually in 2026):
channel:<id>:latest_message_id | 1392816300451168370 <- mid-2025 id, from the backfill region
channel:<id>:backfill_before_id | 1363347906014482603
Root cause
`internal/syncer/message_sync.go`, in `syncBackfillPages` (~line 448):
newest = maxSnowflake(newest, pageNewest)
messageCount += len(eligible)
if newest != "" {
if err := s.store.SetSyncState(ctx, channelLatestScope(channel.ID), newest); err != nil {`newest` here is the newest id within the backfilled pages, which is by construction older than the stored channel latest. Persisting it to `channelLatestScope` regresses the pointer. `syncFullChannelHistory` then starts the next run with `syncForwardPages(ctx, channel, state.Latest, ...)` from that regressed id, re-fetching the entire span between the backfill cursor and the channel head (all duplicate `INSERT OR IGNORE` work) before the backfill loop gets whatever remains of the timeout. The deeper the backfill cursor, the larger the duplicate span — so resume cost grows roughly quadratically with channel size.
Suggested fix
In `syncBackfillPages`, don't persist `channelLatestScope` at all (backfill should only advance `channelBackfillScope`), or guard it with `maxSnowflake(state.StoredLatest, newest)` so it can never regress.
Workaround
Between resume passes, manually restore the true head id:
sqlite3 discrawl.db "UPDATE sync_state SET cursor='<true head id>' WHERE scope='channel:<id>:latest_message_id'"
With that in place, each resumed pass spends its full timeout on new backfill pages again.
Summary
When a `sync --full` of a large channel is interrupted by the per-channel timeout (`channel message crawl deferred` / `context deadline exceeded`), each retry makes less and less net progress — eventually almost none. The cause is that the backfill loop overwrites the channel's `latest_message_id` sync state with message ids from the backfill region (old messages), so the next pass's forward sync re-crawls everything between that stale pointer and the channel's true head before any backfill resumes, consuming most of the channel timeout budget on duplicate fetches.
Version: 0.10.0 (darwin_arm64 release binary).
Reproduction
Observed state after a few passes (channel head is actually in 2026):
Root cause
`internal/syncer/message_sync.go`, in `syncBackfillPages` (~line 448):
`newest` here is the newest id within the backfilled pages, which is by construction older than the stored channel latest. Persisting it to `channelLatestScope` regresses the pointer. `syncFullChannelHistory` then starts the next run with `syncForwardPages(ctx, channel, state.Latest, ...)` from that regressed id, re-fetching the entire span between the backfill cursor and the channel head (all duplicate `INSERT OR IGNORE` work) before the backfill loop gets whatever remains of the timeout. The deeper the backfill cursor, the larger the duplicate span — so resume cost grows roughly quadratically with channel size.
Suggested fix
In `syncBackfillPages`, don't persist `channelLatestScope` at all (backfill should only advance `channelBackfillScope`), or guard it with `maxSnowflake(state.StoredLatest, newest)` so it can never regress.
Workaround
Between resume passes, manually restore the true head id:
With that in place, each resumed pass spends its full timeout on new backfill pages again.