fix(sync): emit a reconcile-sweep-completion event instead of relying on a gauge - #2533
Merged
Merged
Conversation
… on a gauge
The work-starvation alert built earlier today fired repeatedly on a healthy
fleet. jobs.pending in the health snapshot is a point-in-time gauge sampled
every 5 minutes; once job draining got fast (a direct result of this
morning's own reliability fixes), the queue was non-empty for well under a
minute per ~10-minute sweep cycle. A 5-minute sampler missed that window in
59 of 60 samples over a verified 5-hour stretch during which the sweep ran
flawlessly the entire time (confirmed via logs: 100 pulls enqueued on
schedule, zero gaps, zero errors).
No window width fixes a gauge with a ~2% duty cycle — the math needs a
20+ hour window to reliably catch even one nonzero sample, which defeats
the point of alerting same-day.
Emits sync_reconcile_sweep on every sweep cycle completion (including
enqueued: 0), independent of the 5-minute health-snapshot cadence. Counting
discrete events in a window ('did N sweeps complete recently') can't be
fooled by sampling gaps the way a periodic gauge read can — a sweep either
ran or it didn't.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A work-starvation alert built this morning (fires when
jobs.pendingshows zero for too long) started firing repeatedly on a healthy fleet — 5 notification emails over 9 hours.Investigated with real data rather than trusting the first plausible explanation: cross-checked
compass-sync-1's logs against a 5-hour window (06:21–11:12) where PostHog'sjobs.pendingsnapshot showed 0 in 59 of 60 samples. The sweep ran flawlessly the entire time — 100 pulls enqueued every ~10 minutes, zero gaps, zero errors. The queue really is draining in well under a minute now (a direct result of this morning's own reliability fixes), and a 5-minute point-in-time sample essentially never lands while anything is queued.No window width fixes this. At a ~2% hit rate, reliably catching even one nonzero sample needs a window pushing 20+ hours — which defeats the purpose of a same-day alert. A gauge sampled periodically is the wrong tool once the thing it measures moves faster than the sample rate.
Change
Emit
sync_reconcile_sweepon every sweep cycle completion (includingenqueued: 0), independent of the 5-minute health-snapshot cadence. This sidesteps the sampling problem entirely — counting discrete completion events in a window ("did N sweeps complete in the last hour") can't be fooled by sampling gaps the way reading a periodic gauge can. A sweep either ran or it didn't; there's no brief window for a sampler to miss.Threads
posthogandidentityintobuildSchedulers, following the samecaptureSafelypattern the health-snapshot sweep already uses.Tests
753/753 sync tests pass, type-check and lint clean. No dedicated test added for the
app.tswiring itself — that file has no existing test coverage (it's integration wiring exercised by the full service), consistent with the current codebase.Follow-up (not in this PR)
Once this ships and events start flowing in prod, the disabled
Sync reconcile sweep starved (production)PostHog alert will be rebuilt on top ofcount(sync_reconcile_sweep events in trailing N minutes)instead of the gauge, then re-enabled.🤖 Generated with Claude Code