Skip to content

Commit f20bf92

Browse files
committed
fix(selfhost): widen orb-relay-drain checkin margin to match redeploy cadence
LOOPOVER-12 fired 1600+ missed-checkin outages over 2+ weeks. Ruled out the reentrancy-guard overlap-skip (0 occurrences in 14 days of live instrumentation) and in-process timeouts (~13 events, too few to explain the volume). The container's own StartedAt resets on every redeploy unlike its stable sidecars, and boot (image pull + Postgres-readiness wait + init) routinely exceeds the old 2-minute margin on a fleet redeploying multiple times a day. Raise checkinMargin from 2 to 6 minutes so a normal redeploy doesn't trip a false outage, while a genuinely stuck instance still alerts within ~21 minutes.
1 parent 47b34e9 commit f20bf92

2 files changed

Lines changed: 39 additions & 1 deletion

File tree

src/selfhost/sentry.ts

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,21 @@ const SENTRY_MONITORS: Record<SentryMonitorName, { slug: string; config: SentryM
109109
slug: "orb-relay-drain",
110110
config: {
111111
schedule: { type: "interval", value: 1, unit: "minute" },
112-
checkinMargin: 2,
112+
// #6685-followup-2 (LOOPOVER-12, 1600+ "missed check-in" occurrences over 2+ weeks, still recurring):
113+
// checkinMargin: 2 was too tight for a self-host container that gets fully recreated on every redeploy,
114+
// not just restarted in place -- both leading in-process hypotheses were investigated and ruled out with
115+
// real production evidence: the reentrancy-guard overlap-skip added in #instrument-drain-skip (see
116+
// withOrbRelayDrainReentrancyGuard's own header comment) never fired once in 14 days of live coverage
117+
// (loopover_orb_relay_drain_skipped_total stayed at 0), and the genuine in-process timeout errors
118+
// (LOOPOVER-1Y/1Q, kind: orb_relay_drain) total only ~13 occurrences -- nowhere near enough to explain
119+
// 1600+ missed check-ins. What DOES correlate: this container's own StartedAt resets on every redeploy
120+
// (unlike its sidecars', which stay up for days), and boot involves an image pull + up to a 30s Postgres-
121+
// readiness retry loop (waitForPostgres, server.ts) + app init BEFORE this drain loop's first tick can
122+
// even register -- routinely exceeding the old 2-minute margin on a fleet that redeploys multiple times a
123+
// day. 6 minutes gives a normal single redeploy cycle room to complete without a false "outage" issue,
124+
// while a genuinely stuck/crash-looping instance still alerts within ~3 x (1+6) = ~21 minutes
125+
// (failureIssueThreshold below is unchanged -- only the margin, which is what the evidence points at).
126+
checkinMargin: 6,
113127
maxRuntime: 1,
114128
failureIssueThreshold: 3,
115129
recoveryThreshold: 1,

test/unit/selfhost-sentry.test.ts

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -816,6 +816,30 @@ describe("enabled when SENTRY_DSN is set", () => {
816816
expect(mocks.captureException).not.toHaveBeenCalled();
817817
});
818818

819+
it("gives orb-relay-drain a 6-minute checkin margin so a normal redeploy cycle doesn't false-positive as a missed check-in (LOOPOVER-12 regression)", async () => {
820+
await initSentry({
821+
SENTRY_DSN: "d",
822+
SENTRY_ENVIRONMENT: "prod",
823+
} as unknown as NodeJS.ProcessEnv);
824+
825+
await expect(
826+
withSentryMonitor("orb-relay-drain", { jobType: "orb-relay-drain" }, async () => "ok"),
827+
).resolves.toBe("ok");
828+
829+
expect(mocks.captureCheckIn).toHaveBeenNthCalledWith(
830+
1,
831+
{ monitorSlug: "loopover-selfhost-prod-orb-relay-drain", status: "in_progress" },
832+
expect.objectContaining({
833+
schedule: { type: "interval", value: 1, unit: "minute" },
834+
checkinMargin: 6,
835+
maxRuntime: 1,
836+
failureIssueThreshold: 3,
837+
recoveryThreshold: 1,
838+
}),
839+
);
840+
expect(mocks.captureException).not.toHaveBeenCalled();
841+
});
842+
819843
it("records failed Sentry cron monitor check-ins with sanitized context", async () => {
820844
await initSentry({
821845
SENTRY_DSN: "d",

0 commit comments

Comments
 (0)