Skip to content

fix(selfhost): widen orb-relay-drain checkin margin to match redeploy cadence - #7385

Merged
JSONbored merged 1 commit into
mainfrom
claude/merge-queue-vs-orb-research-ab9cfa
Jul 20, 2026
Merged

fix(selfhost): widen orb-relay-drain checkin margin to match redeploy cadence#7385
JSONbored merged 1 commit into
mainfrom
claude/merge-queue-vs-orb-research-ab9cfa

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Sentry monitor LOOPOVER-12 fired 1,600+ "missed check-in" outages over 2+ weeks on orb-relay-drain. Ruled out the reentrancy-guard overlap-skip (0 occurrences across 14 days of live instrumentation) and in-process timeouts (~13 events, far too few to explain the volume).
  • Correlates instead with the self-host container's redeploy cadence: its StartedAt resets on every redeploy (unlike stable sidecars), and boot (image pull + Postgres-readiness wait + init) routinely exceeds the old 2-minute checkinMargin on a fleet redeploying multiple times a day.
  • Raises orb-relay-drain's checkinMargin from 2 to 6 minutes so a normal redeploy doesn't trip a false outage, while a genuinely stuck instance still alerts within ~21 minutes (failureIssueThreshold unchanged).

Validation

  • npx vitest run test/unit/selfhost-sentry.test.ts test/unit/selfhost-monitored-work.test.ts — 109/109 passing
  • npx tsc --noEmit — clean
  • Added a regression test locking in the new checkinMargin value

Safety

  • Config-only change to a Sentry monitor threshold; no behavioral change to the drain loop itself, no new write paths, no secrets touched.

@JSONbored JSONbored self-assigned this Jul 20, 2026
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.17%. Comparing base (47b34e9) to head (f20bf92).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7385      +/-   ##
==========================================
- Coverage   91.18%   91.17%   -0.01%     
==========================================
  Files         716      716              
  Lines       72697    72697              
  Branches    20423    20423              
==========================================
- Hits        66288    66283       -5     
- Misses       5367     5368       +1     
- Partials     1042     1046       +4     
Flag Coverage Δ
shard-1 35.92% <ø> (-0.01%) ⬇️
shard-2 41.41% <ø> (-0.09%) ⬇️
shard-3 39.03% <ø> (-0.01%) ⬇️
shard-4 39.93% <ø> (ø)
shard-5 27.18% <ø> (ø)
shard-6 31.31% <ø> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/selfhost/sentry.ts 99.31% <ø> (ø)

... and 2 files with indirect coverage changes

… cadence

LOOPOVER-12 fired 1600+ missed-checkin outages over 2+ weeks. Ruled out the
reentrancy-guard overlap-skip (0 occurrences in 14 days of live
instrumentation) and in-process timeouts (~13 events, too few to explain the
volume). The container's own StartedAt resets on every redeploy unlike its
stable sidecars, and boot (image pull + Postgres-readiness wait + init)
routinely exceeds the old 2-minute margin on a fleet redeploying multiple
times a day. Raise checkinMargin from 2 to 6 minutes so a normal redeploy
doesn't trip a false outage, while a genuinely stuck instance still alerts
within ~21 minutes.
@JSONbored
JSONbored force-pushed the claude/merge-queue-vs-orb-research-ab9cfa branch from e9a3b34 to f20bf92 Compare July 20, 2026 03:17
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 20, 2026
@loopover-orb

loopover-orb Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Warning

⏸️ LoopOver review result - manual review recommended

Review updated: 2026-07-20 03:26:08 UTC

2 files · 1 AI reviewer · 2 blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
This change is a straightforward config-only tweak raising orb-relay-drain's checkinMargin from 2 to 6 minutes to accommodate self-host container redeploy boot time (image pull + Postgres-readiness wait + init), with a well-documented rationale citing production evidence that ruled out the reentrancy-guard and in-process-timeout hypotheses. The regression test correctly locks in the new value via the captureCheckIn mock assertion, and failureIssueThreshold is left unchanged as stated so the effective alert window becomes ~21 minutes. The diff is small, well-scoped, and the description matches the code exactly.

Nits — 4 non-blocking
  • src/selfhost/sentry.ts:126 — the checkinMargin value 6 is a bare numeric literal inline; a named constant (e.g. ORB_RELAY_DRAIN_CHECKIN_MARGIN_MINUTES) would make the derivation (redeploy boot budget) easier to tune later, though the comment already documents the reasoning thoroughly.
  • The new test in test/unit/selfhost-sentry.test.ts largely duplicates the surrounding test structure (initSentry + withSentryMonitor + captureCheckIn assertion) rather than reusing a shared helper, but this matches the existing file's convention so it's a minor consistency nit only.
  • Consider extracting a named constant for the 6-minute margin in src/selfhost/sentry.ts so future threshold tuning doesn't require parsing the long comment to find the literal.
  • If other self-host monitors share the same redeploy-boot-time exposure, it may be worth auditing their checkinMargin values in a follow-up rather than this PR.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
📋 Copy for AI agents — paste into your coding agent
Fix the following blocker(s) from this PR review:

1. No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue — Link the relevant issue (for example `Closes #123`) before opening the PR.

Decision drivers

  • ❌ Code review — 2 blockers (1 reviewer)
  • ❌ Gate result — Blocking (Repo-configured hard blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 28 registered-repo PR(s), 21 merged, 259 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 28 PR(s), 259 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, JavaScript, MDX, Shell, Solidity
  • Official Gittensor activity: 28 PR(s), 259 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Then work through the remaining 2 steps in the Signals table above.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 20, 2026
@JSONbored
JSONbored merged commit 7760fc9 into main Jul 20, 2026
18 checks passed
@JSONbored
JSONbored deleted the claude/merge-queue-vs-orb-research-ab9cfa branch July 20, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant