feat(selfhost): add automatic dead-letter job retry for self-host queues

[JSONbored]

Summary

Closes #2534.

Neither self-host queue backend (SQLite or Postgres) has a way to move a job out of status='dead' once it lands there. The Cloudflare Workers Queues DLQ redrive path (src/queue/dlq.ts) is wired exclusively to that platform's native queue consumer and is not reachable from the self-host tables. In practice this means: a job that dies from a transient bug, or a bug that's since been fixed and redeployed, requires a manual database UPDATE/DELETE to ever run again — indefinitely.

What changed

• Added reviveDeadLetterJobs() to both createSqliteQueue and createPgQueue: requeues dead jobs whose lifetime attempts is still under a ceiling (maxRetries + a small configurable extra-attempts budget, default 3), clearing last_error. attempts itself is left untouched, so a revived job gets exactly one more attempt before either succeeding or dying again — not a fresh full retry budget — bounding how many times a permanently-broken job can cycle before it stops being revived and requires manual intervention.
• Runs on a separate, much slower timer (default 30 minutes, QUEUE_DEAD_LETTER_REVIVE_INTERVAL_MS) than the normal poll tick — that interval is the cooldown between auto-retry rounds, so no separate timestamp bookkeeping was needed. Reuses the existing attempts/job_key columns rather than a parallel table, per the issue's requirement.
• New gittensory_jobs_dead_letter_revived_total metric, distinct from the existing crash-recovery gittensory_jobs_recovered_total counter (a different scenario — process-restart recovery vs. dead-letter revival).
• reviveDeadLetterJobs() is exposed on the public queue interface (not just wired to the internal timer), so it's directly testable without waiting on real intervals, and available for a future operator-triggered repair path.
• Documented the new behavior (and both env-var knobs) in the self-hosting Operations doc.

Correctness/safety notes

• No retry-storm risk: the revive interval bounds how often any one job can be retried, and the attempts < ceiling filter bounds how many total revivals a job can get before it's left dead permanently.
• A job that dies again after revival returns to normal dead-letter handling (including the sibling alerting work in fix(selfhost): close the dead-letter alerting blind spot and require alerting setup #2549) — this feature only gives a bounded number of extra chances, it never suppresses the existing dead-letter signal.

Validation

• Full local gate (npm run test:ci) green, npm audit --audit-level=moderate clean, git diff --check clean.
• Every new/changed line and branch across sqlite-queue.ts, pg-queue.ts, and queue-common.ts is covered — cross-checked the diff hunks directly against coverage/lcov.info; the handful of uncovered lines/branches in those files are all pre-existing and unrelated to this change.
• New tests cover: revival under the ceiling with last_error cleared, the ceiling actually stopping further revival, no-op when nothing is dead, the periodic timer firing a real end-to-end revival (SQLite, fake timers), the equivalent mocked-pool behavior for Postgres, and both new env-var config knobs (defaults + overrides).

Safety

• No secrets or credentials introduced or exposed.
• No behavior change to existing retry/dead-letter thresholds — purely additive.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	ae2472b	Commit Preview URL Branch Preview URL	Jul 02 2026, 11:50 AM

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-02 11:52:16 UTC}

7 files · no blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

• The AI review could not be completed for this change, so the gate is held for a human reviewer rather than passed automatically. It re-evaluates on the next update.

Review summary
AI review is already running for this PR head in another Gittensory pass. Gittensory is holding this PR for manual review until that pass completes.

Nits — 2 non-blocking

• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• AI review already in progress for this PR head — The gate is held for a human reviewer rather than passed automatically; it re-evaluates once the in-flight review completes or on the next update.

Signal	Result	Evidence
Code review	✅ No blockers	No AI review summary
Linked issue	✅ Linked	#2534
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:L; 1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 65 registered-repo PR(s), 55 merged, 536 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 65 PR(s), 536 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 65 PR(s), 536 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• No action.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.02%. Comparing base (335d9cd) to head (ae2472b).
⚠️ Report is 9 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
+ Coverage   95.98%   96.02%   +0.04%     
==========================================
  Files         229      233       +4     
  Lines       25813    26079     +266     
  Branches     9390     9475      +85     
==========================================
+ Hits        24777    25043     +266     
  Misses        425      425              
  Partials      611      611              

Files with missing lines	Coverage Δ
src/selfhost/queue-common.ts	93.24% <100.00%> (+0.08%)	⬆️
src/selfhost/sqlite-queue.ts	99.30% <100.00%> (+0.04%)	⬆️

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JSONbored]

Resolved the flagged defect: `reviveEligibleDeadJobs`'s SELECT-then-UPDATE is a stale snapshot — an overlapping reviver (a second self-host instance sharing the same Postgres DB, or a slow prior revive tick still running when the next fires) could move a row out of `'dead'` between the SELECT and this row's own UPDATE, and the UPDATE (keyed on id alone) would then blindly flip whatever the row's current status is back to `'pending'` — including a row already claimed into `'processing'`, letting it run a second time concurrently.

Both backends now add `AND status='dead'` to the revive UPDATE, mirroring the identical defensive predicate already used elsewhere in both files for the same class of stale-snapshot sweep (`reclaimExpiredProcessingJobs` / `deferPendingJobsForRateLimit`). The revived count now reflects rows the UPDATE actually matched, not the raw SELECT count.

Added a regression test per backend:

• Postgres: a mock pool with per-call configurable UPDATE `rowCount`s, proving a 0-rowCount UPDATE isn't counted as revived.
• SQLite: a `driver.query` spy that injects the competing status change at the exact point the real UPDATE would otherwise race against it, proving the row is left untouched (not reverted to pending).

Also rebased onto current `main` (two other PRs merged in the interim). Full local gate (`test:ci`, unsharded `test:coverage`, `npm audit`) green; every changed line/branch in `pg-queue.ts`/`sqlite-queue.ts` is covered (cross-checked the remaining uncovered branches directly against the diff — all pre-existing and unrelated).