docs(ops): add runbooks for rate-limit, cache, Qdrant, Orb, and AI provider recovery

[JSONbored]

Advances #1943

Investigation

Before implementing, I audited the current state of every deliverable category in the issue (Grafana dashboards, alert rules, emitted metrics, existing ops/troubleshooting docs) against the actual code, rather than assuming the issue's framing was still accurate:

• Alert rules: already comprehensive — 13 groups, ~21 rules, covering queue backlog, dead-letter growth, GitHub rate-limits, Postgres, backups, Qdrant, Orb export, HTTP latency/5xx, webhook enqueue failures, and (as of this session's fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker #2540) AI provider circuit-breaker/inconclusive-review alerts.
• Dashboards: 6 dashboards already cover queue, jobs, GitHub cache/rate-limits, Postgres, HTTP, and AI usage.
• Metrics: well-instrumented (24 counters, 3 gauges, 1 histogram), no significant drift between docs and actual incr()/gauge()/observe() call sites.
• Docs: docs.self-hosting-operations.tsx and docs.self-hosting-troubleshooting.tsx cover the basics (health endpoints, log events, alerting setup, routine checks) but had no runbook for five specific failure modes an operator can actually hit.

Given the infrastructure was already substantially complete, this PR is scoped to the genuine gap: five new troubleshooting runbooks.

Changes

Adds to docs.self-hosting-troubleshooting.tsx, each grounded in metric names/labels/alert names verified directly against the source (not inferred):

• GitHub rate-limit responses and admission deferrals — the kind/key_scope/job_type label vocabulary, PromQL to break a spike down by token pool, and how to distinguish an expected brief admission hold from a sustained problem.
• Low GitHub response-cache hit rate — the result/class labels and a hit-rate PromQL query.
• Qdrant/vector-store errors — the op label, the embedding-model-change dimension-mismatch scenario, and how to recreate a collection.
• Orb export/relay reconciliation — pointing at the export loop's Sentry cron monitor as the fastest first check before digging into error counters.
• AI provider circuit breaker (shipped this session in fix(selfhost): validate AI reviewer-provider configuration and add a failure circuit breaker #2540) — what it does, that it self-heals in 60s with no manual reset, and how to tell a transient trip from a persistent credential/reachability problem.

Test plan

• npm run ui:lint — 0 errors (pre-existing warnings in other files unrelated to this change)
• npm run ui:typecheck clean
• npm run ui:test — 33/33 passing
• npm run ui:build succeeds
• Root npm run typecheck clean
• npm audit --audit-level=moderate — 0 vulnerabilities
• Verified in a live preview: navigated to /docs/self-hosting-troubleshooting, confirmed all 5 new sections render with correct heading text, code blocks, and TOC entries

[loopover-orb]

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

_{Review updated: 2026-07-02 23:26:23 UTC}

3 files · 1 AI reviewer · no blockers · readiness 91/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR adds concrete self-host troubleshooting runbooks and updates the docs sidebar to keep the larger self-hosting section navigable. The changed React/route code is structurally sound, and the new drift test provides a useful guard that referenced metric and alert names still exist. The main concern is maintainability/scope: a runbook-only PR also changes the exported docs navigation shape, which is fine inside this file but worth keeping tightly justified because downstream consumers may have assumed flat groups.

Nits — 5 non-blocking

• nit: apps/gittensory-ui/src/components/site/docs-nav.tsx:6 adds a long policy comment about information architecture that reads more like PR rationale than code documentation; trim it to the structural invariant or move the reasoning to docs/PR notes.
• nit: apps/gittensory-ui/src/routes/docs.self-hosting-troubleshooting.tsx:219 shows a Qdrant delete command without the optional API-key header even though the preceding bullet tells operators to verify QDRANT_API_KEY, so authenticated deployments need an adjusted command.
• nit: test/unit/docs-selfhost-troubleshooting-metric-names.test.ts:16 hard-codes the metric source file list, so a future metric moved to a new emitter file could make the docs guard fail even though the metric still exists in source.
• apps/gittensory-ui/src/routes/docs.self-hosting-troubleshooting.tsx:219: include the authenticated Qdrant command variant, e.g. `curl -H "api-key: $QDRANT_API_KEY" ...`, or explicitly say the shown command is for unauthenticated local Qdrant.
• test/unit/docs-selfhost-troubleshooting-metric-names.test.ts:16: consider deriving metric sources with a repository glob over `src/**/*.ts` or centralizing the emitter list so this drift guard does not become its own stale inventory.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:L; no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 64 registered-repo PR(s), 55 merged, 522 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 64 PR(s), 522 issue(s).
Gate result	✅ Passing	No configured blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 64 PR(s), 522 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Explain no-issue PR.
• No action.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	cc404f7	Commit Preview URL Branch Preview URL	Jul 02 2026, 11:13 PM

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.06%. Comparing base (62262af) to head (cc404f7).
⚠️ Report is 12 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2638   +/-   ##
=======================================
  Coverage   96.06%   96.06%           
=======================================
  Files         234      234           
  Lines       26305    26305           
  Branches     9538     9538           
=======================================
  Hits        25270    25270           
  Misses        425      425           
  Partials      610      610           

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.