Skip to content

fix(health): probe GitHub App auth and AI-provider reachability in /ready #2497

Description

@JSONbored

Parent: #1936

Problem

/ready (readiness probes built in src/server.ts) only pushes redis and, conditionally, qdrant into readinessProbes. If the GitHub App private key is invalid/expired or every configured AI provider is unreachable (bad API key, CLI missing auth, provider outage), the review pipeline is completely dead, but GET /ready still returns 200 {ok:true}readiness() in src/selfhost/health.ts only checks DB + migrations + registered probes.

Any orchestrator, load balancer, or uptime check that trusts /ready (its own doc comment says it's for "a multi-instance load balancer should stop routing to it") will keep routing traffic to — and an operator will believe is healthy — an instance that reviews nothing. Detection currently requires SENTRY_DSN to be set (structured-log forwarding) or manually grepping stdout for selfhost_ai_provider_failed / GitHub App auth errors; self-hosters without Sentry have zero automated signal.

Fix

Add lightweight readiness probes for:

  • GitHub App auth — e.g. attempt/verify the cached-or-freshly-minted App JWT via getAppJwt(env) (or createAppJwt) with a short timeout.
  • The configured AI provider — a cheap reachability/auth check per provider, or track "last N AI calls all failed" and gate readiness on that.

Wrap each the same way the existing Redis/Qdrant probes are (withTimeout + push into readinessProbes).

Verification

  • Confirm current /ready probe list against src/server.ts (line numbers may have shifted).
  • A test that stubs a rejected GitHub App JWT mint and confirms /ready now reports unhealthy.
  • A test that stubs every AI provider failing and confirms /ready now reports unhealthy.
  • Confirm no added latency/cost regression to /ready under normal healthy operation (cache the check result briefly if needed).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions