feat(selfhost): defer maintenance work under runtime pressure

Adds a claim-time admission policy so periodic maintenance sweeps
(contributor evidence, RAG indexing, drift scans, product rollups,
notifications...) always yield to live webhook/review work and to
host CPU pressure, instead of competing with it on equal footing.
Denied jobs are deferred with jitter and force-admitted via a
trickle ceiling so sustained load can slow maintenance down but
never starve it forever.

Also: classifies a bare 403 on the branch-protection probe as a
permission gap (not GitHub rate-limit exhaustion) with its own
metric, adds Grafana panels + Prometheus alerts for live vs.
maintenance queue pressure, documents the three self-host runner
deployment modes, and adds a safe Docker build-cache/image prune
script (never touches volumes).

Author: JSONbored

.env.example

@@ -259,6 +259,22 @@ REDIS_URL=redis://redis:6379                # REQUIRED for the self-host review
 #                                            #   per-batch jobs above this so the per-login GitHub reads spread across the
 #                                            #   queue instead of bursting. Set 0 to disable the fan-out (single job).
 
+# --- Maintenance-job backpressure (#selfhost-runtime-pressure) ---
+# Periodic maintenance sweeps (contributor evidence, burden forecasts, RAG re-indexing, drift scans, product
+# rollups, notifications...) already yield to an exhausted GitHub REST budget; these knobs add an orthogonal
+# check at queue-claim time -- is the BOX itself under load right now -- so maintenance never competes with
+# live webhook/review work for CPU/DB time. A denied maintenance job is deferred with jitter, never dropped;
+# TRICKLE (MAINTENANCE_ADMISSION_MAX_DEFER_AGE_MS) force-admits it once it's waited long enough regardless of
+# pressure, so sustained load can slow maintenance down but never starve it forever. All defaults are sane for
+# a small single-node box; every value is optional.
+# MAINTENANCE_ADMISSION_ENABLED=true         # set false/0/off to fully disable this policy (old always-run behavior)
+# MAINTENANCE_ADMISSION_MAX_LIVE_PENDING=5   # defer maintenance once this many live (webhook/regate) jobs are queued
+# MAINTENANCE_ADMISSION_MAX_LIVE_AGE_MS=120000     # defer maintenance once the oldest live job has waited this long (2m)
+# MAINTENANCE_ADMISSION_MAX_PENDING=15       # defer NEW maintenance admissions once this many maintenance jobs are already queued
+# MAINTENANCE_ADMISSION_MAX_HOST_LOAD=1.5    # defer once 1-min load average per CPU core exceeds this (best-effort; see host-pressure.ts)
+# MAINTENANCE_ADMISSION_DEFER_MS=180000      # base defer duration on denial, before jitter (3m)
+# MAINTENANCE_ADMISSION_MAX_DEFER_AGE_MS=14400000  # trickle ceiling: force-admit a maintenance job that has waited this long (4h)
+
 # --- Caddy HTTPS terminator (#1203; requires --profile caddy) ---
 # DOMAIN=gittensory.example.com             # fully-qualified domain; Caddy auto-obtains a Let's Encrypt cert
 
@@ -268,6 +284,18 @@ REDIS_URL=redis://redis:6379                # REQUIRED for the self-host review
 # TS_EXTRA_ARGS=                             # extra tailscale up flags, e.g. --advertise-tags=tag:self-host
 
 # --- Self-hosted GitHub Actions runner (#1205; requires --profile runners) ---
+# NOT part of the default recommended stack -- it is an OPTIONAL profile. Pick a deployment mode:
+#   1. App-only VPS (recommended default): gittensory + Postgres/Redis/Qdrant/observability, no runners here.
+#      Use GitHub-hosted CI for your own repos, or a separate runner host (below).
+#   2. Separate runner host: run `--profile runners` on a DIFFERENT machine from the review stack. CI load
+#      never contends with the app for CPU/disk/network.
+#   3. GitHub-hosted CI: simplest and safest for small teams -- no runner host to manage at all.
+# CI runners sharing a VPS with the review stack CAN starve reviews: a burst of CI jobs (vitest coverage,
+# wrangler builds, migration checks...) will happily consume every spare CPU cycle the app needs to process
+# webhooks and AI review calls, discovered running exactly this pattern in production -- 3 uncapped runner
+# containers on an 8-vCPU box left the app starved under load. If you DO need runners on this same host
+# (mode 2 above, but co-located anyway, or you're testing this locally), see docker-compose.override.yml.example
+# for a proven CPU-priority pattern (cpu_shares + a per-container cpus ceiling) before scaling replicas up.
 # Blank tokens/URLs are valid until --profile runners is enabled.
 # RUNNER_TOKEN=                              # runner registration token (Settings → Actions → Runners → New)
 # RUNNER_REPO_URL=https://github.com/org/repo
@@ -276,6 +304,12 @@ REDIS_URL=redis://redis:6379                # REQUIRED for the self-host review
 # RUNNER_NAME=gittensory-runner
 # RUNNER_LABELS=self-hosted,linux
 
+# --- Docker disk hygiene (#selfhost-runtime-pressure) ---
+# Build cache and unused images accumulate fast on a box that builds from source or runs CI runners; a root
+# disk over ~80-85% full slows down the WHOLE host (fsync latency, container scheduling), not just Docker.
+# ./scripts/docker-prune.sh reports usage read-only by default; pass --yes to actually reclaim space (never
+# touches volumes/application data). See the script's header for a cron example.
+
 # --- Observability: metrics + alerts + logs (#1206; requires --profile observability) ---
 # The observability profile starts Prometheus (scrapes /metrics) + Alertmanager (alert rules in
 # prometheus/rules/, routing in alertmanager/alertmanager.yml — silent until you fill in a receiver) +

apps/gittensory-ui/src/lib/selfhost-env-reference.ts

@@ -11,11 +11,11 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "AI_EMBED_API_KEY",
-    firstReference: "src/server.ts:417",
+    firstReference: "src/server.ts:419",
   },
   {
     name: "AI_EMBED_BASE_URL",
-    firstReference: "src/server.ts:414",
+    firstReference: "src/server.ts:416",
   },
   {
     name: "AI_EMBED_MODEL",
@@ -43,7 +43,7 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "BACKUP_ACKNOWLEDGED",
-    firstReference: "src/server.ts:356",
+    firstReference: "src/server.ts:358",
   },
   {
     name: "BROWSER_WS_ENDPOINT",
@@ -75,11 +75,11 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "CRON_INTERVAL_MS",
-    firstReference: "src/server.ts:819",
+    firstReference: "src/server.ts:839",
   },
   {
     name: "DATABASE_PATH",
-    firstReference: "src/server.ts:239",
+    firstReference: "src/server.ts:241",
   },
   {
     name: "DATABASE_URL",
@@ -103,19 +103,23 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "GITHUB_CACHE_TTL_SECONDS",
-    firstReference: "src/server.ts:485",
+    firstReference: "src/server.ts:487",
   },
   {
     name: "GITTENSORY_REPO_CONFIG_DIR",
-    firstReference: "src/server.ts:273",
+    firstReference: "src/server.ts:275",
   },
   {
     name: "GITTENSORY_VERSION",
     firstReference: "src/selfhost/health.ts:29",
   },
+  {
+    name: "MAINTENANCE_ADMISSION_ENABLED",
+    firstReference: "src/selfhost/maintenance-admission.ts:83",
+  },
   {
     name: "MIGRATIONS_DIR",
-    firstReference: "src/server.ts:369",
+    firstReference: "src/server.ts:371",
   },
   {
     name: "OBSERVABILITY_SMOKE_POLL_MS",
@@ -175,7 +179,7 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "ORB_BROKER_URL",
-    firstReference: "src/server.ts:863",
+    firstReference: "src/server.ts:883",
   },
   {
     name: "ORB_COLLECTOR_TOKEN",
@@ -191,7 +195,7 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "ORB_RELAY_MODE",
-    firstReference: "src/server.ts:865",
+    firstReference: "src/server.ts:885",
   },
   {
     name: "OTEL_EXPORTER_OTLP_ENDPOINT",
@@ -223,11 +227,11 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "PGVECTOR_ENABLED",
-    firstReference: "src/server.ts:219",
+    firstReference: "src/server.ts:221",
   },
   {
     name: "PORT",
-    firstReference: "src/server.ts:623",
+    firstReference: "src/server.ts:643",
   },
   {
     name: "PUBLIC_API_ORIGIN",
@@ -243,7 +247,7 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "QDRANT_URL",
-    firstReference: "src/server.ts:504",
+    firstReference: "src/server.ts:506",
   },
   {
     name: "QUEUE_BACKGROUND_CONCURRENCY",
@@ -255,7 +259,7 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "REVIEW_AUDIT_DIR",
-    firstReference: "src/server.ts:549",
+    firstReference: "src/server.ts:551",
   },
   {
     name: "SELFHOST_BUNDLE_ALL",
@@ -291,41 +295,42 @@ export const SELFHOST_ENV_REFERENCE_ROWS: SelfHostEnvReferenceRow[] = [
   },
   {
     name: "SETUP_OUTPUT_PATH",
-    firstReference: "src/server.ts:740",
+    firstReference: "src/server.ts:760",
   },
 ];
 
 export const SELFHOST_ENV_REFERENCE_MARKDOWN = [
   "| Name | First reference |",
   "| --- | --- |",
   "| `AI_COMBINE` | `src/selfhost/ai.ts:848` |",
-  "| `AI_EMBED_API_KEY` | `src/server.ts:417` |",
-  "| `AI_EMBED_BASE_URL` | `src/server.ts:414` |",
+  "| `AI_EMBED_API_KEY` | `src/server.ts:419` |",
+  "| `AI_EMBED_BASE_URL` | `src/server.ts:416` |",
   "| `AI_EMBED_MODEL` | `src/selfhost/ai.ts:744` |",
   "| `AI_ON_MERGE` | `src/selfhost/ai.ts:850` |",
   "| `AI_PROVIDER` | `src/selfhost/ai-config.ts:43` |",
   "| `ANTHROPIC_AI_BASE_URL` | `src/selfhost/ai.ts:748` |",
   "| `ANTHROPIC_AI_MODEL` | `src/selfhost/ai.ts:57` |",
   "| `ANTHROPIC_API_KEY` | `src/selfhost/ai.ts:747` |",
-  "| `BACKUP_ACKNOWLEDGED` | `src/server.ts:356` |",
+  "| `BACKUP_ACKNOWLEDGED` | `src/server.ts:358` |",
   "| `BROWSER_WS_ENDPOINT` | `src/selfhost/stubs/puppeteer.ts:11` |",
   "| `CLAUDE_AI_EFFORT` | `src/selfhost/ai.ts:108` |",
   "| `CLAUDE_AI_MODEL` | `src/selfhost/ai.ts:49` |",
   "| `CLAUDE_AI_TIMEOUT_MS` | `src/selfhost/ai.ts:108` |",
   "| `CODEX_AI_EFFORT` | `src/selfhost/ai.ts:112` |",
   "| `CODEX_AI_MODEL` | `src/selfhost/ai.ts:53` |",
   "| `CODEX_AI_TIMEOUT_MS` | `src/selfhost/ai.ts:112` |",
-  "| `CRON_INTERVAL_MS` | `src/server.ts:819` |",
-  "| `DATABASE_PATH` | `src/server.ts:239` |",
+  "| `CRON_INTERVAL_MS` | `src/server.ts:839` |",
+  "| `DATABASE_PATH` | `src/server.ts:241` |",
   "| `DATABASE_URL` | `src/selfhost/preflight.ts:201` |",
   "| `DISCORD_REPO_WEBHOOKS` | `src/selfhost/discord-notify.ts:31` |",
   "| `DISCORD_WEBHOOK_URL` | `src/selfhost/discord-notify.ts:40` |",
   "| `GITHUB_APP_ID` | `src/selfhost/orb-collector.ts:59` |",
   "| `GITHUB_APP_PRIVATE_KEY` | `src/selfhost/orb-collector.ts:166` |",
-  "| `GITHUB_CACHE_TTL_SECONDS` | `src/server.ts:485` |",
-  "| `GITTENSORY_REPO_CONFIG_DIR` | `src/server.ts:273` |",
+  "| `GITHUB_CACHE_TTL_SECONDS` | `src/server.ts:487` |",
+  "| `GITTENSORY_REPO_CONFIG_DIR` | `src/server.ts:275` |",
   "| `GITTENSORY_VERSION` | `src/selfhost/health.ts:29` |",
-  "| `MIGRATIONS_DIR` | `src/server.ts:369` |",
+  "| `MAINTENANCE_ADMISSION_ENABLED` | `src/selfhost/maintenance-admission.ts:83` |",
+  "| `MIGRATIONS_DIR` | `src/server.ts:371` |",
   "| `OBSERVABILITY_SMOKE_POLL_MS` | `scripts/smoke-observability-traces.mjs:8` |",
   "| `OBSERVABILITY_SMOKE_TIMEOUT_MS` | `scripts/smoke-observability-traces.mjs:6` |",
   "| `OLLAMA_AI_API_KEY` | `src/selfhost/ai.ts:741` |",
@@ -340,27 +345,27 @@ export const SELFHOST_ENV_REFERENCE_MARKDOWN = [
   "| `ORB_AIR_GAP` | `src/selfhost/orb-collector.ts:161` |",
   "| `ORB_ANONYMIZE` | `src/selfhost/orb-collector.ts:174` |",
   "| `ORB_APP_ID` | `src/selfhost/orb-collector.ts:59` |",
-  "| `ORB_BROKER_URL` | `src/server.ts:863` |",
+  "| `ORB_BROKER_URL` | `src/server.ts:883` |",
   "| `ORB_COLLECTOR_TOKEN` | `src/selfhost/orb-collector.ts:205` |",
   "| `ORB_COLLECTOR_URL` | `src/selfhost/orb-collector.ts:172` |",
   "| `ORB_ENROLLMENT_SECRET` | `src/selfhost/orb-collector.ts:165` |",
-  "| `ORB_RELAY_MODE` | `src/server.ts:865` |",
+  "| `ORB_RELAY_MODE` | `src/server.ts:885` |",
   "| `OTEL_EXPORTER_OTLP_ENDPOINT` | `src/selfhost/otel.ts:47` |",
   "| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | `src/selfhost/otel.ts:45` |",
   "| `OTEL_SERVICE_ENVIRONMENT` | `src/selfhost/otel.ts:60` |",
   "| `OTEL_SERVICE_NAME` | `src/selfhost/otel.ts:59` |",
   "| `OTEL_TRACES_EXPORTER` | `src/selfhost/otel.ts:40` |",
   "| `OTEL_TRACES_SAMPLER` | `src/selfhost/otel.ts:74` |",
   "| `OTEL_TRACES_SAMPLER_ARG` | `src/selfhost/otel.ts:76` |",
-  "| `PGVECTOR_ENABLED` | `src/server.ts:219` |",
-  "| `PORT` | `src/server.ts:623` |",
+  "| `PGVECTOR_ENABLED` | `src/server.ts:221` |",
+  "| `PORT` | `src/server.ts:643` |",
   "| `PUBLIC_API_ORIGIN` | `src/selfhost/preflight.ts:192` |",
   "| `QDRANT_API_KEY` | `src/selfhost/qdrant-vectorize.ts:50` |",
   "| `QDRANT_DIM` | `src/selfhost/qdrant-vectorize.ts:71` |",
-  "| `QDRANT_URL` | `src/server.ts:504` |",
+  "| `QDRANT_URL` | `src/server.ts:506` |",
   "| `QUEUE_BACKGROUND_CONCURRENCY` | `src/selfhost/queue-common.ts:102` |",
   "| `REDIS_URL` | `src/selfhost/preflight.ts:144` |",
-  "| `REVIEW_AUDIT_DIR` | `src/server.ts:549` |",
+  "| `REVIEW_AUDIT_DIR` | `src/server.ts:551` |",
   "| `SELFHOST_BUNDLE_ALL` | `scripts/build-selfhost.mjs:13` |",
   "| `SELFHOST_SERVICE` | `scripts/smoke-observability-traces.mjs:5` |",
   "| `SELFHOST_SETUP_TOKEN` | `src/selfhost/preflight.ts:186` |",
@@ -369,5 +374,5 @@ export const SELFHOST_ENV_REFERENCE_MARKDOWN = [
   "| `SENTRY_RELEASE` | `src/selfhost/otel.ts:62` |",
   "| `SENTRY_SERVER_NAME` | `src/selfhost/sentry.ts:373` |",
   "| `SENTRY_TRACES_SAMPLE_RATE` | `src/selfhost/sentry.ts:161` |",
-  "| `SETUP_OUTPUT_PATH` | `src/server.ts:740` |",
+  "| `SETUP_OUTPUT_PATH` | `src/server.ts:760` |",
 ].join("\n");

docker-compose.override.yml.example

@@ -25,6 +25,15 @@
 # strongly you want review work to win contention (8:1 below is deliberately aggressive), and keep each
 # runner replica's `cpus` ceiling at roughly (host vCPUs) / (number of runner replicas you plan to run),
 # so a single busy runner can't monopolize the box even before shares kick in.
+#
+# This is the HOST-level (Docker CPU scheduling) half of the fix. The APPLICATION-level half is the
+# MAINTENANCE_ADMISSION_* knobs in .env.example: gittensory itself defers its own maintenance sweeps
+# (contributor evidence, RAG indexing, drift scans...) under queue/host pressure so live webhook/review
+# work always wins there too. Also see ./scripts/docker-prune.sh for the disk-usage side of running CI
+# builds on the same box (build cache/image growth, not just CPU).
+# The RECOMMENDED alternative to all of this: don't co-locate runners with the review stack at all -- use
+# GitHub-hosted CI, or a separate dedicated runner host (see the runner service's comment in
+# docker-compose.yml).
 
 services:
   gittensory:

docker-compose.yml

@@ -609,12 +609,17 @@ services:
     network_mode: host # Tailscale needs host networking to advertise the host's address
 
   # ── Self-hosted GitHub Actions runner (--profile runners) ─────────────────
+  # NOT part of the default recommended stack -- purely optional. Three deployment modes (see .env.example
+  # for the full explanation): (1) app-only VPS + GitHub-hosted CI (recommended default, no runner here at
+  # all), (2) a separate dedicated runner host, (3) co-located runners on this SAME box (this profile).
   # Runs `runs-on: self-hosted` jobs on this machine. Set RUNNER_TOKEN= (or ACCESS_TOKEN=)
   # and RUNNER_REPO_URL= (e.g. https://github.com/your-org/your-repo) in .env.
   # Get a token at: https://github.com/<owner>/<repo>/settings/actions/runners/new
   # Running this alongside the main `gittensory` app on the SAME host? A burst of CI jobs can starve the
-  # app of CPU with no limits set (the default below). See docker-compose.override.yml.example for a
-  # proven CPU-priority pattern (relative cpu_shares + a per-container cpus ceiling) before scaling this up.
+  # app of CPU with no limits set (the default below) -- confirmed in production: 3 uncapped runner
+  # containers on an 8-vCPU box left the app starved under load. See docker-compose.override.yml.example
+  # for a proven CPU-priority pattern (relative cpu_shares + a per-container cpus ceiling) before scaling
+  # this up, and ./scripts/docker-prune.sh for reclaiming the disk space CI builds accumulate.
   runner:
     # Pin to a specific version tag — `latest` is mutable. Find a digest via:
     # docker pull myoung34/github-runner:ubuntu-jammy && docker inspect --format='{{index .RepoDigests 0}}' myoung34/github-runner:ubuntu-jammy