perf(selfhost): document retention/concurrency sizing, fix stale runner docs (#1828)

Prometheus's 180d retention default had no sizing rationale and no comment,
unlike the explicitly-justified 336h/14d (Loki) and 168h/7d (Tempo) siblings;
git history shows it silently replaced a deliberate 30d default when it
became an override-able env var in #1678. Add a comment justifying why
metrics can reasonably outlive logs/traces (far cheaper per-day TSDB
storage) while lowering the default to 90d, a considered middle ground
rather than an unexplained six-month default.

QUEUE_CONCURRENCY/QUEUE_BACKGROUND_CONCURRENCY had no core-count-based
sizing guidance, unlike PGPOOL_MAX's existing "raise if you see X, watch Y"
treatment. Add equivalent guidance anchored to vCPU count.

The self-hosting-operations docs claimed the runner service "ships with no
CPU/memory limit at all" in two places; this became false once
RUNNER_MEM_LIMIT (default 2g) was added for #3893. Fix both occurrences,
plus the adjacent table cell making the same now-incorrect claim, to
describe the current state: a default memory ceiling with CPU-priority
pinning remaining opt-in via docker-compose.override.yml.example.

Author: JSONbored

.env.example

@@ -298,11 +298,21 @@ REDIS_URL=redis://redis:6379                # REQUIRED for the self-host review
 #                                            #   a moderate-load instance (several active repos / a steady
 #                                            #   contributor-PR stream) on a host with a few spare cores -- review
 #                                            #   jobs are I/O-bound (GitHub + AI awaits dominate), so raising this
-#                                            #   mostly buys parallelism, not CPU. Watch gittensory_queue_live_pending
+#                                            #   mostly buys parallelism, not CPU. As a core-count anchor (#1828):
+#                                            #   a 2-vCPU host is comfortable around the default of 4; a 4+ vCPU
+#                                            #   host can reasonably go to 8-12 before the loops' own CPU work
+#                                            #   (JSON parsing, diff rendering, AI-response handling) starts
+#                                            #   competing for the cycles the I/O awaits were supposed to free up.
+#                                            #   Watch gittensory_queue_live_pending
 #                                            #   / gittensory_queue_oldest_live_pending_age_seconds after raising it;
 #                                            #   if those stay high, the bottleneck is elsewhere (GitHub rate limit,
 #                                            #   AI latency, Postgres pool -- see PGPOOL_MAX above), not concurrency.
 # QUEUE_BACKGROUND_CONCURRENCY=1             # max low-priority/background jobs allowed to occupy QUEUE_CONCURRENCY slots
+#                                            #   (default 1 reserves the rest of even a small 2-vCPU host's default
+#                                            #   4 slots for live review work). Scale it WITH QUEUE_CONCURRENCY, not
+#                                            #   independently -- keep it to roughly a quarter to a third of
+#                                            #   QUEUE_CONCURRENCY (e.g. 2-3 of a 4+ vCPU host's 8-12) so background
+#                                            #   maintenance can never crowd out every live-review slot.
 # CONTRIBUTOR_EVIDENCE_BATCH_SIZE=150        # logins per build-contributor-evidence job; the scheduled run fans out into
 #                                            #   per-batch jobs above this so the per-login GitHub reads spread across the
 #                                            #   queue instead of bursting. Set 0 to disable the fan-out (single job).

apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx

@@ -328,16 +328,20 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
                 + <code>--profile runners</code>
               </td>
               <td className="py-2 pr-4 align-top text-muted-foreground">
-                Unbounded by default — can starve the app under CI load
+                Unbounded by default — can still starve the app under CI load
+              </td>
+              <td className="py-2 pr-4 align-top text-muted-foreground">
+                Bounded by <code>RUNNER_MEM_LIMIT</code> (default 2g) per replica
               </td>
-              <td className="py-2 pr-4 align-top text-muted-foreground">Unbounded by default</td>
               <td className="py-2 align-top text-muted-foreground">
-                Estimated, and explicitly a known risk, not a guess about typical usage: the{" "}
-                <code>runner</code> service ships with no CPU/memory limit at all. Production
-                experience already documented in <code>docker-compose.override.yml.example</code>{" "}
-                found 3 uncapped runner containers starving the app for CPU on an 8-vCPU box under
-                real CI load — see that file for the <code>cpu_shares</code>/<code>cpus</code>{" "}
-                mitigation before co-locating runners with the review stack.
+                Estimated, and explicitly a known risk on the CPU side, not a guess about typical
+                usage: the <code>runner</code> service ships with a default memory ceiling (
+                <code>RUNNER_MEM_LIMIT</code>, default 2g, added by #3893) but no CPU limit.
+                Production experience already documented in{" "}
+                <code>docker-compose.override.yml.example</code> found 3 uncapped runner containers
+                starving the app for CPU on an 8-vCPU box under real CI load — see that file for the{" "}
+                <code>cpu_shares</code>/<code>cpus</code> mitigation before co-locating runners with
+                the review stack.
               </td>
             </tr>
             <tr>
@@ -402,10 +406,11 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
         and still has real headroom), and nothing is so oversized relative to plausible usage that
         it should be lowered — including Ollama&apos;s comparatively large 8GiB ceiling, which is
         sized for holding one quantized model in memory, not idle overhead. The one real gap is{" "}
-        <code>--profile runners</code>, which ships with no limit at all; that is a known,
-        documented tradeoff (see the table above and{" "}
+        <code>--profile runners</code>&apos;s CPU side: the service has a default memory ceiling (
+        <code>RUNNER_MEM_LIMIT</code>, default 2g) but ships with no CPU limit at all; that is a
+        known, documented tradeoff (see the table above and{" "}
         <code>docker-compose.override.yml.example</code>) rather than an oversight, since the right
-        ceiling depends entirely on the host's core count and how many runner replicas you run.
+        CPU ceiling depends entirely on the host's core count and how many runner replicas you run.
       </p>
 
       <h3>Capacity planning: how much disk for N repos at M PRs/month</h3>

docker-compose.yml

@@ -510,7 +510,15 @@ services:
     command:
       - "--config.file=/etc/prometheus/prometheus.yml"
       - "--storage.tsdb.path=/prometheus"
-      - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-180d}"
+      # 90d, not the 336h/14d of Loki or 168h/7d of Tempo below (#1828): that gap is deliberate, not an
+      # oversight — Prometheus's TSDB compresses numeric samples to ~1-2 bytes each after compaction, so
+      # metrics cost far less disk per retained day than Loki's raw log lines or Tempo's full span trees,
+      # and the self-hosting-operations docs' own capacity-planning guidance wants a multi-week/month
+      # trend window for metrics specifically. That said, 180d (six months) had no sizing rationale of
+      # its own — it silently replaced a deliberate 30d default when this became an override-able var in
+      # #1678 — so 90d is the considered number here: a full quarter of history for trend/capacity
+      # review, without defaulting a small self-host box to open-ended, unbounded-looking retention.
+      - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-90d}"
     deploy:
       resources:
         limits: