From e16e74f4ecaec33c2f658053d47b6470a6c7cc24 Mon Sep 17 00:00:00 2001
From: JSONbored <49853598+JSONbored@users.noreply.github.com>
Date: Wed, 8 Jul 2026 00:37:23 -0700
Subject: [PATCH] perf(selfhost): document retention/concurrency sizing, fix
stale runner docs (#1828)
Prometheus's 180d retention default had no sizing rationale and no comment,
unlike the explicitly-justified 336h/14d (Loki) and 168h/7d (Tempo) siblings;
git history shows it silently replaced a deliberate 30d default when it
became an override-able env var in #1678. Add a comment justifying why
metrics can reasonably outlive logs/traces (far cheaper per-day TSDB
storage) while lowering the default to 90d, a considered middle ground
rather than an unexplained six-month default.
QUEUE_CONCURRENCY/QUEUE_BACKGROUND_CONCURRENCY had no core-count-based
sizing guidance, unlike PGPOOL_MAX's existing "raise if you see X, watch Y"
treatment. Add equivalent guidance anchored to vCPU count.
The self-hosting-operations docs claimed the runner service "ships with no
CPU/memory limit at all" in two places; this became false once
RUNNER_MEM_LIMIT (default 2g) was added for #3893. Fix both occurrences,
plus the adjacent table cell making the same now-incorrect claim, to
describe the current state: a default memory ceiling with CPU-priority
pinning remaining opt-in via docker-compose.override.yml.example.
---
.env.example | 12 ++++++++-
.../routes/docs.self-hosting-operations.tsx | 27 +++++++++++--------
docker-compose.yml | 10 ++++++-
3 files changed, 36 insertions(+), 13 deletions(-)
diff --git a/.env.example b/.env.example
index 8c35898486..b59476ffcd 100644
--- a/.env.example
+++ b/.env.example
@@ -298,11 +298,21 @@ REDIS_URL=redis://redis:6379 # REQUIRED for the self-host review
# # a moderate-load instance (several active repos / a steady
# # contributor-PR stream) on a host with a few spare cores -- review
# # jobs are I/O-bound (GitHub + AI awaits dominate), so raising this
-# # mostly buys parallelism, not CPU. Watch gittensory_queue_live_pending
+# # mostly buys parallelism, not CPU. As a core-count anchor (#1828):
+# # a 2-vCPU host is comfortable around the default of 4; a 4+ vCPU
+# # host can reasonably go to 8-12 before the loops' own CPU work
+# # (JSON parsing, diff rendering, AI-response handling) starts
+# # competing for the cycles the I/O awaits were supposed to free up.
+# # Watch gittensory_queue_live_pending
# # / gittensory_queue_oldest_live_pending_age_seconds after raising it;
# # if those stay high, the bottleneck is elsewhere (GitHub rate limit,
# # AI latency, Postgres pool -- see PGPOOL_MAX above), not concurrency.
# QUEUE_BACKGROUND_CONCURRENCY=1 # max low-priority/background jobs allowed to occupy QUEUE_CONCURRENCY slots
+# # (default 1 reserves the rest of even a small 2-vCPU host's default
+# # 4 slots for live review work). Scale it WITH QUEUE_CONCURRENCY, not
+# # independently -- keep it to roughly a quarter to a third of
+# # QUEUE_CONCURRENCY (e.g. 2-3 of a 4+ vCPU host's 8-12) so background
+# # maintenance can never crowd out every live-review slot.
# CONTRIBUTOR_EVIDENCE_BATCH_SIZE=150 # logins per build-contributor-evidence job; the scheduled run fans out into
# # per-batch jobs above this so the per-login GitHub reads spread across the
# # queue instead of bursting. Set 0 to disable the fan-out (single job).
diff --git a/apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx b/apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
index e5dc7c6248..0b1edb6517 100644
--- a/apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
+++ b/apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
@@ -328,16 +328,20 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
+ --profile runners
- Unbounded by default — can starve the app under CI load
+ Unbounded by default — can still starve the app under CI load
+ |
+
+ Bounded by RUNNER_MEM_LIMIT (default 2g) per replica
|
- Unbounded by default |
- Estimated, and explicitly a known risk, not a guess about typical usage: the{" "}
- runner service ships with no CPU/memory limit at all. Production
- experience already documented in docker-compose.override.yml.example{" "}
- found 3 uncapped runner containers starving the app for CPU on an 8-vCPU box under
- real CI load — see that file for the cpu_shares/cpus{" "}
- mitigation before co-locating runners with the review stack.
+ Estimated, and explicitly a known risk on the CPU side, not a guess about typical
+ usage: the runner service ships with a default memory ceiling (
+ RUNNER_MEM_LIMIT, default 2g, added by #3893) but no CPU limit.
+ Production experience already documented in{" "}
+ docker-compose.override.yml.example found 3 uncapped runner containers
+ starving the app for CPU on an 8-vCPU box under real CI load — see that file for the{" "}
+ cpu_shares/cpus mitigation before co-locating runners with
+ the review stack.
|
@@ -402,10 +406,11 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
and still has real headroom), and nothing is so oversized relative to plausible usage that
it should be lowered — including Ollama's comparatively large 8GiB ceiling, which is
sized for holding one quantized model in memory, not idle overhead. The one real gap is{" "}
- --profile runners, which ships with no limit at all; that is a known,
- documented tradeoff (see the table above and{" "}
+ --profile runners's CPU side: the service has a default memory ceiling (
+ RUNNER_MEM_LIMIT, default 2g) but ships with no CPU limit at all; that is a
+ known, documented tradeoff (see the table above and{" "}
docker-compose.override.yml.example) rather than an oversight, since the right
- ceiling depends entirely on the host's core count and how many runner replicas you run.
+ CPU ceiling depends entirely on the host's core count and how many runner replicas you run.
Capacity planning: how much disk for N repos at M PRs/month
diff --git a/docker-compose.yml b/docker-compose.yml
index 48ce389266..3b63c174be 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -510,7 +510,15 @@ services:
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-180d}"
+ # 90d, not the 336h/14d of Loki or 168h/7d of Tempo below (#1828): that gap is deliberate, not an
+ # oversight — Prometheus's TSDB compresses numeric samples to ~1-2 bytes each after compaction, so
+ # metrics cost far less disk per retained day than Loki's raw log lines or Tempo's full span trees,
+ # and the self-hosting-operations docs' own capacity-planning guidance wants a multi-week/month
+ # trend window for metrics specifically. That said, 180d (six months) had no sizing rationale of
+ # its own — it silently replaced a deliberate 30d default when this became an override-able var in
+ # #1678 — so 90d is the considered number here: a full quarter of history for trend/capacity
+ # review, without defaulting a small self-host box to open-ended, unbounded-looking retention.
+ - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-90d}"
deploy:
resources:
limits: