Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -298,11 +298,21 @@ REDIS_URL=redis://redis:6379 # REQUIRED for the self-host review
# # a moderate-load instance (several active repos / a steady
# # contributor-PR stream) on a host with a few spare cores -- review
# # jobs are I/O-bound (GitHub + AI awaits dominate), so raising this
# # mostly buys parallelism, not CPU. Watch gittensory_queue_live_pending
# # mostly buys parallelism, not CPU. As a core-count anchor (#1828):
# # a 2-vCPU host is comfortable around the default of 4; a 4+ vCPU
# # host can reasonably go to 8-12 before the loops' own CPU work
# # (JSON parsing, diff rendering, AI-response handling) starts
# # competing for the cycles the I/O awaits were supposed to free up.
# # Watch gittensory_queue_live_pending
# # / gittensory_queue_oldest_live_pending_age_seconds after raising it;
# # if those stay high, the bottleneck is elsewhere (GitHub rate limit,
# # AI latency, Postgres pool -- see PGPOOL_MAX above), not concurrency.
# QUEUE_BACKGROUND_CONCURRENCY=1 # max low-priority/background jobs allowed to occupy QUEUE_CONCURRENCY slots
# # (default 1 reserves the rest of even a small 2-vCPU host's default
# # 4 slots for live review work). Scale it WITH QUEUE_CONCURRENCY, not
# # independently -- keep it to roughly a quarter to a third of
# # QUEUE_CONCURRENCY (e.g. 2-3 of a 4+ vCPU host's 8-12) so background
# # maintenance can never crowd out every live-review slot.
# CONTRIBUTOR_EVIDENCE_BATCH_SIZE=150 # logins per build-contributor-evidence job; the scheduled run fans out into
# # per-batch jobs above this so the per-login GitHub reads spread across the
# # queue instead of bursting. Set 0 to disable the fan-out (single job).
Expand Down
27 changes: 16 additions & 11 deletions apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -328,16 +328,20 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
+ <code>--profile runners</code>
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Unbounded by default — can starve the app under CI load
Unbounded by default — can still starve the app under CI load
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">
Bounded by <code>RUNNER_MEM_LIMIT</code> (default 2g) per replica
</td>
<td className="py-2 pr-4 align-top text-muted-foreground">Unbounded by default</td>
<td className="py-2 align-top text-muted-foreground">
Estimated, and explicitly a known risk, not a guess about typical usage: the{" "}
<code>runner</code> service ships with no CPU/memory limit at all. Production
experience already documented in <code>docker-compose.override.yml.example</code>{" "}
found 3 uncapped runner containers starving the app for CPU on an 8-vCPU box under
real CI load — see that file for the <code>cpu_shares</code>/<code>cpus</code>{" "}
mitigation before co-locating runners with the review stack.
Estimated, and explicitly a known risk on the CPU side, not a guess about typical
usage: the <code>runner</code> service ships with a default memory ceiling (
<code>RUNNER_MEM_LIMIT</code>, default 2g, added by #3893) but no CPU limit.
Production experience already documented in{" "}
<code>docker-compose.override.yml.example</code> found 3 uncapped runner containers
starving the app for CPU on an 8-vCPU box under real CI load — see that file for the{" "}
<code>cpu_shares</code>/<code>cpus</code> mitigation before co-locating runners with
the review stack.
</td>
</tr>
<tr>
Expand Down Expand Up @@ -402,10 +406,11 @@ DISCORD_REPO_WEBHOOKS={"owner/repoA":"https://discord.com/api/webhooks/...","own
and still has real headroom), and nothing is so oversized relative to plausible usage that
it should be lowered — including Ollama&apos;s comparatively large 8GiB ceiling, which is
sized for holding one quantized model in memory, not idle overhead. The one real gap is{" "}
<code>--profile runners</code>, which ships with no limit at all; that is a known,
documented tradeoff (see the table above and{" "}
<code>--profile runners</code>&apos;s CPU side: the service has a default memory ceiling (
<code>RUNNER_MEM_LIMIT</code>, default 2g) but ships with no CPU limit at all; that is a
known, documented tradeoff (see the table above and{" "}
<code>docker-compose.override.yml.example</code>) rather than an oversight, since the right
ceiling depends entirely on the host's core count and how many runner replicas you run.
CPU ceiling depends entirely on the host's core count and how many runner replicas you run.
</p>

<h3>Capacity planning: how much disk for N repos at M PRs/month</h3>
Expand Down
10 changes: 9 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -510,7 +510,15 @@ services:
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-180d}"
# 90d, not the 336h/14d of Loki or 168h/7d of Tempo below (#1828): that gap is deliberate, not an
# oversight — Prometheus's TSDB compresses numeric samples to ~1-2 bytes each after compaction, so
# metrics cost far less disk per retained day than Loki's raw log lines or Tempo's full span trees,
# and the self-hosting-operations docs' own capacity-planning guidance wants a multi-week/month
# trend window for metrics specifically. That said, 180d (six months) had no sizing rationale of
# its own — it silently replaced a deliberate 30d default when this became an override-able var in
# #1678 — so 90d is the considered number here: a full quarter of history for trend/capacity
# review, without defaulting a small self-host box to open-ended, unbounded-looking retention.
- "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME:-90d}"
deploy:
resources:
limits:
Expand Down
Loading