-
-
Notifications
You must be signed in to change notification settings - Fork 74
Expand file tree
/
Copy pathdocker-compose.override.yml.example
More file actions
55 lines (53 loc) · 3.91 KB
/
Copy pathdocker-compose.override.yml.example
File metadata and controls
55 lines (53 loc) · 3.91 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
# Optional CPU-priority tuning for boxes running `--profile runners` alongside the main `loopover`
# app on the SAME host. Copy to `docker-compose.override.yml` (gitignored, auto-loaded by `docker
# compose` alongside docker-compose.yml) and adjust the numbers for your host if you use this pattern.
#
# WHY THIS EXISTS: with no limits set (the default -- see docker-compose.yml), every container competes
# for the host's CPUs on equal footing. On a box that ALSO runs `--profile runners`, a burst of CI jobs
# can starve the `loopover` app itself of CPU, slowing down (or queuing up) the very review work the
# app exists to do -- discovered running this exact pattern in production on an 8-vCPU box: 3 uncapped
# runner containers left the app starved under load.
#
# THE FIX IS PRIORITY, NOT A HARD SPLIT: `cpu_shares` is a RELATIVE weight that only matters once the
# host is actually contended (all CPUs busy at once) -- a job still gets 100% of an idle box regardless
# of its share. Setting the app's shares high and the runner's low means: engine work always wins a race
# for CPU when both want it at the same instant, but CI is never blocked from running merely because the
# app exists -- it just yields faster than it otherwise would under real contention.
#
# `cpus` is a separate, absolute ceiling (not a reservation) -- it caps how many cores a SINGLE container
# can burst to, independent of shares. Size it to your own host: these examples assume an 8-vCPU box
# running the app + 3 runner replicas; the runner values below intentionally sum to MORE than the host's
# core count (12 > 8) because `cpus` is a per-container ceiling, not a guarantee -- the host's own CPU
# scheduler (driven by the relative `cpu_shares` below) resolves real contention when multiple containers
# actually want the CPU at once.
#
# Sizing guide for a different host: pick a `loopover` : `runner` cpu_shares ratio reflecting how
# strongly you want review work to win contention (8:1 below is deliberately aggressive), and keep each
# runner replica's `cpus` ceiling at roughly (host vCPUs) / (number of runner replicas you plan to run),
# so a single busy runner can't monopolize the box even before shares kick in.
#
# This is the HOST-level (Docker CPU scheduling) half of the fix. The APPLICATION-level half is the
# MAINTENANCE_ADMISSION_* knobs in .env.example: loopover itself defers its own maintenance sweeps
# (contributor evidence, RAG indexing, drift scans...) under queue/host pressure so live webhook/review
# work always wins there too. Also see ./scripts/selfhost-docker-prune.sh (and its systemd timer under
# systemd/loopover-docker-prune.*.example) for the disk-usage side of running CI builds on the same box
# (build cache/image growth, not just CPU).
# The RECOMMENDED alternative to all of this: don't co-locate runners with the review stack at all -- use
# GitHub-hosted CI, or a separate dedicated runner host (see the runner service's comment in
# docker-compose.yml).
#
# SCALING TO MULTIPLE RUNNERS: adding a second/third runner service here (e.g. to raise CI throughput)?
# Each one MUST set the same TMPDIR/TMP/TEMP env (pointed at its own mounted runner-work-style volume,
# never left at the container's plain /tmp) and mount its own named volume at /tmp/runner -- otherwise its
# CI job temp writes fall back to the container's overlay storage (the exact growth/disk-pressure problem
# this repo's runner service is set up to avoid). YAML anchors don't cross separate compose files, so you
# cannot reference docker-compose.yml's `x-runner-tmp-env` anchor from this file directly -- either repeat
# the three env vars literally, or redeclare the SAME `x-runner-tmp-env: &runner-tmp-env` extension block
# at the top of your own override file and merge it with `<<: *runner-tmp-env`. See the self-hosting
# operations docs for a complete, copy-pasteable multi-runner service snippet.
services:
loopover:
cpu_shares: 4096
runner:
cpu_shares: 512
cpus: "4.0"