Skip to content

fix(selfhost): keep runner temp files off docker overlay#2720

Merged
JSONbored merged 1 commit into
mainfrom
fix/selfhost-runner-tmpdir-overlay
Jul 3, 2026
Merged

fix(selfhost): keep runner temp files off docker overlay#2720
JSONbored merged 1 commit into
mainfrom
fix/selfhost-runner-tmpdir-overlay

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Problem: the self-hosted GitHub Actions runner service in docker-compose.yml left TMPDIR/TMP/TEMP unset, so CI job temp writes (language toolchain caches, build artifacts, ad hoc mktemp calls) landed in the container's plain /tmp, which lives in Docker's overlay/containerd snapshot storage — not the mounted runner-work volume. This grows the host's Docker root storage directly and is invisible to volume-scoped cleanup, causing unbounded overlay growth, disk pressure, and docker system df races against disappearing temp files under sustained CI load.
  • Fix: the runner service now sets TMPDIR, TMP, and TEMP to /tmp/runner/tmp — a subdirectory of the existing mounted runner-work:/tmp/runner volume — via a reusable x-runner-tmp-env YAML anchor. RUNNER_WORKDIR stays on the same mounted volume as before.
  • Guaranteed to exist: a new one-shot runner-tmp-init service (alpine:3.20, mkdir -p + world-writable chmod) bootstraps that directory on the volume before the runner container starts, gated via depends_on: condition: service_completed_successfully. This is idempotent (safe on every docker compose up) and never touches the runner image's own entrypoint/registration flow — it's a pure Compose-level dependency, not an image wrapper.
  • Multi-runner scalability: the temp-env pattern is centralized in the x-runner-tmp-env anchor. docker-compose.override.yml.example and the self-hosting operations docs both document that additional runner services (for higher CI throughput) must repeat this same pattern — including the YAML-anchors-don't-cross-files caveat, since an override file can't reference the base file's anchor directly.
  • No behavior changes to: RUNNER_WORKDIR placement, Docker socket exposure (still never mounted), or volume-pruning behavior (none introduced; the init step only creates a directory, never deletes anything).

Why

Confirmed in production: runner containers wrote high-volume CI temp files into the container default /tmp, causing Docker root storage growth and disk pressure. The live mitigation was setting TMPDIR=/tmp/runner/tmp by hand per-container; this PR makes that the shipped default so a fresh self-host deploy is correct without any manual step.

Validation

  • npm run typecheck
  • docker compose --profile runners -f docker-compose.yml config --quiet (also verified with every profile active simultaneously)
  • npm run ui:lint / npm run ui:typecheck (docs page change)
  • New tests: test/unit/selfhost-compose-runner-tmpdir.test.ts (13 cases — volume mount, RUNNER_WORKDIR, TMPDIR/TMP/TEMP all set and pointed at mounted storage not raw /tmp, no Docker-socket mount, no volume-deleting behavior in the init step, runners-profile gating, and that the runner service actually merges the x-runner-tmp-env anchor rather than a hand-duplicated copy that could drift) and test/unit/docs-selfhost-operations-runner-tmpdir.test.ts (6 cases — drift guard so the docs' multi-runner snippet can't silently diverge from the real compose pattern, mirroring the existing docs-selfhost-troubleshooting-metric-names.test.ts source-of-truth-diff approach).
  • npm run test:ci (full local gate) — 7077 tests passed, fully green.
  • npm audit --audit-level=moderate — 0 vulnerabilities.
  • No src/** files touched by this PR, so Codecov's patch-coverage gate has nothing to measure here; the new regression tests are pure structural/parity checks against docker-compose.yml and the docs page.

Notes

While investigating this, found that an earlier PR (#2717) had already added a different, overlapping Docker-prune script (scripts/docker-prune.sh) without noticing scripts/selfhost-docker-prune.sh (systemd-timer-integrated, docs-referenced, tested) already existed from PR #2576. That's an unrelated cleanup item, tracked separately — not addressed in this PR to keep it focused on the runner-tmpdir fix.

Points TMPDIR/TMP/TEMP for the self-hosted GitHub Actions runner
service at a subdirectory of its mounted runner-work volume instead
of the container's plain /tmp, which lives in Docker's overlay/
containerd snapshot storage and grows the host's Docker root
storage directly (invisible to volume-scoped cleanup).

A one-shot runner-tmp-init service bootstraps that directory on the
volume (mkdir -p + world-writable chmod) before the runner starts,
gated on depends_on: condition: service_completed_successfully, so
this works on a fresh deploy with no manual steps and never touches
the runner image's own entrypoint. The temp-env values live in a
reusable x-runner-tmp-env anchor so additional runner services in a
multi-runner override follow the same pattern.

Also documents why runner temp storage must stay off overlay, a
copy-pasteable multi-runner override snippet, and safe Docker
cleanup guidance (containers/images/build cache only, never
volumes) in the self-hosting operations docs.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 3018a99 Commit Preview URL

Branch Preview URL
Jul 03 2026, 07:57 AM

@JSONbored JSONbored self-assigned this Jul 3, 2026
@JSONbored
JSONbored merged commit f14e487 into main Jul 3, 2026
11 checks passed
@JSONbored
JSONbored deleted the fix/selfhost-runner-tmpdir-overlay branch July 3, 2026 08:03
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.03%. Comparing base (a6f4a36) to head (3018a99).
⚠️ Report is 16 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2720      +/-   ##
==========================================
- Coverage   96.13%   96.03%   -0.11%     
==========================================
  Files         241      244       +3     
  Lines       27033    27271     +238     
  Branches     9822     9909      +87     
==========================================
+ Hits        25988    26189     +201     
- Misses        433      457      +24     
- Partials      612      625      +13     

see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant