Skip to content

ci: use remote ccache only, drop the duplicate local cache - #77

Merged
igorpecovnik merged 1 commit into
mainfrom
perf/ccache-remote-only
Sep 27, 2026
Merged

igorpecovnik merged 1 commit into
mainfrom
perf/ccache-remote-only

Conversation

@igorpecovnik

Copy link
Copy Markdown
Member

Builds already point ccache at the shared Redis — CCACHE_REMOTE_STORAGE is set per-runner by armbian/actions runner-clean from github-runners.jq, and passed to compile.sh at line 143 along with ENABLE_EXTENSIONS=ccache-remote.

The local ccache directory is then a second copy of the same objects, written to the disk that is the builders' actual constraint.

Measured

On rack-ryzen under the normal nightly load:

local ccache growth +44 MB per 30 s, per runner
kernel worktree growth +194 MB per 30 s, per runner
NVMe write latency 2161 ms average, sustained
queue depth 178
write throughput 5.6 MB/s at 82 IOPS
PSI io full avg10=53.7% vs cpu some avg10=12.3%

The box is not short of CPU; it cannot retire writes. The local cache is ~20% of each runner's write rate and buys nothing once a shared remote exists.

Why here and not in runner-clean

docker.sh forwards a fixed env allowlist into the build container (CCACHE_DIR and CPUTHREADS are on it; CCACHE_* is not). An env var exported to $GITHUB_ENV never reaches the build — which is exactly why CCACHE_REMOTE_STORAGE is passed as a compile.sh argument rather than relying on the export. CCACHE_REMOTE_ONLY has to travel the same way.

It's set inside the existing if [ -n "${CCACHE_REMOTE_STORAGE}" ] guard, so a build with no remote keeps its local cache untouched.

Trade-off

There is no local fallback. If Redis is unreachable the cache is simply absent for that build and objects recompile. CCACHE_REDIS_CONNECT_TIMEOUT is 500 ms, so that costs lookups rather than stalling. Builder and Redis (10.0.40.2) are on the same LAN.

CCACHE_REMOTE_ONLY is already supported by the ccache-remote extension (it's in CCACHE_PASSTHROUGH_VARS, and documented in the extension header as "Disable local cache, use remote only (saves local disk space)") and by ccache 4.9.1 on the runners. It was simply never passed.

Not changed

build-images-chunk.yml exports CCACHE_REMOTE_STORAGE to $GITHUB_ENV (line 138) but never builds a CCACHE_PARAMS array or passes anything to compile.sh — so image builds use no ccache at all today and this flag would be inert there. Kernel compiles happen in the artifacts path. Worth a separate look at whether that export is meant to be wired up.

@github-actions github-actions Bot added 11 Milestone: Fourth quarter release size/small PR with less then 50 lines GitHub Actions GitHub Actions code Needs review Seeking for review labels Sep 20, 2026
@coderabbitai

coderabbitai Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 13c96f49-9c91-47fc-8023-c84476cacb0d

📥 Commits

Reviewing files that changed from the base of the PR and between 514ab51 and 0054ef0.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: aa7a00f6-d9b4-4f7c-aec3-dbb1b1bbf744

📥 Commits

Reviewing files that changed from the base of the PR and between 2e0b2af and 514ab51.

📒 Files selected for processing (1)
  • .github/workflows/build-artifacts-chunk.yml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

When CCACHE_REMOTE_STORAGE is non-empty, the workflow adds CCACHE_REMOTE_ONLY=yes to CCACHE_PARAMS. Builds using remote ccache storage therefore skip the local ccache directory. Existing remote storage and extension parameters remain unchanged.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: using remote ccache only and removing the duplicate local cache.
Description check ✅ Passed The description directly explains the remote-only ccache change, its rationale, measured impact, trade-offs, and unchanged behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@igorpecovnik igorpecovnik removed the Needs review Seeking for review label Sep 20, 2026
@igorpecovnik
igorpecovnik requested a review from iav September 20, 2026 03:04
@iav

iav commented Sep 21, 2026

Copy link
Copy Markdown

Mechanism checks out: CCACHE_REMOTE_ONLY is in CCACHE_PASSTHROUGH_VARS, so it reaches both
the docker --env and the env -i kernel/u-boot make env, and ccache 4.9.1 reads "yes" as true.

Three corrections to the text:

  1. CCACHE_REDIS_CONNECT_TIMEOUT is the extension's own variable, not a ccache option — ccache
    ignores it, and the URLs in github-runners.jq carry no attributes. So the real bounds are
    ccache's defaults: 100 ms connect, 10000 ms operation, not 500 ms.
  2. CCACHE_REMOTE_ONLY=no is not an off switch: ccache 4.9.1 errors with invalid boolean environment variable value "no" and exits 1, i.e. the build fails. Off means dropping the
    param or CCACHE_NOREMOTE_ONLY=true — worth a line in the code comment.
  3. The flag turns on for 5 runners across three Redis endpoints (10.0.40.2, 10.128.132.124,
    193.40.103.61). For kspace the "same LAN" assumption isn't visible from the map, and it
    loses its local cache entirely.

Builds already point ccache at the shared Redis (CCACHE_REMOTE_STORAGE, set
per-runner by armbian/actions runner-clean). The local ccache directory is then
a second copy of the same objects, written to the disk that is the builders'
actual constraint: measured ~44 MB per 30 s per runner, against a device
showing 2.2 s average write latency and queue depth 178 under the nightly load.

CCACHE_REMOTE_ONLY is already supported by the ccache-remote extension and by
ccache 4.9.1 on the runners; it just was never passed. It has to go here rather
than in runner-clean's $GITHUB_ENV export, because docker.sh forwards a fixed
env allowlist into the build container and CCACHE_* is not on it -- which is
why CCACHE_REMOTE_STORAGE is passed as a compile.sh argument too.

Set inside the existing 'remote storage is configured' guard, so a build
without a remote keeps its local cache untouched.

Trade-off, stated because it is real: there is no local fallback. If Redis is
unreachable the cache is simply absent for that build and objects recompile.
CCACHE_REDIS_CONNECT_TIMEOUT is 500 ms, so that costs lookups, not a stall.

build-images-chunk.yml needs no change: it exports CCACHE_REMOTE_STORAGE to
$GITHUB_ENV but never passes ccache params to compile.sh, so image builds use
no ccache at all today. Kernel compiles happen in the artifacts path.

Signed-off-by: Igor Pecovnik <igor@armbian.com>
@igorpecovnik
igorpecovnik force-pushed the perf/ccache-remote-only branch from 514ab51 to 0054ef0 Compare September 26, 2026 08:55
@github-actions github-actions Bot added the Needs review Seeking for review label Sep 26, 2026

@iav iav left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. One fix for the code comment: 500 ms is the extension's CCACHE_REDIS_CONNECT_TIMEOUT, which only applies to endpoints it discovers via DNS-SD; with the URL from github-runners.jq ccache uses its own 100 ms connect timeout.

@github-actions

Copy link
Copy Markdown

✅ This PR has been reviewed and approved — all set for merge!

@github-actions github-actions Bot added Ready to merge Reviewed, tested and ready for merge and removed Needs review Seeking for review labels Sep 27, 2026
@igorpecovnik

Copy link
Copy Markdown
Member Author

lets try live

@igorpecovnik
igorpecovnik merged commit 59d46ed into main Sep 27, 2026
6 checks passed
@igorpecovnik
igorpecovnik deleted the perf/ccache-remote-only branch September 27, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

11 Milestone: Fourth quarter release GitHub Actions GitHub Actions code Ready to merge Reviewed, tested and ready for merge size/small PR with less then 50 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants