Skip to content

ci: cache the cargo registry and build layers in the container jobs - #34

Open
spacedevin wants to merge 1 commit into
mainfrom
ci/cargo-cache
Open

spacedevin wants to merge 1 commit into
mainfrom
ci/cargo-cache

Conversation

@spacedevin

Copy link
Copy Markdown
Member

Motivation

The crates, rom and verify jobs run in ghcr.io/schlopai/chuggie-toolchain, which ships rustc, libmgba and agb-gbafix but no warm build. Every job (92 per full run) started with an empty registry and an empty target dir: crates.io index fetch, a clone of the agb fork, and a from-scratch compile of core/alloc (-Z build-std), agb and everything under it, before the example's own code was touched. Latest main run: the verify grid-demo "Run the verifier" step (build + 4000-frame emulation) is 1m44s of a ~2m50s job.

Change

crates

actions/cache@v6 over the registry (/usr/local/cargo/registry/{index,cache}, /usr/local/cargo/git/db — the image sets CARGO_HOME=/usr/local/cargo, not ~/.cargo) and crates/*/target (these crates are excluded from the root workspace, so each has its own target dir, as scripts/clean.sh documents).

rom and verify: one shared target dir

tish build normally writes each example's build under its own .tish/gba/<pkg>/target, a path no cache key could share across the 90 verify jobs, and CARGO_TARGET_DIR is deliberately scrubbed by tish (it breaks the ELF lookup; scripts/rom.sh documents the trap). tish has an explicit opt-in for exactly this case — TISH_GBA_TARGET_DIR (crates/tish_native/src/build.rs in @tishlang/tish 3.10.3: "share ONE cargo target dir across every example's ROM build … measured at ~45 GB across the fleet") — which points cargo and the ELF lookup at one directory. Both jobs set it to ${{ github.workspace }}/target/gba.

The dependency layer has the same fingerprint whichever example compiled it (same absolute paths for the path deps, same git rev for agb, same sysroot for build-std), so a target dir warmed by grid-demo is warm for every example.

  • rom is the single writer (actions/cache@v6): it always runs, there is exactly one of it, and its post-step save only happens on success.
  • verify jobs restore only (actions/cache/restore@v6) with the same key and paths. 90 jobs saving the same key would be 89 "cache already exists" warnings and 89 archives built for nothing.

Keys

crates-<sha256 rust-toolchain.toml>-<sha256 crates/**/Cargo.lock, .cargo/config.toml, package.json>
gba-target-<sha256 rust-toolchain.toml>-<sha256 .cargo/config.toml, package.json, crates/**/Cargo.toml>

with restore-keys on the <prefix>-<toolchain hash>- part. The nightly pin leads so a restore never crosses a rustc change (cargo would ignore those artifacts, and they would only bloat the next save). .cargo/config.toml carries the agb rev, package.json the exact tish version; package-lock.json is deliberately not in the key, since deps bumps like the deckfile chores would otherwise miss the cache for a change that does not touch the Rust graph.

What this does and does not cache

Cargo fingerprints path dependencies by mtime, and a fresh checkout / npm ci is always newer than the cached fingerprints, so crates/tish-agb, tish-gba-scenepack, tish-agb-sio, tish-gba-game-engine, node_modules/@tishlang/tish/crates/tish_runtime_gba and the game crate itself still compile in every job. What stays warm is everything beneath them: the registry index, the agb fork clone, core/alloc/compiler_builtins from build-std (rust-src in the image has fixed mtimes), agb and its crates.io dependencies, and scenepack's proc-macro dependencies (syn/quote/proc-macro2, host builds). That is the bulk of a cold build, but not all of it.

Caveats, stated plainly

  • Billing is per job rounded up to the minute. A verify job that goes from 2m50s to 2m10s still bills 3 minutes. The real saving depends on where each job's wall time lands relative to the minute boundary; expect the first few runs to tell. Restore cost is one download + untar per job (the archive is probably several hundred MB), so the net is compile time saved minus that.
  • The toolchain image has no zstd, so actions/cache will fall back to gzip in these jobs (save and restore are both in the container, so they agree). Adding zstd to the image's apt list (schlopai/chuggie-toolchain, not this repo) would make both ends faster.
  • Nothing in the repo reads .tish/gba/<pkg>/target after this: the verifiers that reach into .tish/gba/ read src/main.rs and Cargo.toml (checked: grep -rn 'target/' examples/*/verify.sh scripts/*.sh scripts/*.py); check_agb_fork.sh uses cargo metadata, which does not involve the target dir. The two verify.sh that unset CARGO_TARGET_DIR do not touch TISH_GBA_TARGET_DIR.
  • Swatinem/rust-cache was not used: it assumes ~/.cargo and a workspace target/, neither of which is the layout here.

Both files parse (actionlint, YAML load); the one SC2155 note actionlint reports is pre-existing on main.

The toolchain image ships rustc, libmgba and agb-gbafix, but every crates, rom and verify
job still started from an empty CARGO_HOME registry and an empty target dir: the crates.io
index, the agb fork clone and the whole dependency layer (core/alloc from build-std, agb
and everything under it) were downloaded and compiled from scratch, ~92 times per run.

- crates: actions/cache over /usr/local/cargo/{registry/index,registry/cache,git/db} (the
  image's CARGO_HOME is /usr/local/cargo, not ~/.cargo) and crates/*/target — these crates
  are excluded from the root workspace, so each has its own target dir.
- rom + verify: every ROM build in a run now goes through ONE target dir, via tish's own
  TISH_GBA_TARGET_DIR opt-in (the ambient CARGO_TARGET_DIR breaks tish's ELF lookup, which
  is why rom.sh scrubs it). The dependency layer has the same fingerprint whichever example
  built it, so a target dir warmed by grid-demo is warm for all 90. The rom job — always
  present, exactly one of it — is the single writer; the verify jobs restore only, so 89
  jobs do not race to save the same key.

Keys lead with the nightly pin so restore-keys never hand one rustc's artifacts to another,
then hash what fixes the dependency graph (.cargo/config.toml for the agb rev, package.json
for the tish version, the crate manifests / lockfiles). Path dependencies under crates/ and
node_modules are fingerprinted by mtime and a fresh checkout is always newer, so those and
the game crate itself still compile per job; the cache is everything beneath them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant