CI: SDK smoke has no cargo cache — 3m59s of cold compile for a 1.32s test

[ndizazzo]

Summary

sdk-smoke.yml compiles the entire Rust dependency graph from a cold registry on every run, and separately rebuilds the whole web UI, inside the smoke step. On a recent run this was 3m59s of cargo build + ~18s of UI build to execute a 1.32s test.

The root cause is that .github/workflows/sdk-smoke.yml is the only compiling workflow in the repo with no cargo cache of any kind — no configure-sccache-gha, no Swatinem/rust-cache, no actions/cache on ~/.cargo. This is declared, not accidental: ci/slices.yml:240 sets the sdk slice to "cache_mode": "none".

This is pre-existing on main, not a regression from any open PR.

Evidence

Run 32282107616, job 96175554873 (Linux / Linux SDK smoke / Rust SDK smoke / rust SDK Smoke), step Rust SDK smoke test:

Marker	Value
Updating crates.io index	18:13:43 — cold registry
Crates Downloaded	647
Crates Compiling	613
Finished \test` profile [unoptimized + debuginfo] target(s)`	3m 59s
test result: ok. 1 passed; ...	1.32s
UI build (tsc -b && vite build, 5616 modules)	~18s
Step total	4m41s
Job total	~5m30s

Why the downloaded artifact does not help

Worth stating explicitly, because the job log looks like a cache restore followed by a redundant compile — it isn't.

./.github/actions/restore-smoke-inputs (sdk-smoke.yml:248) downloads the composed product: the release mesh-llm host binary plus its native runtime, staged at target/release/mesh-llm. scripts/ci-sdk-fixture.sh launches that as the live server-under-test and MESHLLM_NATIVE_RUNTIME_ARTIFACT_DIR points the SDK at the prebuilt runtime. It is a runtime input. It populates neither cargo's target/ nor the registry.

The compile is a different build entirely — scripts/ci-rust-sdk-smoke.sh:25:

cargo test -p mesh-llm-ffi --test live_sdk_smoke -- --nocapture

Different crate, and the test profile (unoptimized + debuginfo) rather than the artifact's release. Even if the tarball carried Rust build outputs, none of it would be reusable here. CARGO_INCREMENTAL: 0 is also set, so there is no incremental reuse either.

Design constraints — read before choosing an approach

sdk-smoke.yml must NOT gain a runner_policy / select-ci-runners job. Two contract tests govern this:

• scripts/tests/test_reusable_workflow_runner_trust.py:103 — test_all_eligible_pr_slices_bind_runner_and_cache_policy enumerates every slice required to bind the central runner/cache policy. sdk-smoke.yml is deliberately excluded, and the docstring says why: "credential-bearing smoke consumers and the GPU exception are intentionally outside it." sdk-smoke.yml carries HF_TOKEN.
• scripts/tests/test_reusable_workflow_runner_trust.py:25 — test_credential_smokes_remain_fixed_github_hosted requires sdk-smoke.yml to keep ubuntu-24.04 and never contain depot-ubuntu. Its runs-on (sdk-smoke.yml:92) is hardcoded and must stay hardcoded.

Do not add configure-sccache-gha. That action is titled "Configure baked sccache remote backend" and invokes sccache directly with no install step — it assumes the binary is baked into a runner image. sdk-smoke's rust job runs on a plain hosted ubuntu-24.04 with no container:, where sccache is not present. It would fail at Unable to start baked sccache. (mozilla-actions/sccache-action is the repo's pattern for hosted runners, but see "Recommended approach" — it isn't needed here.)

The allow_native_github_cache gate would be a constant here anyway. select-ci-runners/action.yml:169 defaults it to true and only flips it to false inside the depot_enabled == true branch (:251). Since sdk-smoke is hosted-only by contract, the gate can never be false.

Recommended approach

ci-macos-host-slice.yml:108-116 is the precedent to copy — a non-container hosted runner using Swatinem/rust-cache alone, no sccache. Swatinem/rust-cache caches ~/.cargo/registry, ~/.cargo/git and target/, so it addresses both the 647 downloads and the 613 compiles in one step.

Part 1 — cargo cache (the 3m59s)

In .github/workflows/sdk-smoke.yml:

1. Add CACHE_NAMESPACE: mesh-llm to the workflow-level env: block at :82 (matching ci-macos-host-slice.yml:48).
2. Insert immediately after the dtolnay/rust-toolchain step at :212, gated on the same sdk_kind == 'rust' condition:

      - if: ${{ inputs.sdk_kind == 'rust' }}
        uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 snapshot 2026-03-12
        continue-on-error: true
        with:
          workspaces: . -> target
          cache-bin: "false"
          prefix-key: ${{ env.CACHE_NAMESPACE }}-rust-${{ runner.os }}-${{ hashFiles('.github/cache-version.txt') }}
          shared-key: ci-sdk-smoke-rust
          save-if: ${{ github.ref == 'refs/heads/main' && github.event.inputs.original_event_name != 'pull_request' && github.event.inputs.original_event_name != 'pull_request_target' }}

Use the pinned SHA and # v2 snapshot 2026-03-12 comment verbatim — it is the only Swatinem/rust-cache pin in the repo. .github/cache-version.txt is currently v3.

The save-if expression restricts writes to main, so PRs read a main-seeded cache and never write. This means the first benefit lands only after a main run has populated the cache; the PR that adds this will not speed up its own CI.

3. In ci/slices.yml:240, change the sdk slice from "cache_mode": "none" to "cache_mode": "pr-isolated".

_planned_cache_mode (scripts/plan-ci.py:543) resolves pr-isolated to trusted-readwrite on the main profile, trusted-readonly on manual-full, and leaves it pr-isolated on PR profiles — exactly the semantics above. Note this field is declarative: nothing in .github/workflows/ reads cache_modes except the PR assertion in ci-runner-contract-slice.yml:48 (all(.cache_modes[]; . != "trusted-readwrite")), which pr-isolated satisfies on PR profiles. The actual caching is hand-wired per slice.

Part 2 — UI rebuild (~18s, and it affects all three SDKs)

All three smoke scripts shell out to scripts/package-sdk-console-assets.sh, which runs a full build-ui.sh unless told otherwise:

• scripts/ci-rust-sdk-smoke.sh:13 — --sdk node
• scripts/ci-kotlin-sdk-smoke.sh:13 — --sdk kotlin
• scripts/ci-swift-sdk-smoke.sh:24 — --sdk swift

package-sdk-console-assets.sh already supports --skip-build and --dist DIR, and the lane already builds this exact artifact:

• Linux: ci-ui-dist-linux-${{ github.run_id }} (ci-linux-lane.yml:66)
• macOS: ci-ui-dist-macos-${{ github.run_id }} (ci-macos-lane.yml:77)

The dependency chain guarantees availability — sdk → runtime_product → hosts → ui_artifact — so if the SDK slice runs, the UI artifact exists. ci-macos-host-slice.yml:118-124 is the download-and-verify precedent (actions/download-artifact into crates/mesh-llm-ui/dist, then test -s crates/mesh-llm-ui/dist/index.html).

Implementation: add a ui_artifact_name input to sdk-smoke.yml, thread it from ci-linux-sdk-slice.yml and ci-macos-sdk-slice.yml (and their lane callers), download it into crates/mesh-llm-ui/dist, and pass --skip-build through the smoke scripts.

Part 2 is separable from Part 1 and touches more files. It is reasonable to land Part 1 alone first.

Files to change

Part 1

• .github/workflows/sdk-smoke.yml
• ci/slices.yml
• .agents/skills/manage-ci/references/current-inventory.md (row for sdk-smoke.yml, line 78)
• scripts/tests/test_plan_ci.py if any assertion pins the sdk slice's cache_modes value

Part 2 (additional)

• .github/workflows/ci-linux-sdk-slice.yml, .github/workflows/ci-macos-sdk-slice.yml
• .github/workflows/ci-linux-lane.yml, .github/workflows/ci-macos-lane.yml
• scripts/ci-rust-sdk-smoke.sh, scripts/ci-kotlin-sdk-smoke.sh, scripts/ci-swift-sdk-smoke.sh

Process requirements

.github/AGENTS.md is normative for this change. Before editing, read in full:

1. .agents/skills/manage-ci/SKILL.md
2. .agents/skills/manage-ci/references/current-inventory.md
3. ci/ci.md

The inventory must be updated in the same change. Do not change Depot settings or runner groups.

Validation

• python3 -m unittest discover -s scripts/tests — the full suite, not a scoped run. test_reusable_workflow_runner_trust.py, test_sccache_evidence.py, test_ci_lane_workflows.py and test_plan_ci.py all touch this surface.
• actionlint over the changed workflows.
• Caveat on CI validation: PR CI runs lane workflows pinned to @main, and the plan job checks out ref: default_branch. So the sdk-smoke.yml and ci/slices.yml changes in this PR will not be exercised by that PR's own CI — they go live on merge. Plan for a main run to seed the cache, then measure a follow-up PR to confirm the improvement.

Expected result

Steady state, a warm Swatinem/rust-cache should take the rust SDK smoke step from ~4m41s to well under a minute. Part 2 removes a further ~18s from each of the three SDK smoke jobs.

Open question for review

sdk-smoke.yml's exclusion from the runner/cache policy census is deliberate because it carries HF_TOKEN. Adding a GHA cache to a credential-bearing smoke workflow should get an explicit sign-off even though save-if restricts writes to main and GitHub scopes PR-written caches to the PR ref. Worth a second opinion before merge.

[alektebel]

Happy to pick this up if nobody's on it.

My plan would be to add Swatinem/rust-cache (or a plain actions/cache on ~/.cargo + target/) to the SDK smoke job, keyed on Cargo.lock plus the runner OS/arch, and post before/after wall-clock numbers from my fork.

Since this is labelled spec, is there a caching contract you want followed across jobs (shared key vs per-job, what gets cached for the native-runtime builds), or is a self-contained fix on this one job fine?

[ndizazzo]

Hey @alektebel, thanks for offering! CI has changed a little bit in the last few days, but this issue is still valid.

We probably want a key like:

mesh-llm-sdk-rust-cargo-v1-${os}-${arch}-${target}-${image-digest}-${toolchain-epoch}-${profile/linker}-${recipe-hash}

... and the branch/event should be controlled by the save policy (main is the only producer), so PRs can restore from a stable cache.

For same-run sharing, use artifacts for the built UI/product/runtime and download them through explicit needs edges. The native runtime should remain owned by its existing producer rather than being folded into the Rust Cargo cache. Kotlin bindgen should probably have a separate cache/path/key; Rust target/ should not be shared with Kotlin, Swift, host/runtime, or native-ABI jobs.

Swatinem/rust-cache is probably fine if it can express an exact key and save policy. Otherwise, explicit actions/cache/restore plus actions/cache/save is easier to get working.

If you still wanna take it, feel free to assign it to yourself and tag me on the PR.