Skip to content

refactor: improve CI times#1950

Closed
juan518munoz wants to merge 67 commits intonextfrom
jmunoz-optimze-ci-times
Closed

refactor: improve CI times#1950
juan518munoz wants to merge 67 commits intonextfrom
jmunoz-optimze-ci-times

Conversation

@juan518munoz
Copy link
Copy Markdown
Collaborator

@juan518munoz juan518munoz commented Apr 1, 2026

No description provided.

igamigo and others added 30 commits March 12, 2026 00:07
The beta API now takes supported_types as the second argument instead
of using builder methods with_supports_all_types() / with_supported_type().
Bring all non-RPC-retry changes (RC bump, integration tests, web-client
updates, CLI fixes, etc.) into the base branch so the PR diff only
shows the RPC retry feature.
@juan518munoz juan518munoz changed the title ci: add Rust caching to 5 uncached compilation jobs refactor: improve CI times Apr 1, 2026
@juan518munoz
Copy link
Copy Markdown
Collaborator Author

CI Cache Benchmark — Run 1 (all jobs switched to --release)

Compared PR run (25e7986c) against latest next baseline.

Job PR (cached) next (baseline) Delta
Clippy 2:32 7:51 -5:19
Clippy WASM 3:17 3:15 +0:02
Build rust documentation 2:21 1:59 +0:22
Documentation tests 4:51 2:34 +2:17
Build Client for Wasm 6:43 3:41 +3:02
Build Client and CLI (control) 4:36 4:44 -0:08
Unit tests (control) 15:35 15:42 -0:07

Findings: Clippy is the big winner (-5:19) because rust-release cache has all release dependencies pre-compiled. doc-tests and build-wasm regressed because switching from debug → release adds optimization overhead that outweighs the cache benefit.

Next step: Keep --release only for clippy, revert the rest.

Switch clippy to --release so it can reuse the existing rust-release
cache (saved by the build job). On the next branch this reduced clippy
from ~8 min to ~2.5 min.

Note: on this branch clippy uses +nightly, so the stable release cache
won't have matching compiled artifacts — but registry/git downloads
will still be cached. The full speedup will materialize once toolchains
are aligned.
@juan518munoz juan518munoz changed the base branch from next to release/v0.14.0-beta April 1, 2026 17:23
@juan518munoz juan518munoz force-pushed the jmunoz-optimze-ci-times branch from 0453ba0 to df22f01 Compare April 1, 2026 17:23
@juan518munoz
Copy link
Copy Markdown
Collaborator Author

CI Cache Benchmark — Run 2 (rebased on release/v0.14.0-beta, clippy-only --release)

Compared against baseline run on release/v0.14.0-beta (68672550).

Lint workflow (the target)

Job PR Baseline Delta
Clippy [target] 9:50 8:16 +1:34
Clippy WASM 3:09 3:11 -0:02
Build rust documentation 2:07 2:06 +0:01
Wasm bindgen types 9:44 10:10 -0:26

Test workflow

Job PR Baseline Delta
Unit tests 41:10 39:59 +1:11
Documentation tests 2:58 2:50 +0:08
Integration tests 54:12 43:17 +10:55

Build workflow

Job PR Baseline Delta
Build Client and CLI 20:15 19:59 +0:16
Build Client for Wasm 5:45 5:30 +0:15

Analysis

Clippy is 1:34 slower, not faster. On this branch clippy uses +nightly while the rust-release cache was built with stable — the compiled artifacts are incompatible. Adding --release just added optimization overhead with no cache benefit.

The integration test variance (+10:55) is unrelated to our changes — likely caused by binary cache misses (node-builder: 0:05 → 15:51 due to Cargo.lock difference) cascading into delayed dependent jobs.

Conclusion: This optimization only works when clippy and build use the same toolchain (as on next). On release/v0.14.0-beta where clippy uses nightly, we'd need either a nightly-specific cache or to align the clippy toolchain with stable.

@igamigo igamigo added the no changelog This PR does not require an entry in the `CHANGELOG.md` file label Apr 1, 2026
node-builder, note-transport, and remote-prover binary caches used
exact Cargo.lock hash keys with no fallback. When Cargo.lock differs
between branches (e.g. release vs next), every cache lookup misses
and binaries rebuild from scratch.

Add restore-keys so a stale binary can be restored as a starting
point. For build jobs this enables incremental recompilation. For
consumer jobs the build dependency ensures the correct binary is
already cached by the time they run.
@juan518munoz
Copy link
Copy Markdown
Collaborator Author

CI Cache Benchmark — Run 3 (added restore-keys to binary caches)

Compared against Run 2 (same branch, no restore-keys) and baseline on release/v0.14.0-beta.

Binary build jobs (the target of this change)

Job Run 3 (restore-keys) Run 2 (no fallback) Baseline
Build node-builder 0:06 15:51 0:05
Build remote-prover 0:05 5:12 0:10
Build note-transport 0:04 4:07 0:07

The restore-keys fallback found cached binaries from other branches/PRs. Cache hits are now on par with the baseline.

Other jobs (controls)

Job Run 3 Run 2 Baseline
Clippy 9:22 9:50 8:16
Unit tests 40:25 41:10 39:59
Documentation tests 2:56 2:58 2:50
Build Client and CLI 19:34 20:15 19:59
Build Client for Wasm 5:23 5:45 5:30

Control jobs are stable — confirms the improvement comes purely from the restore-keys addition.

Summary

Adding restore-keys to the 3 binary caches saved ~25 min of build time that was previously wasted rebuilding node-builder, remote-prover, and note-transport from scratch on every PR with a different Cargo.lock. Downstream jobs (integration tests) also start sooner since their dependencies finish faster.

The clippy --release + rust-release cache optimization only works when
clippy and build use the same toolchain. On this branch clippy uses
+nightly while the cache was built with stable, so the compiled
artifacts are incompatible and --release just adds overhead (+1:34).

Keeps the restore-keys improvement to binary caches which saves ~25 min.
Workflows only triggered pushes on [main, next], and caches were only
saved on refs/heads/next. PRs targeting release/* branches had no
warm cache — Unit tests and other jobs compiled everything from scratch.

- Add release/** to push triggers for build, test, and lint workflows
- Change save-if from next-only to any non-PR push, so caches are
  saved on release branches too
Base automatically changed from release/v0.14.0-beta to next April 4, 2026 02:29
@igamigo
Copy link
Copy Markdown
Collaborator

igamigo commented Apr 10, 2026

@juan518munoz , do you think some of hte changes here still make sense to apply?

@juan518munoz
Copy link
Copy Markdown
Collaborator Author

Closing as the branch became stale, and no real improvement was found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changelog This PR does not require an entry in the `CHANGELOG.md` file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants