Skip to content

Commit 2e41681

Browse files
authored
ci: speed up builds, tests, and benchmark artifact transfers (#10653)
| Check | What changed | Time change | | --- | --- | --- | | Windows Rust tests | Deterministic `/Brepro` linking improves compiler cache hits | Warm compilation: [296s](https://github.com/vortex-data/vortex/actions/runs/39492337565) → [162s](https://github.com/vortex-data/vortex/actions/runs/39520806593/job/118973748964) / [167s](https://github.com/vortex-data/vortex/actions/runs/39520806593/job/118970433451) | | Windows Python wheel | `LINK=/Brepro` also covers host proc-macro linking | Warm compilation: [226s](https://github.com/vortex-data/vortex/actions/runs/39524887291) → [46.5s](https://github.com/vortex-data/vortex/actions/runs/39565064197/job/119105255496) / [62s](https://github.com/vortex-data/vortex/actions/runs/39560723056/job/119100841650) | | ARM Rust tests | Use m9g.4xlarge with the same 16 CPUs | Job: [270s](https://github.com/vortex-data/vortex/actions/runs/39492337565) → [196–198s](https://github.com/vortex-data/vortex/actions/runs/39501054993) | | Rust docs | Increase runner allocation from 4 to 8 CPUs | Job: [288s](https://github.com/vortex-data/vortex/actions/runs/39513360796) → [212–235s](https://github.com/vortex-data/vortex/actions/runs/39507489285) (runner families differ) | | SQL logic tests | Increase runner allocation to 8 CPUs | Test execution: [84s](https://github.com/vortex-data/vortex/actions/runs/39524887291) → [69s](https://github.com/vortex-data/vortex/actions/runs/39560723056/job/119091134964) / [76s](https://github.com/vortex-data/vortex/actions/runs/39565064197/job/119105361098) | | CPU/CUDA benchmark artifacts | Parallel zstd packaging; disable redundant upload compression | CPU transport: [125–160s](https://github.com/vortex-data/vortex/actions/runs/39492337436) → [25–45s](https://github.com/vortex-data/vortex/actions/runs/39518753128); CUDA packaging/upload: [36s](https://github.com/vortex-data/vortex/actions/runs/39518753128) → [4s](https://github.com/vortex-data/vortex/actions/runs/39520806524) | | CPU walltime benchmark builds | Preserve the bench profile's limited debug information | Warm AVX2 compilation: [21.48s](https://github.com/vortex-data/vortex/actions/runs/39560723037/job/119091041059) → [16.69s](https://github.com/vortex-data/vortex/actions/runs/39565064178/job/119109008645) | --------- Signed-off-by: blaginin <github@blaginin.me>
1 parent 633c52d commit 2e41681

2 files changed

Lines changed: 22 additions & 8 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,8 @@ jobs:
181181
env:
182182
# Keep this constant to avoid pyo3 invalidating itself
183183
PYO3_ENVIRONMENT_SIGNATURE: "cpython3"
184+
# Also applies to host proc-macro links when Cargo receives an explicit target.
185+
LINK: /Brepro
184186

185187
# rust-test-windows excludes vortex-python, so this is the only check that the
186188
# win_amd64 wheel imports and can round-trip a file. Runs outside the checkout so
@@ -284,7 +286,7 @@ jobs:
284286
timeout-minutes: 30
285287
runs-on: >-
286288
${{ github.repository == 'vortex-data/vortex'
287-
&& format('runs-on={0}/runner=amd64-ci-small/image=ubuntu24-full-x64-pre-v2/extras=s3-cache/tag=rust-docs', github.run_id)
289+
&& format('runs-on={0}/runner=amd64-ci-docs/image=ubuntu24-full-x64-pre-v2/extras=s3-cache/tag=rust-docs', github.run_id)
288290
|| 'ubuntu-latest' }}
289291
steps:
290292
- parallel:
@@ -505,6 +507,9 @@ jobs:
505507
506508
rust-test-windows:
507509
name: "Rust tests (windows-x64)"
510+
env:
511+
# Stable proc-macro DLL contents let dependent crates reuse sccache entries.
512+
RUSTFLAGS: "-C link-arg=/Brepro"
508513
needs: duckdb-ready
509514
if: ${{ !cancelled() && needs.duckdb-ready.result == 'success' }}
510515
timeout-minutes: 30
@@ -556,7 +561,7 @@ jobs:
556561
!cancelled() && needs.duckdb-ready.result == 'success' &&
557562
github.repository == 'vortex-data/vortex'
558563
timeout-minutes: 30
559-
runs-on: runs-on=${{ github.run_id }}/runner=arm64-medium/image=ubuntu24-full-arm64-pre-v2/extras=s3-cache/tag=rust-test-linux-arm64
564+
runs-on: runs-on=${{ github.run_id }}/runner=arm64-ci-test/image=ubuntu24-full-arm64-pre-v2/extras=s3-cache/tag=rust-test-linux-arm64
560565
steps:
561566
- parallel:
562567
- uses: runs-on/action@v2
@@ -786,7 +791,7 @@ jobs:
786791
if: ${{ !cancelled() && needs.duckdb-ready.result == 'success' }}
787792
runs-on: >-
788793
${{ github.repository == 'vortex-data/vortex'
789-
&& format('runs-on={0}/runner=amd64-ci-medium/image=ubuntu24-full-x64-pre-v2/extras=s3-cache/tag=sql-logic-test', github.run_id)
794+
&& format('runs-on={0}/runner=amd64-ci-sql-logic-test/image=ubuntu24-full-x64-pre-v2/extras=s3-cache/tag=sql-logic-test', github.run_id)
790795
|| 'ubuntu-latest' }}
791796
timeout-minutes: 30
792797
steps:

‎.github/workflows/codspeed.yml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -247,15 +247,18 @@ jobs:
247247
VORTEX_BENCH_PREFIX: "${{ matrix.features }}::"
248248
VORTEX_BENCH_SUFFIX: "_${{ matrix.features }}"
249249
run: |
250+
# Preserve the bench profile debug level instead of cargo-codspeed's full debug info.
251+
export CARGO_ENCODED_RUSTFLAGS="${RUSTFLAGS// /$'\x1f'}"
250252
cargo codspeed build --locked -m walltime --profile bench \
251253
${{ steps.select.outputs.packages }} ${{ steps.select.outputs.benches }}
252254
- name: Package benchmark executables
253-
run: tar -C target -cf codspeed-cpu-benchmarks.tar codspeed
255+
run: tar -C target -I 'zstd -T0 -3' -cf codspeed-cpu-benchmarks.tar.zst codspeed
254256
- name: Upload benchmark executables
255257
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
256258
with:
257259
name: codspeed-cpu-benchmarks-${{ matrix.features }}
258-
path: codspeed-cpu-benchmarks.tar
260+
path: codspeed-cpu-benchmarks.tar.zst
261+
compression-level: 0
259262
retention-days: 1
260263
if-no-files-found: error
261264

@@ -289,7 +292,7 @@ jobs:
289292
- name: Unpack benchmark executables
290293
run: |
291294
mkdir -p target
292-
tar -C target -xf codspeed-cpu-benchmarks.tar
295+
tar -C target --zstd -xf codspeed-cpu-benchmarks.tar.zst
293296
# Pinning clocks and reserving CPUs only pays off for walltime measurements, so this
294297
# runs here but not in the simulation job. It must come after setup, which itself
295298
# spawns processes that would otherwise be pinned to the housekeeping CPUs.
@@ -360,11 +363,14 @@ jobs:
360363
find target/release/build -path '*/out/libvortex_cub.so' \
361364
-exec cp {} target/codspeed/walltime/vortex-cuda/libvortex_cub.so \;
362365
test -f target/codspeed/walltime/vortex-cuda/libvortex_cub.so
366+
- name: Package CUDA benchmark executables
367+
run: tar -C target -I 'zstd -T0 -3' -cf codspeed-cuda-benchmarks.tar.zst codspeed
363368
- name: Upload benchmark executables
364369
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
365370
with:
366371
name: codspeed-cuda-benchmarks
367-
path: target/codspeed/
372+
path: codspeed-cuda-benchmarks.tar.zst
373+
compression-level: 0
368374
retention-days: 1
369375
if-no-files-found: error
370376

@@ -403,7 +409,10 @@ jobs:
403409
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8
404410
with:
405411
name: codspeed-cuda-benchmarks
406-
path: target/codspeed
412+
- name: Unpack CUDA benchmark executables
413+
run: |
414+
mkdir -p target
415+
tar -C target --zstd -xf codspeed-cuda-benchmarks.tar.zst
407416
- name: Restore executable permissions
408417
run: find target/codspeed -type f -exec chmod +x {} +
409418
- name: Run benchmarks

0 commit comments

Comments
 (0)