Skip to content

[CI] Bazel: Phase 1 — wire stage-b-test-1-gpu-small into pr-test.yml (demo, stage 2 of 3) - #3

Draft
Kangyan-Zhou wants to merge 2 commits into
feat/bazel-tag-codegenfrom
feat/bazel-pr-test-yml
Draft

[CI] Bazel: Phase 1 — wire stage-b-test-1-gpu-small into pr-test.yml (demo, stage 2 of 3)#3
Kangyan-Zhou wants to merge 2 commits into
feat/bazel-tag-codegenfrom
feat/bazel-pr-test-yml

Conversation

@Kangyan-Zhou

Copy link
Copy Markdown
Owner

Demo stack — stage 2 of 3

This stage adds a parallel-shipping bazel test step to the existing stage-b-test-1-gpu-small job. The Bazel run is gated to matrix.partition == 0 and continue-on-error, so it can never break the existing pipeline while we verify parity with run_suite.py.

Stack:

Edit

Two new steps in the stage-b-test-1-gpu-small job, both gated to matrix.partition == 0:

  • Install bazelisk (idempotent, atomic via temp file + mv, arch-aware)
  • Run test (Bazel, parallel-shipping verification) — bazel test //test/... --test_tag_filters=sgl-suite-stage-b-test-1-gpu-small

Plus a Bazel-specific upload-cuda-coredumps step with always() guard + distinct artifact suffix to avoid collision with the existing run_suite.py coredump upload.

Validation on H100

  • yaml parses cleanly
  • Bazel command exact-shape ran on lmsysorg/sglang:dev: 9 tagged tests dispatched, 5 pass / 4 fail. The 4 failures are model-loading test paths missing the full ci_install_dependency.sh venv — not a Bazel issue.

Review history (in chat)

code-reviewer agent flagged 2 important findings on commit c317514:

  • Coredump collision (continue-on-error swallows Bazel-side dumps; suffix collides on partition 0)
  • Corrupt-binary bypass (command -v returns true for half-written binary)

Both fixed in aed1282:

  • bazel version >/dev/null gate instead of command -v
  • temp file + mv for atomic install
  • uname -m arch detection (x86_64/arm64 fallback)
  • Bazel-specific coredump upload with always() + 0-bazel suffix

🤖 Generated with Claude Code

Kangyan-Zhou and others added 2 commits May 4, 2026 22:10
Phase-1 demo: wires `bazel test` into the existing stage-b-test-1-gpu-small
pr-test.yml job alongside the existing run_suite.py invocation. The Bazel
step is gated to matrix.partition == 0 so the full suite runs once per
workflow (vs run_suite.py's 8-way est_time partitioning), and continues-on-
error so a Bazel-side failure cannot break the existing pipeline while
we're verifying parity.

Edit at .github/workflows/pr-test.yml stage-b-test-1-gpu-small job adds
two steps after the existing "Run test":
  - "Install bazelisk" — fetches bazelisk to /usr/local/bin/bazel if the
    runner doesn't already have it.
  - "Run test (Bazel, parallel-shipping verification)" — invokes
    `bazel test //test/... --test_tag_filters=sgl-suite-stage-b-test-1-gpu-small`
    with --jobs=1 --local_test_jobs=1 to avoid GPU contention on the
    1-gpu-5090 runner.

Both steps drop the partition gate + continue-on-error once the Bazel
path has been validated against run_suite.py for one or two release
cycles.

Validated on H100 (lmsysorg/sglang:dev container kangyan_bazel_test):
- yaml parses cleanly via PyYAML.
- bazel test //test/... --test_tag_filters=sgl-suite-stage-b-test-1-gpu-small
  dispatches the 9 tagged tests across the 3 wired BUILDs, runs them all
  with --cache_test_results=no, returns "5 pass and 4 fail" — the 4
  failures are real environment gaps (model-loading and fork-test paths
  needing the full ci_install_dependency.sh venv), not Bazel issues.

Depends on feat/bazel-tag-codegen for the sgl-suite-* tags. The Bazel
side never ran without that layer's registry.

Next layer (feat/bazel-manifest-job) replaces partition matrix +
auto_partition with a compute-test-manifest job that does dep-graph-aware
selection and est_time-aware bin-packing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes from the code-reviewer agent on c317514:

- Bazelisk install: gate on `bazel version` succeeding instead of
  `command -v bazel`. The 1-gpu-5090 runner is self-hosted with shared
  /usr/local/bin; a half-written binary from a prior aborted job would
  pass the `command -v` check but fail at runtime. Also writes the
  download to a temp file and `mv`s into place atomically so a partial
  curl can't poison the path.

- Architecture detection: `uname -m` with x86_64/arm64 fallback.
  Currently lock-in to amd64 was fine for 1-gpu-5090, but the one-line
  guard future-proofs the pool against an arm64 runner being added.

- Coredump capture for Bazel-side failures: `continue-on-error: true`
  on the Bazel step means a Bazel-only failure does NOT mark the job
  failed, so the existing `if: failure()` upload-cuda-coredumps never
  fires for those runs. Even if it did, both upload calls share
  `artifact-suffix: ${{ matrix.partition }}` (= "0") and would
  collide. Add a Bazel-specific upload with `if: matrix.partition == 0
  && always()` and suffix `0-bazel` so dumps are deterministically
  captured during the parallel-shipping window.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant