Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions .buildkite/README-NPU-CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,26 @@ list in `SUITES`.

## Adding or removing a patch

Patches live in `docker/npu_patch/` and are registered in the `PATCH_CONFIGS`
dict in [`update-npu-environment.sh`](./scripts/update-npu-environment.sh).
Update both when adding or removing a patch.

Patches are applied explicitly by [`docker/Dockerfile.npu`](../docker/Dockerfile.npu)
and reconciled at runtime in the order declared by
[`docker/npu_patch/series.conf`](../docker/npu_patch/series.conf).
Each entry declares `target_worktree|image_patch|source_patch`: the image keeps
the flat `image_patch` under `/opt/npu_patch`, while runtime CI reads
`source_patch` relative to the current VIME checkout.

When adding a patch file, add the matching Dockerfile COPY/apply operations and
the `series.conf` entry in the same order. When deleting a patch, remove the
Dockerfile operation and `series.conf` entry before deleting the patch file;
the previous image retains the OLD bytes required for runtime revert. Rename
and reorder changes must likewise update both declarations. Ordinary patch
changes must not add patch-specific branches to the CI script.

Patch-only changes reuse the default image and are reconciled when the test
container starts. Changes that add, remove, or upgrade external repositories,
modify installed dependencies, or otherwise change the Docker image must be
validated with `image-build`.

The initial rollout of this mechanism requires publishing a new default image
that contains `/opt/npu_patch/series.conf`, then updating
`DEFAULT_CI_IMAGE` in `npu_suites.py`. The legacy default image contains patch
bytes under `/tmp/npu_patch` but not the OLD series required by this reconciler.
4 changes: 3 additions & 1 deletion .buildkite/npu_suites.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
import sys

NPU_QUEUE = "ascend-a3"
DEFAULT_CI_IMAGE = "quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1"
DEFAULT_CI_IMAGE = "quay.io/ascend/vime:vime-latest"
IMAGE_REGISTRY = "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
IMAGE_NAME = "vime-ci-npu"
VIME_IMAGE_TAG = os.environ.get("BUILDKITE_COMMIT", "latest")
Expand All @@ -30,6 +30,7 @@
"smk": [
("test_qwen3_4B_npu.py", "npu-8", "", {}),
("test_qwen3_30B_A3B_npu.py", "npu-16", "", {}),
("test_qwen3_vl_8B_npu.py", "npu-8", "", {}),
],
"nightly": [],
}
Expand Down Expand Up @@ -89,6 +90,7 @@ def npu_step(suite: str, test_name: str, resource_class: str, extra_args: str, e
'if [ -n "${BUILDKITE_COMMIT}" ]; then',
" source /workspace/build/buildkite/.buildkite/scripts/update-npu-environment.sh",
"fi",
"export HF_HUB_OFFLINE=0",
f"python tests/{test_name}{' ' + extra_args if extra_args else ''}",
]
)
Expand Down
11 changes: 3 additions & 8 deletions .buildkite/pipeline-npu.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,8 @@
# Usage: buildkite-agent pipeline upload .buildkite/pipeline-npu.yaml
#
# This pipeline is for running NPU tests on PRs.
# Before execution, it checks:
# 1. Trigger type: PR trigger or scheduled trigger
# 2. Changed files: whether docker/Dockerfile.npu is modified (PR trigger only)
# Based on the check results:
# - Image build decision: scheduled trigger always builds; PR trigger builds only if Dockerfile.npu is modified
# - Test image selection: uses newly built image ($IMAGE_REGISTRY/$IMAGE_NAME:$VIME_IMAGE_TAG) when build is required,
# otherwise uses pre-built default image (quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1)
# Scheduled builds always build a fresh image. PR builds reuse DEFAULT_CI_IMAGE
# unless image-build is selected; patch-only changes are reconciled at runtime.

steps:
- group: ":pipeline: Run NPU test"
Expand Down Expand Up @@ -75,4 +70,4 @@ steps:
SKIP_IMAGE_BUILD=true
fi
sed -e "s/__SKIP_IMAGE_BUILD__/$${SKIP_IMAGE_BUILD}/g" .buildkite/pipeline-npu-image.yaml | buildkite-agent pipeline upload
python .buildkite/npu_suites.py | buildkite-agent pipeline upload
python .buildkite/npu_suites.py | buildkite-agent pipeline upload
Loading