Skip to content

Commit f08003a

Browse files
committed
fix: recover exporter runtime from retained squash
从已校验的留存 squash 恢复 exporter 运行时。
1 parent 2cbdb9a commit f08003a

7 files changed

Lines changed: 612 additions & 23 deletions

File tree

‎.github/workflows/amd-exporter-image.yml‎

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ jobs:
1919
runs-on: ubuntu-24.04
2020
permissions:
2121
contents: read
22-
actions: read # Download the pinned upstream build artifact.
22+
actions: read # Reuse the pinned prepared runtime artifact.
2323
packages: write # Publish the verified image to the repository-linked GHCR package.
2424
timeout-minutes: 30
2525
env:
@@ -28,8 +28,9 @@ jobs:
2828
GPU_AGENT_SHA: bcae57f9b395363f3340ad0b7393659b7df13f37
2929
ARCHIVE_NAME: device-metrics-exporter-v1.5.4-10.2.0a20261001.tar.gz
3030
ARCHIVE_SHA256: 0ba397bc31abc1c538e24f8d7264fb421cc5fb8dd59b2e1c493a8efec404c2ba
31-
BASE_IMAGE: rocm/device-metrics-exporter:10.2.0a20261001
31+
BASE_IMAGE: powerx-amd-recovered:runtime-36985488002
3232
BASE_CONFIG_ID: sha256:aa151c70573fd5d3f94538c73adac6b235f052bb7dc9a8681ad2bfd0a8417929
33+
RECOVERY_SQUASH_SHA256: 0f2cbb4e2c41c1506db74c6653ea09ec5e7d42236d9532e2bc38fb9e0be5879e
3334
IMAGE_REF: ghcr.io/semianalysisai/amd-device-metrics-exporter:powerx-9d0eb8c-cache-${{ github.sha }}
3435
ENROOT_PACKAGE: enroot_4.2.0-1_amd64.deb
3536
ENROOT_PACKAGE_SHA256: 50bfcf8336bfe1a5d19481081d7800ae80300a27795595757f7d20957e079a62
@@ -50,12 +51,12 @@ jobs:
5051
with:
5152
go-version: 1.25.13
5253
cache: false
53-
- name: Download exact official runtime image
54+
- name: Download the retained prepared runtime
5455
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
5556
with:
56-
repository: ROCm/device-metrics-exporter
57-
run-id: 36867107435
58-
name: build-dme-10.2.0a20261001
57+
repository: SemiAnalysisAI/InferenceX
58+
run-id: 36985488002
59+
artifact-ids: 11217112734
5960
github-token: ${{ github.token }}
6061
path: ${{ runner.temp }}/amd-exporter-image
6162
- name: Apply cache patch and test the exporter
@@ -80,11 +81,19 @@ jobs:
8081
run: |
8182
set -eo pipefail
8283
source inferencex-e2e/benchmarks/benchmark_lib.sh --validation-only
83-
check_env_vars IMAGE_DIR OUTPUT_DIR ARCHIVE_NAME ARCHIVE_SHA256 BASE_IMAGE BASE_CONFIG_ID IMAGE_REF \
84-
SOURCE_SHA PATCH_SHA256 GITHUB_REPOSITORY GITHUB_SHA
85-
printf '%s %s\n' "$ARCHIVE_SHA256" "$IMAGE_DIR/$ARCHIVE_NAME" | sha256sum --check -
86-
docker image load --input "$IMAGE_DIR/$ARCHIVE_NAME"
87-
test "$(docker image inspect "$BASE_IMAGE" --format '{{.Id}}')" = "$BASE_CONFIG_ID"
84+
check_env_vars IMAGE_DIR OUTPUT_DIR RECOVERY_SQUASH_SHA256 BASE_IMAGE IMAGE_REF \
85+
ENROOT_PACKAGE ENROOT_PACKAGE_SHA256 SOURCE_SHA PATCH_SHA256 GITHUB_REPOSITORY GITHUB_SHA
86+
printf '%s %s\n' "$RECOVERY_SQUASH_SHA256" "$IMAGE_DIR/amd-exporter.sqsh" | sha256sum --check -
87+
curl --fail --location --output "$IMAGE_DIR/$ENROOT_PACKAGE" \
88+
"https://github.com/NVIDIA/enroot/releases/download/v4.2.0/$ENROOT_PACKAGE"
89+
printf '%s %s\n' "$ENROOT_PACKAGE_SHA256" "$IMAGE_DIR/$ENROOT_PACKAGE" | sha256sum --check -
90+
sudo apt-get update
91+
sudo apt-get install --yes squashfs-tools "$IMAGE_DIR/$ENROOT_PACKAGE"
92+
sudo unsquashfs -d "$IMAGE_DIR/rootfs" "$IMAGE_DIR/amd-exporter.sqsh"
93+
mkdir -p "$OUTPUT_DIR"
94+
sudo python3 tools/amd-exporter/recover-base.py "$IMAGE_DIR/rootfs" \
95+
"$IMAGE_DIR/provenance.json" tools/amd-exporter/base-runtime.json \
96+
"$BASE_IMAGE" "$OUTPUT_DIR/base-recovery.json"
8897
docker build --pull=false --build-arg "BASE_IMAGE=$BASE_IMAGE" \
8998
--label "org.opencontainers.image.source=https://github.com/$GITHUB_REPOSITORY" \
9099
--label "org.opencontainers.image.revision=$GITHUB_SHA" \
@@ -120,11 +129,6 @@ jobs:
120129
check_env_vars IMAGE_DIR OUTPUT_DIR IMAGE_REF ENROOT_PACKAGE ENROOT_PACKAGE_SHA256 SOURCE_SHA \
121130
PATCH_SHA256 GPU_AGENT_SHA ARCHIVE_NAME ARCHIVE_SHA256 BASE_CONFIG_ID GITHUB_REPOSITORY \
122131
GITHUB_RUN_ID GITHUB_RUN_ATTEMPT GITHUB_SHA
123-
curl --fail --location --output "$IMAGE_DIR/$ENROOT_PACKAGE" \
124-
"https://github.com/NVIDIA/enroot/releases/download/v4.2.0/$ENROOT_PACKAGE"
125-
printf '%s %s\n' "$ENROOT_PACKAGE_SHA256" "$IMAGE_DIR/$ENROOT_PACKAGE" | sha256sum --check -
126-
sudo apt-get update
127-
sudo apt-get install --yes "$IMAGE_DIR/$ENROOT_PACKAGE"
128132
enroot import --output "$OUTPUT_DIR/amd-exporter.sqsh" "dockerd://$IMAGE_REF"
129133
unsquashfs -s "$OUTPUT_DIR/amd-exporter.sqsh"
130134
cp tools/amd-exporter/cache.patch "$OUTPUT_DIR/cache.patch"

‎tools/amd-exporter/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@
44

55
The CPU-only `AMD exporter image` workflow builds a fresh-sample exporter and publishes a unique tag to `ghcr.io/semianalysisai/amd-device-metrics-exporter`. It does not start a GPU benchmark or change production data.
66

7-
The base is the official ROCm nightly artifact from run `36867107435`, source `9d0eb8c88af99f1dbe2914d803382092671456d9`. Its archive checksum and image config digest are verified before use. The runtime includes the upstream 255 W sentinel correction. `cache.patch` applies the configurable GPUGet cache change from [functionstackx/device-metrics-exporter#1](https://github.com/functionstackx/device-metrics-exporter/pull/1), with a default-cache regression and concurrent zero-cache tests.
7+
The official ROCm nightly artifact from run `36867107435` has been deleted upstream. Reuse the retained Enroot conversion from InferenceX run `36985488002`, artifact `11217112734`, verified against squash SHA256 `0f2cbb4e2c41c1506db74c6653ea09ec5e7d42236d9532e2bc38fb9e0be5879e` and its original provenance. It contains source `9d0eb8c88af99f1dbe2914d803382092671456d9`. The runtime includes the upstream 255 W sentinel correction. `cache.patch` applies the configurable GPUGet cache change from [functionstackx/device-metrics-exporter#1](https://github.com/functionstackx/device-metrics-exporter/pull/1), with a default-cache regression and concurrent zero-cache tests.
88

9-
Only `/home/amd/bin/server` is replaced. The original ROCm and GPUAgent layers and entrypoint are retained. `AMD_GPU_GET_CACHE_TTL=0s` forces each exporter request to issue a new GPUGet RPC. This removes exporter response reuse; actual hardware freshness, overhead and one-second benchmark collection still require live validation.
9+
`base-runtime.json` records the original Docker configuration and hashes/symlink targets for 61 critical paths, extracted from the retained official OCI archive. Recovery verifies GPUAgent, entrypoint, AMD-SMI, ROCm sysdeps and companion binaries before importing the root filesystem with the original environment, entrypoint, labels and working directory. Enroot flattened layers, normalized ownership/permissions and generated its `/etc` startup files; the reconstructed base has a new identity and is not the original OCI image. Only `/home/amd/bin/server` is replaced in the subsequent overlay. `AMD_GPU_GET_CACHE_TTL=0s` forces each exporter request to issue a new GPUGet RPC. This removes exporter response reuse; actual hardware freshness, overhead and one-second benchmark collection still require live validation.
1010

11-
The workflow verifies the patch checksum, runs focused cache race tests, builds Linux/amd64, checks the image's version and invalid-setting rejection, and verifies the unchanged base layers and runtime configuration. It then publishes a commit-specific tag and saves an Enroot squash with `SHA256SUMS` and `provenance.json` in `amd-dme-cache-squash`. Consumers must record the published registry digest and the squash checksum. The source commit, patch hash, GPUAgent commit and workflow revision are separate provenance fields; the embedded ROCm commit is explicitly unknown.
11+
The workflow verifies the patch checksum, runs focused cache race tests, builds Linux/amd64, checks the image's version and invalid-setting rejection, and verifies the unchanged reconstructed base layers and runtime configuration. It then publishes a commit-specific tag and saves an Enroot squash with `SHA256SUMS` and `provenance.json` in `amd-dme-cache-squash`. Consumers must record the published registry digest and the squash checksum. `base-recovery.json` records the recovery run/artifact, source receipt, runtime manifest hash, verified-file count and reconstructed image identity. The source commit, patch hash, GPUAgent commit and workflow revision are separate provenance fields; the embedded ROCm commit is explicitly unknown.
1212

1313
Dispatch this workflow manually on the reviewed branch, or push the dedicated `chore/powerx-amd-exporter-image` branch. The job has a 30-minute CPU limit and needs `packages: write`. It does not move a `latest` tag. A successful image build is preparation, not AMD measurement acceptance.

‎tools/amd-exporter/README_zh.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@
44

55
仅使用 CPU 的 `AMD exporter image` 工作流构建可获取新读数的 exporter,并向 `ghcr.io/semianalysisai/amd-device-metrics-exporter` 发布唯一标签。它不启动 GPU 基准测试,也不修改生产数据。
66

7-
基础镜像来自官方 ROCm nightly 工作流 `36867107435`,源码为 `9d0eb8c88af99f1dbe2914d803382092671456d9`。使用前校验归档 SHA256 和镜像配置摘要。该运行时包含上游 255 W 哨兵值修复。`cache.patch` 基于 [functionstackx/device-metrics-exporter#1](https://github.com/functionstackx/device-metrics-exporter/pull/1) 的可配置 GPUGet 缓存修改,并补充默认缓存回归测试和零缓存并发测试。
7+
官方 ROCm nightly 工作流 `36867107435` 的原始产物已被上游删除。现在复用 InferenceX 工作流 `36985488002` 保留的 Enroot 转换产物 `11217112734`,校验 squash SHA256 `0f2cbb4e2c41c1506db74c6653ea09ec5e7d42236d9532e2bc38fb9e0be5879e` 及原始来源回执。其源码为 `9d0eb8c88af99f1dbe2914d803382092671456d9`。该运行时包含上游 255 W 哨兵值修复。`cache.patch` 基于 [functionstackx/device-metrics-exporter#1](https://github.com/functionstackx/device-metrics-exporter/pull/1) 的可配置 GPUGet 缓存修改,并补充默认缓存回归测试和零缓存并发测试。
88

9-
只替换 `/home/amd/bin/server`,保留原有 ROCm、GPUAgent 镜像层及入口。`AMD_GPU_GET_CACHE_TTL=0s` 强制每次 exporter 请求发起新的 GPUGet RPC。这消除了 exporter 响应复用;硬件实际刷新率、采集开销和基准测试中的一秒采集仍需实机验证。
9+
`base-runtime.json` 从保留的官方 OCI 归档提取原始 Docker 配置,以及 61 个关键路径的文件哈希和符号链接目标。恢复过程先验证 GPUAgent、入口脚本、AMD-SMI、ROCm sysdeps 和配套二进制,再使用原始环境变量、入口、标签及工作目录导入根文件系统。Enroot 已合并镜像层、规范化所有权和权限,并生成 `/etc` 启动文件,因此重建基础镜像使用新标识,不能称为原始 OCI 镜像。之后仅在新增镜像层替换 `/home/amd/bin/server`。`AMD_GPU_GET_CACHE_TTL=0s` 强制每次 exporter 请求发起新的 GPUGet RPC。这消除了 exporter 响应复用;硬件实际刷新率、采集开销和基准测试中的一秒采集仍需实机验证。
1010

11-
工作流校验补丁摘要,运行缓存竞态测试,构建 Linux/amd64 二进制,检查版本与非法配置拒绝行为,并验证基础镜像层及运行时配置未变。随后发布提交专属标签,生成 Enroot squash,并将 `SHA256SUMS`、`provenance.json` 保存为 `amd-dme-cache-squash` 产物。使用方应记录发布的注册表摘要及 squash 校验值。源码提交、补丁哈希、GPUAgent 提交和工作流版本分别记录;二进制中的 ROCm 提交明确标记为未知。
11+
工作流校验补丁摘要,运行缓存竞态测试,构建 Linux/amd64 二进制,检查版本与非法配置拒绝行为,并验证重建基础镜像层及运行时配置未变。随后发布提交专属标签,生成 Enroot squash,并将 `SHA256SUMS`、`provenance.json` 保存为 `amd-dme-cache-squash` 产物。使用方应记录发布的注册表摘要及 squash 校验值。`base-recovery.json` 记录恢复工作流和产物、原始来源回执、运行时清单哈希、已验证文件数及重建镜像标识。源码提交、补丁哈希、GPUAgent 提交和工作流版本分别记录;二进制中的 ROCm 提交明确标记为未知。
1212

1313
在已审阅分支手动触发,或推送专用分支 `chore/powerx-amd-exporter-image`。CPU 作业上限为 30 分钟,需要 `packages: write` 权限,不移动 `latest` 标签。镜像构建通过只代表准备完成,不代表 AMD 功耗测量已验收。

0 commit comments

Comments
 (0)