Skip to content
hipObject - Emulated HW Test - MinIO Bridge

Clean up small CMake issues #336

Sign in to view logs

Clean up small CMake issues

Clean up small CMake issues #336

Workflow file for this run

.github/workflows/hipobject-hardware-test-minio-bridge.yml at 66ef2c1
# Copyright (c) Advanced Micro Devices, Inc. All rights reserved.
#
# SPDX-License-Identifier: MIT
#
# MinIO bridge integration test (minio-cpp, two VMs).
#
# This lane used to be three containers, and it could not do what its name
# claimed. rocm-ernic serves an emulated PCI function over vfio-user; turning
# that into a verbs device takes a guest kernel's ionic/ionic_rdma drivers.
# With client and server in separate containers -- each running its own
# loopback emulator, with nothing between them -- /sys/class/infiniband stayed
# empty, HIPOBJ_NIC_HINT resolved to nothing, and hipobj::minio::Client fell
# back to plain HTTP and returned success. The lane went green over TCP.
#
# So it is two VMs now, the same topology as the Two-VM Protocol lane: two
# rocm-ernic instances joined by a tcp:manager / tcp:worker mesh, each bridged
# onto ernicbr0 through its own tap and serving one emulated ionic function.
# VM 1 (192.168.200.10) runs hipobj-rdma-test-server on port 9000;
# VM 2 drives the minio-cpp bridge against it and asserts, from the bridge's
# own counters, that every byte moved over RDMA.
#
# What it adds over Two-VM Protocol is the bridge entrypoint: round-trip
# timing and a throughput figure per run, which is the number this lane
# exists to watch. The GPU-buffer form of the same transfer is
# hipobject-hardware-test-two-vm-gpu-direct.yml.
name: hipObject - Emulated HW Test - MinIO Bridge
on:
# Pull requests, pushes to develop, and manual runs. The push run
# catches a develop that no single PR was tested against.
#
# The paths are what this lane builds and runs, so a change it cannot
# observe doesn't spend VM time. Unit, header and install tests,
# examples, the BNXT provider (built OFF here), docs, Clang options
# (this lane builds with GCC), and the CMake modules for packaging or
# opt-in tooling never reach this lane's binaries; other lanes cover
# them. Negations must come last: the last pattern a file matches
# decides whether it counts.
pull_request:
branches: [develop]
paths: &paths
- "CMakeLists.txt"
- "cmake/HIPOBJAddExecutable.cmake"
- "cmake/HIPOBJAddLibraries.cmake"
- "cmake/HIPOBJCompilerOptions.cmake"
- "cmake/HIPOBJFortifySource.cmake"
- "cmake/HIPOBJGNUCompilerOptions.cmake"
- "cmake/HIPOBJMinioCppDeps.cmake"
- "cmake/HIPOBJUseROCmCMake.cmake"
- "cmake/minio-deps/**"
- "include/**"
- "integrations/minio-cpp/**"
- "shared/**"
- "src/**"
- "test/CMakeLists.txt"
- "test/integration/CMakeLists.txt"
- "test/integration/rdma-test-server/**"
- "ci/ernic/assert-transfer.sh"
- "ci/ernic/background-task.sh"
- "ci/ernic/minio-client-entrypoint.sh"
- "ci/ernic/server-entrypoint.sh"
- ".github/actions/fetch-guest-vm/**"
- ".github/actions/fetch-ionic-sources/**"
- ".github/workflows/hipobject-hardware-test-minio-bridge.yml"
- "!src/rdma/bnxt/**"
- "!**/*.md"
push:
branches: [develop]
paths: *paths
workflow_dispatch:
# A newer commit on a pull request cancels the run for the older one.
# Every other run (a push, a manual run) gets a group of its own: in a
# shared group, a newer run replaces a pending one even without
# cancel-in-progress, which would leave a commit untested.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
env:
# Bump these intentionally when the ernic or rocjitsu images are updated.
# ERNIC_IMAGE: ernic.0b48aa1, the build whose rocm-ernic carries the s3
# backend. Every tag here is the date-prefixed, content-versioned one
# (<date>.<repo-sha>-<component>.<sha>-vfu.<sha>): immutable, so a rebuild
# cannot change what this lane tests underneath it.
ERNIC_IMAGE: sbates130272/batesste-ci-images-ubuntu-rocm-ernic:20260922.g555f601-ernic.0b48aa1-vfu.8039244
ROCM_IMAGE: rocm/dev-ubuntu-24.04:10.0.0-full
# rocm-ernic's Ansible needs community.general >=13.3.0. Pinned, like the
# images, because the installed collections are cached under this version:
# fetching them from Galaxy took ~3 minutes of every run.
COMMUNITY_GENERAL_VERSION: 13.5.0
# Every run step uses bash, including in container jobs, where the runner
# would otherwise pick sh. "bash -e {0}" and not "bash": the bare name adds
# -o pipefail, which these scripts are not written for.
defaults:
run:
shell: bash -e {0}
jobs:
minio-bridge:
name: MinIO Bridge
runs-on: ubuntu-24.04
timeout-minutes: 120
# The previous container topology could not work and was gated off: with
# no VM in the picture the container's /sys/class/infiniband stays empty,
# so HIPOBJ_NIC_HINT=ernic0 never resolved. The missing piece was the
# guest, not the image tag. This is that topology, taken from rocm-ernic's
# own system-test-two-vms lane: two emulated NICs on one bridge, one
# guest per NIC, hipObject's test server in the first and the minio-cpp
# bridge client in the second.
container:
image: sbates130272/batesste-ci-images-ubuntu-qemu-libvfio-user:20260924.ge1cd640-qemu11.1.1-vfu.8039244
# /dev/net/tun and NET_ADMIN are as load-bearing as /dev/kvm: without
# them ionic_eth_net_send() has no fd and every guest frame is dropped,
# which looks exactly like a healthy-but-silent link.
options: --device /dev/kvm --device /dev/net/tun --cap-add=NET_ADMIN
env:
GUEST_ARTIFACT_REPO: docker.io/sbates130272/batesste-ci-images-ubuntu-qcow2-gen-ernic-rocjitsu
GUEST_ARTIFACT_TAG: 20260925.g4868a2e-vm.resolute-ernic-rocjitsu-qm.e73a1e6-qcow2
ERNIC_TAP_PREFIX: ernic-tap
ERNIC_TAP_BRIDGE: ernicbr0
TCP_PORT: 18515
VM1_SSH_PORT: 2222
VM2_SSH_PORT: 2223
# ernic_guest_setup derives guest addresses as <subnet>.<10 * index>.
VM1_ADDR: 192.168.200.10
SERVER_PORT: 9000
TEST_SIZE: 1048576
# Where the guest role looks for the ionic driver sources. Passed to it
# explicitly, so the copies staged there cannot drift from its default.
IONIC_GUEST_DIR: /var/tmp/ionic-src
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Checkout rocm-ernic (Ansible collection + guest driver sources)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: ROCm/rocm-ernic
# Pinned to the last develop commit known to work here; later commits
# break these lanes. Bump intentionally, like the images.
ref: f7e3ecb18fca02fe9d1e6685952ee10d4a78efe3
path: rocm-ernic
persist-credentials: false
- name: Start building hipObject binaries (ROCm container)
run: |
# In the background: pulling the ROCm image is most of the build's
# ~3 minutes, and nothing needs the binaries until they are copied
# into the guests. "Wait for the build" joins it, and prints its log.
#
# Stage what the binaries load at run time in rocm-libs, which
# travels to the guest VMs with them: the ROCm libraries,
# pugixml, curlpp when it's the system's, and the shared
# libhipobj. In this container the binaries' build-tree run path
# still finds libhipobj in /hipobject-build/src, so the check
# treats that as unstaged.
bash ci/ernic/background-task.sh start build << 'EOF'
docker run --rm --name hipobj-build \
--user root \
-v "${{ github.workspace }}:/hipobject:ro" \
-v "${{ github.workspace }}/ernic-build:/hipobject-build" \
-e ROCM_PATH=/opt/rocm \
-e CC=/opt/rocm/llvm/bin/amdclang \
-e CXX=/opt/rocm/llvm/bin/amdclang++ \
"$ROCM_IMAGE" \
bash -c "
apt-get update -qq && \
apt-get install -y -qq git cmake ninja-build libibverbs-dev \
libcurl4-openssl-dev libssl-dev \
nlohmann-json3-dev libpugixml-dev && \
cmake -B /hipobject-build -G Ninja -S /hipobject \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DBUILD_TESTING=ON \
-DHIPOBJ_IONIC=ON \
-DHIPOBJ_BNXT=OFF \
-DHIPOBJ_INTEGRATION_TESTS=ON \
-DHIPOBJ_MINIO_CLIENT=ON \
-DHIPOBJ_BUILD_DOCS=OFF && \
cmake --build /hipobject-build --parallel \$(nproc) \
--target hipobj-rdma-test-server minio-getput-rdma && \
mkdir -p /hipobject-build/rocm-libs && \
export LD_LIBRARY_PATH=/opt/rocm/lib && \
for b in /hipobject-build/test/integration/rdma-test-server/hipobj-rdma-test-server \
/hipobject-build/integrations/minio-cpp/minio-getput-rdma; do \
ldd \$b \
| awk '/=> \//{print \$3}' \
| grep -E '^/opt/(rocm|amdgpu)|/libpugixml\.so|/libcurlpp\.so|/libhipobj\.so' \
| xargs -r -I{} cp -L {} /hipobject-build/rocm-libs/; \
done && \
for b in /hipobject-build/test/integration/rdma-test-server/hipobj-rdma-test-server \
/hipobject-build/integrations/minio-cpp/minio-getput-rdma; do \
if LD_LIBRARY_PATH=/hipobject-build/rocm-libs ldd \$b \
| grep -E 'not found|=> /hipobject-build/src/'; then \
echo 'staged closure incomplete'; exit 1; \
fi; \
done && \
echo 'staged closure complete'
"
EOF
- name: Restore the cached Ansible collections
id: ansible_collections
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.ansible/collections
key: ansible-collections-community.general-${{ env.COMMUNITY_GENERAL_VERSION }}
- name: Start installing deps
env:
COLLECTIONS_CACHED: ${{ steps.ansible_collections.outputs.cache-hit }}
run: |
# In the background, alongside the guest image fetch. "Wait for
# deps" joins it before anything uses what it installs.
bash ci/ernic/background-task.sh start deps << 'EOF'
apt-get update -qq
apt-get install -y -qq iproute2 sudo libibverbs1 libnl-3-200 \
libnl-route-3-200 ansible python3-jmespath
if [ "$COLLECTIONS_CACHED" != true ]; then
# Galaxy drops connections often enough to fail a lane on its own.
for attempt in 1 2 3; do
ansible-galaxy collection install \
"community.general:==${COMMUNITY_GENERAL_VERSION}" && break
# Fail here: carrying on without the collection fails much
# later, in provisioning, with an error that doesn't say why.
if [ "$attempt" -eq 3 ]; then
echo "::error::galaxy install failed after 3 attempts"
exit 1
fi
echo "::warning::galaxy install failed (attempt ${attempt}/3)"
sleep $((attempt * 20))
done
fi
EOF
- name: Start pulling the ernic image
run: |
# In the background, alongside the guest image fetch, so the step
# that extracts rocm-ernic from it doesn't wait for the pull. "Wait
# for the image" joins it before then.
bash ci/ernic/background-task.sh start images << 'EOF'
docker pull "$ERNIC_IMAGE"
EOF
- name: Fetch the ionic driver sources
id: ionic_sources
uses: $/.github/actions/fetch-ionic-sources
- name: Fetch guest VM image
id: guest_image
uses: $/.github/actions/fetch-guest-vm
with:
repo: ${{ env.GUEST_ARTIFACT_REPO }}
tag: ${{ env.GUEST_ARTIFACT_TAG }}
- name: Read VM info
id: vm_info
env:
GUEST_OUTPUT_DIR: ${{ steps.guest_image.outputs.output_dir }}
run: |
OUT="${GUEST_OUTPUT_DIR}"
eval "$(python3 - "$OUT" <<'PYEOF'
import json, os, shlex, sys
out = sys.argv[1]
d = json.load(open(os.path.join(out, "vm-info.json")))
here = lambda p: os.path.join(out, os.path.basename(p)) if p else ""
keys = {
"VM_USER": d.get("username", "ci"),
"VM_DISK": here(d.get("image_path", "")),
"SSH_KEY_PRIVATE": here(d.get("ssh_keys", {}).get("private_key_path", "")),
}
for k, v in keys.items():
print(f"{k}={shlex.quote(str(v))}")
PYEOF
)"
chmod 600 "$SSH_KEY_PRIVATE"
{
echo "vm_user=$VM_USER"
echo "vm_disk=$VM_DISK"
echo "ssh_key=$SSH_KEY_PRIVATE"
} >> "$GITHUB_OUTPUT"
- name: Wait for deps
run: bash ci/ernic/background-task.sh wait deps
- name: Wait for the image
run: bash ci/ernic/background-task.sh wait images
- name: Extract rocm-ernic binary from ernic image
run: |
docker create --name ernic-tmp-2vm "$ERNIC_IMAGE"
# Not the workspace: it is mounted noexec in the job container.
docker cp ernic-tmp-2vm:/usr/local/bin/rocm-ernic /usr/local/bin/rocm-ernic
docker rm ernic-tmp-2vm
chmod +x /usr/local/bin/rocm-ernic
rocm-ernic --help > /tmp/ernic-help.txt
head -3 /tmp/ernic-help.txt
- name: Select QEMU accelerator and guest size
run: |
if [ -c /dev/kvm ] && "${QEMU_PATH:-}qemu-system-x86_64" -accel help 2>/dev/null | grep -qw kvm; then
echo "QEMU_ACCEL=kvm" >> "$GITHUB_ENV"
echo "QEMU_CPU=host" >> "$GITHUB_ENV"
else
echo "QEMU_ACCEL=tcg" >> "$GITHUB_ENV"
echo "QEMU_CPU=qemu64" >> "$GITHUB_ENV"
fi
HOST_MEM_MB=$(awk '/^MemTotal:/ {print int($2/1024)}' /proc/meminfo)
# IONIC_EQ_COUNT_MIN is 4: a guest with fewer online CPUs can never
# register the ibdev, so 4 is a floor and not a preference, even
# when that means oversubscribing the runner.
VM_VCPUS=$(( ($(nproc) - 1) / 2 )); [ "$VM_VCPUS" -lt 4 ] && VM_VCPUS=4
VM_MEM_MB=$(( HOST_MEM_MB * 3 / 4 / 2 ))
[ "$VM_MEM_MB" -lt 1024 ] && VM_MEM_MB=1024
[ "$VM_MEM_MB" -gt 4096 ] && VM_MEM_MB=4096
echo "VM_VCPUS=$VM_VCPUS" >> "$GITHUB_ENV"
echo "VM_MEM_MB=$VM_MEM_MB" >> "$GITHUB_ENV"
- name: Create the per-VM disk overlays
id: overlays
env:
VM_DISK: ${{ steps.vm_info.outputs.vm_disk }}
run: |
# Overlays, not copies. Both VMs get one including the first: with
# VM 1 writing to the base directly, VM 2's backing file would
# mutate underneath it.
BASE="${VM_DISK}"
STEM="$(dirname "$BASE")/$(basename "$BASE" .qcow2)"
for n in 1 2; do
DISK="${STEM}.${n}.qcow2"
rm -f "$DISK"
qemu-img create -f qcow2 -F qcow2 -b "$BASE" "$DISK" > /dev/null
echo "vm_disk_${n}=$DISK" >> "$GITHUB_OUTPUT"
qemu-img info "$DISK" | head -4
done
- name: Create the host bridge and per-instance TAPs
run: |
# ernic_host_setup owns this; run the role rather than open-coding
# `ip tuntap`. Order matters -- rocm-ernic treats a missing --tap
# as a startup failure, so the interfaces have to exist first.
PROJECT_ROOT="${GITHUB_WORKSPACE}/rocm-ernic"
export ANSIBLE_ROLES_PATH="${PROJECT_ROOT}/ansible/roles"
cd "${PROJECT_ROOT}/ansible"
ansible-playbook playbooks/host-setup.yml \
-e ernic_project_root="$PROJECT_ROOT" \
-e ernic_build=false \
-e ernic_install_service=false \
-e ernic_gpu_passthrough=false \
-e ernic_instances=2 \
-e ernic_tap_owner=root \
-e ernic_tap_bridge="${ERNIC_TAP_BRIDGE}" \
-e ernic_tap_prefix="${ERNIC_TAP_PREFIX}"
ip -o link show "${ERNIC_TAP_BRIDGE}"
for n in 1 2; do ip -o link show "${ERNIC_TAP_PREFIX}${n}"; done
- name: Start the two rocm-ernic instances
run: |
# --backend is the RDMA data path, --tap the Ethernet one; both are
# needed. --mac is not optional once two instances share a bridge:
# the emulator's built-in address is a constant, so without it both
# guests come up on the same MAC and ARP can never resolve.
for n in 1 2; do rm -f "/tmp/vfio-user-server${n}.sock"; done
rocm-ernic \
--socket /tmp/vfio-user-server1.sock \
--backend "tcp:manager:listen:${TCP_PORT}" \
--tap "${ERNIC_TAP_PREFIX}1" \
--mac 02:a0:d1:00:00:01 \
--verbose > /tmp/server1.log 2>&1 &
echo "ERNIC_PID1=$!" >> "$GITHUB_ENV"
for _ in $(seq 1 30); do
timeout 1 bash -c "echo > /dev/tcp/localhost/${TCP_PORT}" 2>/dev/null && break
sleep 1
done
timeout 1 bash -c "echo > /dev/tcp/localhost/${TCP_PORT}" 2>/dev/null || {
echo "manager never listened on ${TCP_PORT}"; cat /tmp/server1.log; exit 1; }
rocm-ernic \
--socket /tmp/vfio-user-server2.sock \
--backend "tcp:worker:localhost:${TCP_PORT}" \
--tap "${ERNIC_TAP_PREFIX}2" \
--mac 02:a0:d1:00:00:02 \
--verbose > /tmp/server2.log 2>&1 &
echo "ERNIC_PID2=$!" >> "$GITHUB_ENV"
for n in 1 2; do
for _ in $(seq 1 45); do
[ -S "/tmp/vfio-user-server${n}.sock" ] && break
sleep 1
done
[ -S "/tmp/vfio-user-server${n}.sock" ] || {
echo "server${n} socket never appeared"; cat "/tmp/server${n}.log"; exit 1; }
chmod 666 "/tmp/vfio-user-server${n}.sock"
done
- name: Verify the TCP mesh formed
run: |
for _ in $(seq 1 30); do
grep -q "TCP: Registered with manager" /tmp/server2.log && \
grep -q "TCP: Registered node" /tmp/server1.log && break
sleep 2
done
grep -q "TCP: Registered with manager" /tmp/server2.log || {
echo "worker never registered"
tail -40 /tmp/server1.log; tail -40 /tmp/server2.log; exit 1; }
grep -m1 "TCP: Registered node" /tmp/server1.log
grep -m1 "TCP: Received mesh topology" /tmp/server2.log || true
- name: Start the two QEMU VMs
env:
VM_DISK_1: ${{ steps.overlays.outputs.vm_disk_1 }}
VM_DISK_2: ${{ steps.overlays.outputs.vm_disk_2 }}
run: |
start_vm() {
n=$1; disk=$2; port=$3
"${QEMU_PATH:-}qemu-system-x86_64" \
-machine "q35,accel=${QEMU_ACCEL},memory-backend=mem${n}" \
-cpu "${QEMU_CPU}" -smp "${VM_VCPUS}" -m "${VM_MEM_MB}M" \
-object "memory-backend-memfd,id=mem${n},share=on,size=${VM_MEM_MB}M" \
-nographic \
-drive file="$disk",format=qcow2,if=virtio \
-netdev "user,id=net0,hostfwd=tcp::${port}-:22" \
-device virtio-net-pci,netdev=net0 \
-device '{"driver":"vfio-user-pci","socket":{"type":"unix","path":"/tmp/vfio-user-server'"$n"'.sock"}}' \
-serial "file:/tmp/qemu-vm${n}-console.log" -monitor none \
-qmp "unix:/tmp/qmp-vm${n}.sock,server=on,wait=off" > "/tmp/qemu-vm${n}.log" 2>&1 &
echo "QEMU_PID${n}=$!" >> "$GITHUB_ENV"
}
start_vm 1 "${VM_DISK_1}" "$VM1_SSH_PORT"
start_vm 2 "${VM_DISK_2}" "$VM2_SSH_PORT"
sleep 5
# The loop runs in a subshell: its exit fails the pipeline, and
# with it the step.
awk -F= '/^QEMU_PID/{print $2}' "$GITHUB_ENV" | while read -r pid; do
kill -0 "$pid" 2>/dev/null || { cat /tmp/qemu-vm1.log /tmp/qemu-vm2.log; exit 1; }
done
- name: Wait for both VMs over SSH
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
for port in "$VM1_SSH_PORT" "$VM2_SSH_PORT"; do
for _ in $(seq 1 60); do
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-o ConnectTimeout=5 -p "$port" "${VM_USER}@localhost" true \
2>/dev/null && break
sleep 5
done
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-p "$port" "${VM_USER}@localhost" echo "VM_READY on $port" || {
cat "/tmp/qemu-vm1-console.log" "/tmp/qemu-vm2-console.log"; exit 1; }
done
- name: Disable the guest's amdgpu driver apt sources
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
# The guest image's build added repo.radeon.com/amdgpu/31.60 as an
# apt source to install amdgpu-dkms, and that tree has since
# disappeared from repo.radeon.com. Nothing here installs from it --
# the driver is baked into the image -- but every apt-get update in
# the guest, the ernic Ansible collection's included, fails on its
# missing Release file. Mark the stanza "Enabled: no" rather than
# deleting it, so the change is visible from inside the guest. On an
# image without the source this does nothing; remove it once
# GUEST_ARTIFACT_TAG names such an image.
for port in "$VM1_SSH_PORT" "$VM2_SSH_PORT"; do
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-p "$port" "${VM_USER}@localhost" bash -s << 'EOFVM'
set -euo pipefail
shopt -s nullglob
for f in /etc/apt/sources.list.d/*.sources; do
grep -q 'repo\.radeon\.com/amdgpu/' "$f" || continue
tmp=$(mktemp)
# deb822 field names are case-insensitive, and the stanza may
# already carry an Enabled field (possibly "yes"): rewrite that
# one to "no", and append the field only when there is none.
awk 'BEGIN { RS = ""; ORS = "\n\n" }
/repo\.radeon\.com\/amdgpu\// {
n = split($0, line, "\n"); s = ""; found = 0
for (i = 1; i <= n; i++) {
if (line[i] ~ /^[Ee][Nn][Aa][Bb][Ll][Ee][Dd][ \t]*:/) {
line[i] = "Enabled: no"; found = 1
}
s = s (i > 1 ? "\n" : "") line[i]
}
$0 = found ? s : s "\nEnabled: no"
}
{ print }' "$f" > "$tmp"
sudo install -m 0644 "$tmp" "$f"
rm -f "$tmp"
echo "Disabled the amdgpu driver apt source in $f"
done
EOFVM
done
- name: Install kernel headers and build tools in both VMs
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
for port in "$VM1_SSH_PORT" "$VM2_SSH_PORT"; do
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-p "$port" "${VM_USER}@localhost" bash -s << 'EOFVM'
set -e
KVER=$(uname -r)
# A guest on a mainline kernel already has its headers from the
# .deb, and kernel.ubuntu.com has no Packages index behind it.
HEADER_PKG=""
[ -d "/lib/modules/${KVER}/build" ] || HEADER_PKG="linux-headers-${KVER}"
APT_LOCK_WAIT="-o DPkg::Lock::Timeout=300"
for i in 1 2 3; do sudo apt-get $APT_LOCK_WAIT update -qq && break || sleep 5; done
for i in 1 2 3 4 5; do
sudo DEBIAN_FRONTEND=noninteractive apt-get $APT_LOCK_WAIT install -y -qq \
${HEADER_PKG} dkms git build-essential cmake ninja-build \
pkg-config libnl-3-dev libnl-route-3-dev libudev-dev && break
[ "$i" -eq 5 ] && { echo "apt-get install failed after 5 attempts"; exit 1; }
sleep 10
done
echo "Kernel packages installed for ${KVER}"
EOFVM
done
- name: Stage the ionic driver sources in both VMs
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
IONIC_SRC: ${{ steps.ionic_sources.outputs.source_dir }}
IONIC_REF: ${{ steps.ionic_sources.outputs.ref }}
run: |
# Left to itself, the guest role sparse-clones the kernel from
# git.kernel.org through user-mode NAT in each guest, ~100 s every
# run. It skips that when .patches-applied-<ref> is already in
# place, so copy in the runner's cached tree; provisioning is
# given the same directory and ref below.
set -o pipefail
for port in "$VM1_SSH_PORT" "$VM2_SSH_PORT"; do
tar -C "$IONIC_SRC" -cf - . | \
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-p "$port" "${VM_USER}@localhost" \
"sudo rm -rf '${IONIC_GUEST_DIR}' && sudo mkdir -p '${IONIC_GUEST_DIR}' && \
sudo tar -C '${IONIC_GUEST_DIR}' -xf - && \
test -f '${IONIC_GUEST_DIR}/.patches-applied-${IONIC_REF}'"
done
echo "ionic sources at ${IONIC_REF} staged in ${IONIC_GUEST_DIR}"
- name: Provision both VMs with the ernic Ansible collection
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
IONIC_REF: ${{ steps.ionic_sources.outputs.ref }}
run: |
ERNIC_RUN_DIR=/tmp/ernic-ci
mkdir -p "$ERNIC_RUN_DIR"
cat > "$ERNIC_RUN_DIR/instances.json" << EOFJSON
{
"instances": [
{ "id": 1, "vm": { "vm_name": "hipobj-bridge-1",
"ssh_port": ${VM1_SSH_PORT}, "ssh_user": "${VM_USER}" } },
{ "id": 2, "vm": { "vm_name": "hipobj-bridge-2",
"ssh_port": ${VM2_SSH_PORT}, "ssh_user": "${VM_USER}" } }
]
}
EOFJSON
export ANSIBLE_HOST_KEY_CHECKING=false
export ANSIBLE_PRIVATE_KEY_FILE="$SSH_KEY"
PROJECT_ROOT="${GITHUB_WORKSPACE}/rocm-ernic"
export ANSIBLE_ROLES_PATH="${PROJECT_ROOT}/ansible/roles"
cd "${PROJECT_ROOT}/ansible"
ansible-playbook ci-site.yml --tags guest-setup \
-e ernic_project_root="$PROJECT_ROOT" \
-e ernic_run_dir="$ERNIC_RUN_DIR" \
-e ernic_instances=2 \
-e ernic_vm_ssh_base_port="$VM1_SSH_PORT" \
-e ernic_vm_ssh_user="$VM_USER" \
-e ernic_ionic_source_dir="$IONIC_GUEST_DIR" \
-e ernic_ionic_kernel_ref="$IONIC_REF" \
-e ernic_gpu_passthrough=false \
-e ernic_guest_agent=false \
-e ernic_set_hostname=false
- name: Verify the RDMA device in both VMs
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
vm_ssh() {
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no "$@"
}
for spec in "1:${VM1_SSH_PORT}" "2:${VM2_SSH_PORT}"; do
n=${spec%%:*}; port=${spec##*:}
# By PCI vendor ID, never by name: the kernel registers ionic_%d
# and udev rewrites it twice.
DEV=$(vm_ssh -p "$port" "${VM_USER}@localhost" 'sh -s' \
< "${GITHUB_WORKSPACE}/rocm-ernic/scripts/find-rdma-device.sh" | tr -d '\r') || true
[ -n "$DEV" ] || {
echo "VM ${n}: no RDMA device"
vm_ssh -p "$port" "${VM_USER}@localhost" \
'ibv_devices; lsmod | grep ionic; dmesg | grep -i ionic | tail -30' || true
exit 1
}
vm_ssh -p "$port" "${VM_USER}@localhost" "ibv_devinfo -d $DEV" \
| grep -q PORT_ACTIVE || { echo "VM ${n}: $DEV not PORT_ACTIVE"; exit 1; }
echo "VM ${n}: $DEV is PORT_ACTIVE"
echo "RDMA_DEVICE_${n}=$DEV" >> "$GITHUB_ENV"
done
- name: Wait for the build
run: bash ci/ernic/background-task.sh wait build
- name: Copy hipObject binaries into both VMs
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
BUILD="${GITHUB_WORKSPACE}/ernic-build"
vm_scp() {
scp -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no "$@"
}
for port in "$VM1_SSH_PORT" "$VM2_SSH_PORT"; do
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no -p "$port" \
"${VM_USER}@localhost" \
'mkdir -p /tmp/hipobject-build/test/integration/rdma-test-server /tmp/hipobject-build/integrations/minio-cpp'
vm_scp -P "$port" -r "${BUILD}/rocm-libs" "${VM_USER}@localhost:/tmp/hipobject-build/rocm-libs"
vm_scp -P "$port" "${BUILD}/test/integration/rdma-test-server/hipobj-rdma-test-server" \
"${VM_USER}@localhost:/tmp/hipobject-build/test/integration/rdma-test-server/"
vm_scp -P "$port" "${BUILD}/integrations/minio-cpp/minio-getput-rdma" \
"${VM_USER}@localhost:/tmp/hipobject-build/integrations/minio-cpp/"
vm_scp -P "$port" ci/ernic/server-entrypoint.sh \
ci/ernic/minio-client-entrypoint.sh \
ci/ernic/assert-transfer.sh \
"${VM_USER}@localhost:/tmp/hipobject-build/"
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no -p "$port" \
"${VM_USER}@localhost" 'chmod +x /tmp/hipobject-build/*.sh \
/tmp/hipobject-build/integrations/minio-cpp/* \
/tmp/hipobject-build/test/integration/rdma-test-server/*'
done
- name: Start hipobj-rdma-test-server in VM 1
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
vm_ssh() {
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no "$@"
}
# systemd-run, not `nohup ... &` over ssh: a backgrounded remote
# command leaves nothing to ask about afterwards. If it dies at
# startup the only evidence is an empty log file, which is exactly
# how the first run of this lane failed.
vm_ssh -p "$VM1_SSH_PORT" "${VM_USER}@localhost" \
"sudo systemd-run --unit=hipobj-server \
--setenv=HIPOBJ_NIC_HINT='${RDMA_DEVICE_1}' \
--setenv=SERVER_PORT='${SERVER_PORT}' \
/tmp/hipobject-build/server-entrypoint.sh"
dump_server() {
echo "=== VM 1: systemctl status hipobj-server ==="
vm_ssh -p "$VM1_SSH_PORT" "${VM_USER}@localhost" \
'systemctl status hipobj-server --no-pager -l || true
sudo journalctl -u hipobj-server --no-pager | tail -60
ss -ltnp 2>/dev/null | head -20
ip -4 -br addr show' || true
}
# From VM 2, not from VM 1: the point of the probe is that the
# listener is reachable across the emulated wire.
for _ in $(seq 1 45); do
vm_ssh -p "$VM2_SSH_PORT" "${VM_USER}@localhost" \
"timeout 2 bash -c 'echo > /dev/tcp/${VM1_ADDR}/${SERVER_PORT}'" \
2>/dev/null && { echo "server reachable at ${VM1_ADDR}:${SERVER_PORT}"; break; }
# No point waiting out the loop once the unit has given up.
vm_ssh -p "$VM1_SSH_PORT" "${VM_USER}@localhost" \
'systemctl is-active --quiet hipobj-server' || {
echo "hipobj-server is no longer running"; dump_server; exit 1; }
sleep 2
done
vm_ssh -p "$VM2_SSH_PORT" "${VM_USER}@localhost" \
"timeout 2 bash -c 'echo > /dev/tcp/${VM1_ADDR}/${SERVER_PORT}'" || {
echo "VM 2 cannot reach the test server"
dump_server
exit 1
}
- name: Run the minio-cpp bridge client in VM 2
timeout-minutes: 10
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
# EXPECT_TRANSPORT=rdma is the whole point of the VM conversion: the
# bridge falls back to HTTP silently, so without this assertion a
# lane that lost its wire still passes.
# Bounded at three layers: guest-side timeout, ssh keepalive, step
# timeout. This lane passes today, but its twin in the Two-VM GPU
# Direct workflow wedged for the full 120-minute job budget on the
# same unbounded ssh -- the bound belongs on both.
ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=4 \
-p "$VM2_SSH_PORT" "${VM_USER}@localhost" \
"timeout 420 env HIPOBJ_NIC_HINT='${RDMA_DEVICE_2}' \
BUILD_DIR=/tmp/hipobject-build \
SERVER_ENDPOINT='http://${VM1_ADDR}:${SERVER_PORT}' \
TEST_SIZE='${TEST_SIZE}' \
GPU_MODE=nogpu \
EXPECT_TRANSPORT=rdma \
/tmp/hipobject-build/minio-client-entrypoint.sh"
- name: Collect logs
if: always()
env:
SSH_KEY: ${{ steps.vm_info.outputs.ssh_key }}
VM_USER: ${{ steps.vm_info.outputs.vm_user }}
run: |
for n in 1 2; do
echo "=== server${n}.log ==="; tail -100 "/tmp/server${n}.log" || true
echo "=== qemu-vm${n}.log ==="; cat "/tmp/qemu-vm${n}.log" || true
echo "=== qemu-vm${n}-console ==="; tail -200 "/tmp/qemu-vm${n}-console.log" || true
done
echo "=== VM 1 hipobj-server journal ==="
timeout 60 ssh -i "$SSH_KEY" -o BatchMode=yes -o StrictHostKeyChecking=no \
-o ConnectTimeout=10 -p "$VM1_SSH_PORT" "${VM_USER}@localhost" \
'sudo journalctl -u hipobj-server --no-pager | tail -100' || true
- name: Teardown
if: always()
run: |
kill "${QEMU_PID1:-}" "${QEMU_PID2:-}" 2>/dev/null || true
kill "${ERNIC_PID1:-}" "${ERNIC_PID2:-}" 2>/dev/null || true
# Still running if the job failed before "Wait for the build".
docker rm -f hipobj-build 2>/dev/null || true
docker run --rm -v "${{ github.workspace }}/ernic-build:/d" \
busybox sh -c "rm -rf /d/*" 2>/dev/null || true
rm -rf "${{ github.workspace }}/ernic-build" 2>/dev/null || true