rhel9-ppc64le VMs offline due to NVMe failure at OSUOSL

[richardlau]

Got this email from OSUOSL:

One of our nodes which uses local storage with NVMe has had a failure and all the VMs on that node are offline. Due to the way we had storage configured on this node for performance, we're unlikely to save any of the ephemeral disks. However if you had any volumes attached, those will still be intact since they are on our Ceph cluster.

I'm going to be working on getting the NVMe replaced, in the meantime, if you need us to rebuild any of your VMs, please let us know.

This affects our rhel9-ppc64le VMs, which I had set up with NVMe storage as they had resulted in marginally faster builds:

• test-osuosl-rhel9-ppc64_le-1 is DOWN jenkins-alerts#3262
• test-osuosl-rhel9-ppc64_le-2 is DOWN jenkins-alerts#3263
• test-osuosl-rhel9-ppc64_le-3 is DOWN jenkins-alerts#3264

[mhdawson]

Removed rhel9-ppc64le from these jobs:

• https://ci.nodejs.org/job/node-test-commit-plinux/configure until we get the machines restored.

@richardlau said he'd take a look tomorrow.

[mhdawson]

There are a couple of other jobs that use rhel9-ppc64le, but they have not runs for 5-6 days so they are more infrequent so I've not disabled them yet.

I think we'll get the machine back tomorrow, if not we can exclude those as well:

• https://ci.nodejs.org/job/citgm-smoker/nodes=rhel9-ppc64le/
• https://ci.nodejs.org/job/citgm-smoker-nobuild/nodes=rhel9-ppc64le/

[richardlau]

No updates on the NVMe replacement. I'm going to create new (non-NVMe) VMs.

[richardlau]

I've created a new test-osuosl-rhel9-ppc64_le-4 and replaced test-osuosl-rhel9-ppc64_le-3 (Ansible inventory update in #3999) with non-NVMe RHEL 9 VMs. That will give us two machines for now.

My plan is to wait to see what the outlook is for getting the NVMe replaced before deciding what to do about test-osuosl-rhel9-ppc64_le-1 and test-osuosl-rhel9-ppc64_le-2.

[richardlau]

I've updated node-test-commit-plinux to restore rhel9-ppc64le.

CI runs:

• test-osuosl-rhel9-ppc64_le-3 - https://ci.nodejs.org/job/node-test-commit-plinux/57237/nodes=rhel9-ppc64le/
• test-osuosl-rhel9-ppc64_le-4 - https://ci.nodejs.org/job/node-test-commit-plinux/57236/nodes=rhel9-ppc64le/

[mhdawson]

@richardlau thanks for the quick work.