• Applying this fix to the test cluster involves disabling auto-sync for the cluster-scope-test and nvidia-gpu-operator-test ArgoCD applications.
• Delete the GPU ClusterPolicy and MIG ConfigMap oc --as system:admin delete -k nvidia-gpu-operator/overlays/nerc-ocp-test/
• Delete the NVIDIA GPU Operator oc --as system:admin delete -k cluster-scope/bundles/nvidia-gpu-operator/
• Re-enable auto-sync for the cluster-scope-test, and run a Sync.
• Re-enable auto-sync for the nvidia-gpu-operator-test, and run a Sync.
• Check for the NVIDIA pods to start Running oc get pods -n nvidia-gpu-operator

Another update I recevied is because we set up the validator plugin, which is by default is set to false in the clusterPolicy:

  validator:
    plugin:
      env:
      - name: WITH_WORKLOAD
        value: "true"

We need this workaround to use volume mounts because of the validator pods is not running as privileged.

• kruize
• nerc-ocp-test-2.nerc.mghpcc.org
  applied the changes manually, and it works for them now again

Since this PR was merged, and we still have a pod failing on wrk-97, And currently there is 0 GPU utilization in the prod cluster, I'm going to proceed with uninstalling the NVIDIA GPU Operator and reinstalling it in the prod cluster to finish resolving this issue.

$ oc -n nvidia-gpu-operator get pod/nvidia-operator-validator-nj8hx
NAME                              READY   STATUS                  RESTARTS          AGE
nvidia-operator-validator-nj8hx   0/1     Init:CrashLoopBackOff   1046 (2m4s ago)   5d2h

Sorry again for kicking off this update of the NVIDIA operator in prod. I think these are not restarting because draining the node is blocked:
https://console.apps.shift.nerc.mghpcc.org/api/kubernetes/api/v1/namespaces/nvidia-gpu-operato[…]2407191425-0-4s8cl/log?container=k8s-driver-manager

Auto drain of the node wrk-103 is disabled by the upgrade policy
Failed to uninstall nvidia driver components
Auto eviction of GPU pods on node wrk-103 is disabled by the upgrade policy
Auto drain of the node wrk-103 is disabled by the upgrade policy
Rescheduling all GPU clients on the current node by enabling their component-specific nodeSelector labels
node/wrk-103 labeled

Running with these gpu-cluster-policy settings is allowing the drivers to be installed on 9/11 nodes so far.

    manager:
      env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "true"
      - name: DRAIN_USE_FORCE
        value: "true"
      - name: DRAIN_DELETE_EMPTYDIR_DATA
        value: "true"
    repoConfig:
      configMapName: ""
    upgradePolicy:
      autoUpgrade: false
      drain:
        deleteEmptyDir: true
        enable: true
        force: true
        timeoutSeconds: 300
      maxParallelUpgrades: 1
      maxUnavailable: 25%
      podDeletion:
        deleteEmptyDir: true
        force: true
        timeoutSeconds: 300