controller: run with `NVIDIA_VISIBLE_DEVICES=void`

[jgehrcke]

The controller

has no dependence on GPUs and is forced to run on a control-plane node (preferably without GPUs).

When someone however is

running on a node with the nvidia-container-runtime installed and set as the default runtime for containerd (even though there are no GPUs on the node)

then it can happen that

the nvidia-container-runtime will trigger the nvidia-container-cli which, in turn, will attempt to load NVML to look for „all“ GPUs on the node (which will then fail for obvious
reasons).

That's not a system configuration that we expect, and users might see something like

runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown

which may be difficult to debug.

For a better user experience, @klueska noted that we should set

NVIDIA_VISIBLE_DEVICES=void in our helm template for the controller.

(quotes from Slack).

[elezar]

It could also be argued that the NVIDIA Container Runtime should be made more robust to systems where there is no driver installed (e.g. non-GPU nodes).

In the context of the GPU Operator we have the following check to ensure that the runtime is only invoked on nodes where the driver is active (the GPU Operator further only deploys the toolkit on nodes which have the required PCI devices present).

I have the following PR that would allow the nvidia-container-runtime to act as a no-op on non-GPU systems (or GPU systems where the kernel modules are not persent): NVIDIA/nvidia-container-toolkit#764

[klueska]

I just discovered this is also now an issue (in some environments) with our compute-domain-daemon as well. So fixing it more centrally would be great.