Container Toolkit messed up config.toml in GKE cluster with containerd 2.0

[quoctruong]

I've been using the GPU Operator to install Nvidia driver in my GKE cluster. This has worked fine until last week when the cluster version is upgraded from GKE 1.32 to 1.33. One of the big change is that the containerd version is now 2.0. Now, a lot of GPU operator Daemonset is failing with the following errors:

"failed to \"CreatePodSandbox\" for \"gpu-feature-discovery-nv7f9_gpu-operator(b07a2405-29b1-412e-8400-a1367a10e76d)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"gpu-feature-discovery-nv7f9_gpu-operator(b07a2405-29b1-412e-8400-a1367a10e76d)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"2b62cb2980cd477cefbea1155871bcd2f50b2c6fb3e922039554d8b3fc14784f\\\": plugin type=\\\"gke\\\" failed (add): failed to find plugin \\\"gke\\\" in path [/opt/cni/bin]\"

So we went and looked at the config.toml file and it looks like it was modified and added an incorrect line of config:

    [plugins."io.containerd.cri.v1.runtime".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"

This is most likely because of the nvidia-container-toolkit-daemonset because it was the last pod to be able start successfully (and also it looks like the toolkit modifies this file?).

The toolkit version is from the container nvcr.io/nvidia/k8s/container-toolkit:v1.17.8-ubuntu20.04 and the GPU operator is version 25.3.2 (the latest one).

Any insight on what is happening and why the config.toml was modified incorrectly for containerd 2.0?

[carreter]

Seems like the root cause of this is containerd/containerd#11747

nvidia-container-toolkit uses containerd config dump to get the containerd config by default, even if you specify the file path for the config.toml. This command only partially migrates the output from v2 to v3 and messes things up in the progress.

Easy fixes would be to either have nvidia-container-toolkit prefer reading config from file over containerd config dump, or using containerd config migrate instead which properly does the v2 -> v3 migration.

[klueska]

@ArangoGutierrez, @elezar can one of you please comment. I believe you have been working on something to resolve this.

[elezar]

We have started discussions on how to make this more robust in general, but there is nothing concrete in this version of the toolkit. I will have a look at the implementation and see if I can propose a workaround -- or alternatively accelerate a fix for this.

[carreter]

We have started discussions on how to make this more robust in general, but there is nothing concrete in this version of the toolkit. I will have a look at the implementation and see if I can propose a workaround -- or alternatively accelerate a fix for this.

AFAICT, two workarounds available are:

1. Switch lines 176 and 177 here, so that nvidia-ctk prefers pulling containerd config from file: https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/cmd/nvidia-ctk-installer/container/runtime/containerd/containerd.go#L174-L179
2. Run the migrate subcommand of containerd config instead of dump here: https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/pkg/config/engine/containerd/containerd.go#L169

[elezar]

Thanks @carreter. Since those require a release, I don't see them as "workarounds" per se'. With that said, I think your assessment is correct.

My first thought would be to release a version with the ability for a user to set the config source(s) as we do for the nvidia-ctk runtime configure command:

nvidia-container-toolkit/cmd/nvidia-ctk/runtime/configure/configure.go

Lines 132 to 137 in 7b1b20b

	&cli.StringFlag{
	Name: "config-source",
	Usage: "the source to retrieve the container runtime configuration; one of [command, file]\"",
	Destination: &config.configSource,
	Value: defaultConfigSource,
	},

With that in place we can look into allowing other options such as containerd config migrate instead of containerd config dump as a follow-up.

As @klueska was pointing out, we are working on making this more robust and that should be ready for the next operator feature release.

[elezar]

I have created #1251 as a simple demonstrator for exposing this setting in the toolkit container.

#1252 backports these changes to the release-1.17 branch and the following image could be used to verify the changes: ghcr.io/nvidia/container-toolkit:8334ddec-ubuntu20.04

Note that you would have to set RUNTIME_CONFIG_SOURCES=file explicitly.

[elezar]

@quoctruong would you be able to validate the image mentioned above?

[quoctruong]

I tested with the container image and set RUNTIME_CONFIG_SOURCES=file directly and can verify that it works. So is the workaround container ghcr.io/nvidia/container-toolkit:8334ddec-ubuntu20.04 or will you be releasing another one?

[elezar]

One thing that I just realised may be a workaround in this case (effectively disabling the use of the containerd config dump command) is to set the RUNTIME_EXECUTABLE_PATH envvar to a non-existent executable.

When set to "" (default), this is set to "containerd" here. Setting this to something like "/usr/bin/not-containerd" will force the config file to be used as a source since the /usr/bin/not-containerd config dump command will fail.

[elieserr]

+1 on this issue on GKE, I got plugin type="ptp" failed (add): failed to find plugin "ptp" in path [/opt/cni/bin]
The toolkit image used by the operator is nvcr.io/nvidia/k8s/container-toolkit:v1.17.8-ubuntu20.04

Commented for visibility, will be trying the workarounds