Allow config sources to be specified for containerd and cri-o

[elezar]

By default container extracts the current config by running the containerd config dump command and if that fails we read the existing config file.

This change allows ordering of these sources to be defined as we do for the nvidia-ctk runtime configure command.

See #1222 where setting RUNTIME_CONFIG_SOURCES=file or RUNTIME_CONFIG_SOURCES=file,command should allow the current config to be preferred over the containerd config dump command.

This further allows an explicit path to be specified for the file config source so that there is a separation between the config file being modified by the tooling and the config used as the source for drop-in files, for example.

[coveralls]

Pull Request Test Coverage Report for Build 18193238472

Details

• 16 of 28 (57.14%) changed or added relevant lines in 2 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage increased (+0.03%) to 37.21%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
cmd/nvidia-ctk-installer/container/runtime/containerd/containerd.go	16	22	72.73%
cmd/nvidia-ctk-installer/container/runtime/runtime.go	0	6	0.0%

Files with Coverage Reduction	New Missed Lines	%
cmd/nvidia-ctk-installer/container/runtime/runtime.go	1	0.0%

Totals
Change from base Build 18191171126:	0.03%
Covered Lines:	4972
Relevant Lines:	13362

---

💛 - Coveralls

[elezar]

Note to self: Since command here implies containerd config dump command we may want to make this more expressive. For now I have used the values that we support in the nvidia-ctk runtime configure command:

nvidia-container-toolkit/cmd/nvidia-ctk/runtime/configure/configure.go

Lines 132 to 137 in 7b1b20b

	&cli.StringFlag{
	Name: "config-source",
	Usage: "the source to retrieve the container runtime configuration; one of [command, file]\"",
	Destination: &config.configSource,
	Value: defaultConfigSource,
	},

[ArangoGutierrez]

Flag to be called config-source-order (or similar)

specify the order of config sources

As the user can do --config-source command,file or --config-source file,command

same for the ENV VAR RUNTIME_CONFIG_SOURCES_ORDER

[elezar]

I don't agree that the flag name should be changed. The flag specifies the sources and the ordering is implied by the order in which the sources are specified. Valid values for this argument are currently:

• ""
• "file"
• "command"
• "command,file" (Default)
• "file,command"

[ArangoGutierrez]

Then let's expand the Usage to be more explicit, "specify the config sources [file, command]" as nvidia-ctk runtime configure has a flag with the same name but this one is to point to a path. We should provide guidance to the user via Usage with at least the valid options or the default behav of this flag if not set.

[ArangoGutierrez]

Can we mark this PR as ready for review?

[elezar]

Can we mark this PR as ready for review?

I think we can move this to the next minor (if we decide to implement it at all). See the comment here #1222 (comment) for a workaround in the existing codebase.

[cdesiniotis]

I am testing this with the operator and microk8s, by setting the following with the toolkit container:

env:
    - name: CONTAINERD_CONFIG
      value: /var/snap/microk8s/current/args/containerd-template.toml
    - name: CONTAINERD_SOCKET
      value: /var/snap/microk8s/common/run/containerd.sock
    - name: RUNTIME_CONFIG_SOURCE
      value: file=/var/snap/microk8s/current/args/containerd.toml

This does not work as is since the path provided to RUNTIME_CONFIG_SOURCE is the path to the file on the host and NOT in the container.

[elezar]

I am testing this with the operator and microk8s, by setting the following with the toolkit container:

env:
    - name: CONTAINERD_CONFIG
      value: /var/snap/microk8s/current/args/containerd-template.toml
    - name: CONTAINERD_SOCKET
      value: /var/snap/microk8s/common/run/containerd.sock
    - name: RUNTIME_CONFIG_SOURCE
      value: file=/var/snap/microk8s/current/args/containerd.toml

This does not work as is since the path provided to RUNTIME_CONFIG_SOURCE is the path to the file on the host and NOT in the container.

Yes, I missed this when trying to think this through. The user could provide the container path here, but that's definitely not ideal.

[elezar]

@cdesiniotis what if we also add the following to the GPU Operator:

	if runtimeConfigSources := getContainerEnv(container, "RUNTIME_CONFIG_SOURCE"); runtimeConfigSources != "" {
		var sources []string
		for _, runtimeConfigSource := range strings.Split(runtimeConfigSources, ",") {
			parts := strings.SplitN(runtimeConfigSource, "=", 2)
			if len(parts) == 1 || parts[0] != "file" {
				sources = append(sources, runtimeConfigSource)
				continue
			}
			sourceConfigFile := parts[1]
			containerConfigFile := sourceConfigFile
			if sourceConfigFile != "" {
				sourceConfigFileName := path.Base(sourceConfigFile)
				sourceConfigDir := path.Dir(sourceConfigFile)

				// TODO: If the source config file is not in the same folder as
				// the top-level config file, then we need to use a different
				// container path here.
				containerConfigDir := DefaultRuntimeConfigTargetDir
				containerConfigFile = containerConfigDir + sourceConfigFileName

				// If we've already mounted sourceConfigDir -> containerConfigDir
				// then we don't create a new volume mount.
				if hostToContainerDirs[sourceConfigDir] != "" {
					volMountConfigName := fmt.Sprintf("%s-source-config", runtime)
					volMountConfig := corev1.VolumeMount{Name: volMountConfigName, MountPath: containerConfigDir}
					container.VolumeMounts = append(container.VolumeMounts, volMountConfig)

					configVol := corev1.Volume{Name: volMountConfigName, VolumeSource: corev1.VolumeSource{HostPath: &corev1.HostPathVolumeSource{Path: sourceConfigDir, Type: newHostPathType(corev1.HostPathDirectoryOrCreate)}}}
					obj.Spec.Template.Spec.Volumes = append(obj.Spec.Template.Spec.Volumes, configVol)
				}
			}
			sources = append(sources, "file="+containerConfigFile)
		}
		setContainerEnv(container, "RUNTIME_CONFIG_SOURCE", strings.Join(sources, ","))
	}

(We can definitely factor out some logic here to make things more readable across the multiple instances where we are transforming host paths to container paths).

This would mean that a user would specify file=/path/on/the/host/config.toml as the source and the path will be updated in the same way as other paths are treated.

[cdesiniotis]

A simpler solution would be to prepend /host to the file path -- as the toolkit container already has read-only access to the host's rootfs at /host. This would avoid us potentially having to add an additional volume. I have raised NVIDIA/gpu-operator#1758 with this proposed change.

[cdesiniotis]

I've tested this PR paired with NVIDIA/gpu-operator#1758 and it provides a working solution for microk8s.