Use HostToContainer mount propagation instead of /host/proc/mounts

[jiaeenie]

Description of changes:
Currently, we spawn Mountpoint processes on the host using systemd. As a result, the mounts created by Mountpoint are not visible inside the CSI Driver Pod. To work around this, we were mounting /proc/mounts from host and parsing this file to check existing mounts on the host. Mounting /proc/mounts causes problems with Karpenter sometimes and its also its blocked by some SELinux policies. Such as this issue.

This commit instead uses HostToContainer mount propagation on hostPath mount for /var/lib/kubelet. Thanks to HostToContainer, any new mounts created inside /var/lib/kubelet gets automatically propagated to our Pod from the host.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[unexge]

The change looks good to me. One additional thing we should do is to ensure targetPath passed in NodePublishVolume is inside kubelet path. Because we rely on propagation of mounts under kubelet path, any other mount wouldn't be propagated to our Pod. This shouldn't be the case, but it would be good to emit an error just in case. Something like that in S3NodeServer.NodePublishVolume:

const defaultKubeletPath = "/var/lib/kubelet"

var kubeletPath = getKubeletPath()

func getKubeletPath() string {
	path := os.Getenv("KUBELET_PATH")
	if path == "" {
		return defaultKubeletPath
	}
	return path
}

// ...

func (ns *S3NodeServer) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error) {
        // ...

	if !filepath.HasPrefix(target, kubeletPath) {
		klog.Errorf("NodePublishVolume: target path %q is not in kubelet path %q. This might cause mounting issues, please ensure you have correct kubelet path configured.", req.GetTargetPath(), kubeletPath)
	}
}

[unexge]

The CI failure is unrelated, "Delete cluster" step doesn't handle "NotFound" errors currently and tries to remove some resources for maximum number of retries configured which takes around ~20min and that causes temporary credentials to expire and operations to fail.

I've restarted the failed tasks and will submit a separate PR to fix the CI.

[unexge]

LGTM. Thanks @jiaeenie!

[UnyieldingOrca]

Can you make a patch release for this fix? Or is there a way to deploy a prerelease version of the csi driver myself?

[unexge]

Hey @UnyieldingOrca, we plan to do a release soon, but we don't have any date we can share at the moment. We'll be sharing updates in this ticket: #284.

Regarding deploying a pre-release version, you'd first need to build our Docker image by using our Makefile, and push it to your registry, and then install our Helm chart from source by specifying your registry.

helm upgrade --install aws-mountpoint-s3-csi-driver \
    --namespace kube-system \
    ... \
    --set image.repository="1234567890.dkr.ecr.eu-north-1.amazonaws.com/your-registry" \
    --set image.tag=your-tag \
   ./charts/aws-mountpoint-s3-csi-driver

Unfortunately, it's not trivial at the moment.

[UnyieldingOrca]

Thanks for the guidance. I ended up following this and it fixed my issue.