Why do some k3s clusters consume excess of snapshots ?

[srajappa]

Environment Details

Been managing several k3s clusters on our hosts. Each of these clusters is allocated CPU/Memory. The K3s cluster nodes/servers are running as docker containers. These agents / servers do have persistent storage on the underlying host. The container runtime for the k3s clusters is containerd.

Version info:

docker : 23.0.2, build 569dd73
crictl: v1.26.0-rc.0-k3s1
k3s: v1.28.5+k3s1 (5b2d1271)

Expected Behavior

Clusters shouldn't overwhelm filesystem by storing multiple progressively heavy snapshots.

Observed Behavior

Some clusters tend to house multiple heavy snapshots in the following location:

/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native/snapshots

The snapshots-directory contains numbered directories with lowest number contained a few kbs of data to higher numbered directories containing Gbs worth of data.

To avoid overwhelmed/full disk we restart the cluster-agents. After agents rejoin the k3s cluster, the pods come up very slowly and the snapshots in the new k3s agents fill up, yet again.

Question ?

• How does containerd manage to fill up snapshots, what does it include in snapshots. Can we manipulate/remove/structure them any differently to optimize on storage ?
• What contribute to filling consuming excess of snapshots, do difference in cluster pod-composition lead to snapshot excesses ?

[brandond]

/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native/snapshots

This is really more of a containerd question, not k3s...

This is where containerd stores filesystems for running containers when using the native snapshotter. This is an inefficient snapshotter; generally the overlayfs snapshotter is preferred, but that requires an underlying filesystem that supports overlayfs, and it sounds like you've configured containerd to use the native snapshotter instead.

You can reference the upstream docs for more information: https://github.com/containerd/containerd/blob/main/docs/snapshotters/README.md

[srajappa]

Thank you boss! this is great help, will take it from here.