Why is the storage space consumption so high?

[kgts23]

Hello,

Today I noticed that a ceph pool (only used by the csi driver) is almost full. The problem is, I can't explain why. There is a data amount of ~2 TB net, the pool has a sizing of 3, so ~2 TB * 3 = ~6TB gross, which should be the expected amount. But the pool has a sizing of ~29TB gross, actually.

I tried to find some stale data, even if all storage classes use the delete policy. But I can't retrieve the omap keys/values as described in the cleanup doc: rados listomapkeys csi.volumes.default -p pool_name | grep pv_name. I only get the error message error getting omap key set hde-stage_fs_data/csi.volumes.default: (2) No such file or directory. I also realized that I may need to change the suffix of csi.volumes.default, but I didn't set an instance ID, so “default” should be sufficient.

While troubleshooting, I found that using rados df under “CLONES”, I saw a “3” for the specific pool, while all other pools had “0”. Then I ran a rados lssnap -p hde-stage_fs_data, but the result was not as expected: “0 snaps”. Unfortunately, I can't find a good online reference for the “CLONES” line in rados df, so I'm not quite sure if this is the actual number of clones or if this value just counts all clones all the time or something else.

The log files for the CSI driver are also unremarkable. I have the feeling that the deletion in the CephFS pool apparently did not work correctly and reliably in the past. The Ceph-rbd-driver is also running in the same K8s cluster, which in turn uses an RBD pool in the same Ceph cluster, here I see no anomalies in the storage space utilization.

edit: I've mounted the CephFS-Pool and df told's me, that actually 8.7TB are used. This is definitely more than 2TB of PVC's in sum. How I can identify the stale data?

ls of csi/:

ls of {pool_mount}:

Can I assume, that the directories with the identical .meta-file are currently in use and all other directories are stale?

[kgts23]

I found out, that the FS-Path is mapped in the PV-Object and that the other directories actually stale data.