make delete volume idempotent when node is removed from cluster #413

abhilashshetty04 · 2025-09-03T08:43:50Z

When a Kubernetes node is removed from the cluster while it still has existing PVCs/LVMVolumes, those volumes become orphaned. If a PVC is deleted in this state, the LocalPV-LVM CSI controller marks the corresponding LVMVolume CR for deletion by adding a deletion timestamp.

Normally, the node-agent running on the worker node is responsible for cleaning up the logical volume (LV) and removing the finalizer from the CR. However, in this scenario, the node no longer exists, so there is no agent running to perform the cleanup.

This fix leverages the condition of the Kubernetes Node being absent: in such cases, the CSI controller itself removes the finalizer and cleans up the LVMVolume CR.

Fixes: #407

Abhinandan-Purkait

LGTM

Abhinandan-Purkait · 2025-11-05T14:02:27Z

pkg/lvm/volume.go

 	err = volbuilder.NewKubeclient().WithNamespace(LvmNamespace).Delete(volumeID)
 	if err == nil {
-		klog.Infof("deprovisioned volume %s", volumeID)
+		klog.Infof("deprovisioning volume %s, marking deletion timestamp", volumeID)


What do we mean by marking deletion timestamp, here?

I guess it means the deletion timestamp is being set?

Here we call delete which just marks deletion timestamp as we have finaliser.

Shall i remove marking deletion timestamp ?

I would keep that bit out since it's implied, but fine either way

I will remove it.

tiagolobocastro

What about existing volumes

tiagolobocastro · 2025-11-06T14:49:39Z

pkg/driver/controller.go

 	}
+	present := isNodePresent(vol.Spec.OwnerNodeID)
+	if !present {
+		klog.Infof("Removing finalizer as node %s is not present in cluster", vol.Spec.OwnerNodeID)


Should this be a warning?

Yes, will make it warning

tiagolobocastro · 2025-11-06T14:51:21Z

pkg/lvm/volume.go


 	_, err := volbuilder.NewKubeclient().WithNamespace(LvmNamespace).Update(vol)
+	if err != nil {
+		klog.Infof("Finalizer removed successfully for %s", vol.Name)


Do we need this as info or is debug fine?

Will not reach this code if node is present. Would be better to have it in log by default.

Don't we already have the message from above where we call it if the node is not present?
Oh also this check seems reversed, if err != nil then we didn't remove the finalizer?

Yes, err != nil means we fail to remove finalizer, we return err. If not, we move on to check if lvmvolume CR is deleted.

tiagolobocastro · 2025-11-06T14:52:02Z

pkg/lvm/volume.go

 	err = volbuilder.NewKubeclient().WithNamespace(LvmNamespace).Delete(volumeID)
 	if err == nil {
-		klog.Infof("deprovisioned volume %s", volumeID)
+		klog.Infof("deprovisioning volume %s, marking deletion timestamp", volumeID)


I guess it means the deletion timestamp is being set?

tiagolobocastro · 2025-11-06T14:56:29Z

pkg/driver/controller.go

 				"failed to handle delete volume request for {%s}", volumeID)
 		}
 	}
+	present := isNodePresent(vol.Spec.OwnerNodeID)


what if the finalizer is already removed? Then we don't need to run this?

finalizer is removed from the node-agent when lvremove succeeds. Here we will validate if node is not present. If not, finalizer will not be removed. so will remove finalizer from here.

If the lvm volume has additional finalisers (eg: added by user) but we've already removed ours, on a previous csi call, then we don't need to do this again
We only need to do this stage if our finaliser is present

Right now this removes all Finalizers. I am using the function which is called by node-agent once lvremove is done.

https://github.com/openebs/lvm-localpv/blob/develop/pkg/lvm/volume.go#L217C1-L223C2

lvmvolume CR is managed by localpv-lvm. Users may not add any finalizers onto it.

tiagolobocastro · 2025-11-07T14:45:00Z

pkg/lvm/volume.go


 	_, err := volbuilder.NewKubeclient().WithNamespace(LvmNamespace).Update(vol)
+	if err != nil {
+		klog.Infof("Finalizer removed successfully for %s", vol.Name)


Don't we already have the message from above where we call it if the node is not present?
Oh also this check seems reversed, if err != nil then we didn't remove the finalizer?

tiagolobocastro · 2025-11-07T14:47:24Z

pkg/lvm/volume.go

 	err = volbuilder.NewKubeclient().WithNamespace(LvmNamespace).Delete(volumeID)
 	if err == nil {
-		klog.Infof("deprovisioned volume %s", volumeID)
+		klog.Infof("deprovisioning volume %s, marking deletion timestamp", volumeID)


I would keep that bit out since it's implied, but fine either way

tiagolobocastro · 2025-11-07T14:52:45Z

pkg/driver/controller.go

 				"failed to handle delete volume request for {%s}", volumeID)
 		}
 	}
+	present := isNodePresent(vol.Spec.OwnerNodeID)


If the lvm volume has additional finalisers (eg: added by user) but we've already removed ours, on a previous csi call, then we don't need to do this again
We only need to do this stage if our finaliser is present

tiagolobocastro · 2025-11-07T14:53:06Z

pkg/driver/controller.go

+	if !present {
+		klog.Infof("Removing finalizer as node %s is not present in cluster", vol.Spec.OwnerNodeID)
+		if err = lvm.RemoveVolFinalizer(vol); err != nil {
+			return nil


shouldn't we return err here?

Fail the DeleteVolume call? I guess that will be better.

…e volume is not present Signed-off-by: Abhilash Shetty <[email protected]>

abhilashshetty04 requested a review from a team as a code owner September 3, 2025 08:43

abhilashshetty04 changed the title ~~make delete call idempotent when node is removed from cluster~~ make delete volume idempotent when node is removed from cluster Sep 3, 2025

abhilashshetty04 force-pushed the lvmvolume_leak_fix branch 3 times, most recently from 1671fcc to 9b11bec Compare September 5, 2025 06:55

abhilashshetty04 mentioned this pull request Sep 5, 2025

Slow leak of LVMVolume Kubernetes objects causes extremely slow allocation of new volumes #407

Open

Abhinandan-Purkait approved these changes Nov 5, 2025

View reviewed changes

tiagolobocastro reviewed Nov 6, 2025

View reviewed changes

tiagolobocastro reviewed Nov 7, 2025

View reviewed changes

abhilashshetty04 force-pushed the lvmvolume_leak_fix branch 3 times, most recently from da04e2d to d766fd7 Compare November 10, 2025 14:22

fix(controller): remove finaliser in controller if k8s node owning th…

43a5f2c

…e volume is not present Signed-off-by: Abhilash Shetty <[email protected]>

abhilashshetty04 force-pushed the lvmvolume_leak_fix branch from d766fd7 to 43a5f2c Compare November 10, 2025 14:26

make delete volume idempotent when node is removed from cluster #413

Are you sure you want to change the base?

make delete volume idempotent when node is removed from cluster #413

Uh oh!

Conversation

abhilashshetty04 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abhinandan-Purkait left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tiagolobocastro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhilashshetty04 Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abhilashshetty04 commented Sep 3, 2025 •

edited

Loading

abhilashshetty04 Nov 9, 2025 •

edited

Loading