Issue with delete reclaim policy in docker UCP setup

[virendra-sharma]

In docker UCP setup it is being observed that delete reclaim policy behaving unexpectedly in some use cases, It is not deleting provisioned volume.
I created static as well as dynamic provisioned volume and observed in both cases provisioned volume not deleting which get created during sc,pv,pvc, though reclaim policy set as "Delete".

Expected output:

Volume should delete after deleting corresponding pod,pvc,pv,sc with reclaim policy as "delete".

Steps performed from UCP UI:

Scenario:- 1

1. Created sc,pv,pvc,pod.
2. Volume created by sc,pv,pvc and listed in "docker volume ls".
3. Deleted pod from UI.
4. Deleted pv,pvc,sc together by selecting all in UI.
5. Volume still present on setup.

Scenario:- 2

1. Created sc,pv,pvc,pod.
2. Volume created by sc,pv,pvc and listed in "docker volume ls".
3. Deleted pod from UI.
4. Deleted pvc first, after few second pv deleted automatically and later deleted sc from UCP UI.
5. Created volume deleted as expected as per delete reclaim policy.

Observation:

1. Reclaim policy in dynamic provisioned volume is "Delete" by default and it should delete corresponding volume once sc,pv,pvc,deleted, but in current state volume still exists.
2. Volume deletes as per delete reclaim policy only if it is deleted in sequence of pod,pvc,pv,sc (one by one) from UCP UI, but it allowing to delete pv,pvc,sc together by selecting all from UCI UI and if we do so then volume won't delete.

YAML's

sc-test.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: sc-test
provisioner: hpe.com/hpe
parameters:
  name: TEST-VOLUME

pvc-test.yml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: sc-test

pod-test.yml

apiVersion: v1
kind: Pod
metadata:
  name: pod-test
spec:
  containers:
  - name: minio
    image: minio/minio:latest
    args:
    - server
    - /export
    env:
    - name: MINIO_ACCESS_KEY
      value: minio
    - name: MINIO_SECRET_KEY
      value: doryspeakswhale
    ports:
    - containerPort: 9000
    volumeMounts:
    - name: export
      mountPath: /export
  volumes:
    - name: export
      persistentVolumeClaim:
        claimName: pvc-test

Logs:

Volume not deleted in delete reclaim policy
doryd log for sc,pv,pvc delete together.txt
Plugin container log on master (together delete).txt
Plugin container log on worker1 (pod scheduled node) (All delete together).txt

Volume deleted in delete reclaim policy
doryd log for deleting in sequence of pod,pvc,pv,sc.txt
Plugin container log on master (as pod scheduled node) after deleting in sequnce of pod,pvc,pv,sc.txt

[wdurairaj]

this bug is duplicate of #468

[wdurairaj]

Specifying reclaimPolicy: "Delete" in StorageClass atleast works now, in the new dory_installer (available in 10.50.9.10

Can you provide me the P

Test Log:

[root@master-node ~]# kubectl create -f - << EOF
> ---
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
>  name: mysc13
> provisioner: hpe.com/hpe
> parameters:
>   size: "10"
>   cpg: "SHASHI_SNAP_DEST_CPG"
>   provisioning: "thin"
>   backend: "3PAR1"
> reclaimPolicy: "Delete"
> EOF
storageclass.storage.k8s.io/mysc13 created
[root@master-node ~]#
[root@master-node ~]# kubectl create -f - << EOF
> ---
> kind: PersistentVolumeClaim
> apiVersion: v1
> metadata:
>   name: pvc1
> spec:
>   accessModes:
>     - ReadWriteOnce
>   resources:
>     requests:
>       storage: 20Gi
>   storageClassName: mysc13
> EOF
Error from server (AlreadyExists): error when creating "STDIN": persistentvolumeclaims "pvc1" already exists
[root@master-node ~]# kubectl delete pvc pvc1
persistentvolumeclaim "pvc1" deleted
[root@master-node ~]# kubectl create -f - << EOF
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: mysc13
EOF
persistentvolumeclaim/pvc1 created
[root@master-node ~]#
[root@master-node ~]# kubectl get pvc pvc1
NAME   STATUS   VOLUME                                        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc1   Bound    mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360   20Gi       RWO            mysc13         6s
[root@master-node ~]# kubectl describe pvc pvc1
Name:          pvc1
Namespace:     default
StorageClass:  mysc13
Status:        Bound
Volume:        mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:        <none>
Mounted By:    <none>
[root@master-node ~]#
[root@master-node ~]#
[root@master-node ~]# kubectl describe pv mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360
Name:            mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360
Labels:          <none>
Annotations:     hpe.com/docker-volume-name: mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360
                 pv.kubernetes.io/provisioned-by: hpe.com/hpe
                 volume.beta.kubernetes.io/storage-class: mysc13
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    mysc13
Status:          Bound
Claim:           default/pvc1
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        20Gi
Node Affinity:   <none>
Message:
Source:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     hpe.com/hpe
    FSType:
    SecretRef:  nil
    ReadOnly:   false
    Options:    map[backend:3PAR1 cpg:SHASHI_SNAP_DEST_CPG name:mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360 provisioning:thin size:10]
Events:         <none>
[root@master-node ~]#
[root@master-node ~]# kubectl delete pvc pvc1
persistentvolumeclaim "pvc1" deleted
[root@master-node ~]# kubectl describe pv mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360
Error from server (NotFound): persistentvolumes "mysc13-a9ba727b-4c93-11e9-9965-6cc21739c360" not found

Now without specifying this reclaimPolicy

[root@master-node ~]# kubectl create -f - << EOF
> ---
> kind: StorageClass
> apiVersion: storage.k8s.io/v1
> metadata:
>  name: mysc14
> provisioner: hpe.com/hpe
> parameters:
>   size: "10"
>   cpg: "SHASHI_SNAP_DEST_CPG"
>   provisioning: "thin"
>   backend: "3PAR1"
> EOF
storageclass.storage.k8s.io/mysc14 created
[root@master-node ~]# kubectl create -f - << EOF
> ---
> kind: PersistentVolumeClaim
> apiVersion: v1
> metadata:
>   name: pvc1
> spec:
>   accessModes:
>     - ReadWriteOnce
>   resources:
>     requests:
>       storage: 20Gi
>   storageClassName: mysc14
> EOF
persistentvolumeclaim/pvc1 created
[root@master-node ~]# kubectl get pvc pvc1
NAME   STATUS   VOLUME                                        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc1   Bound    mysc14-acf736f0-4c94-11e9-9965-6cc21739c360   20Gi       RWO            mysc14         9s
[root@master-node ~]#
[root@master-node ~]# kubectl describe pv mysc14-acf736f0-4c94-11e9-9965-6cc21739c360
Name:            mysc14-acf736f0-4c94-11e9-9965-6cc21739c360
Labels:          <none>
Annotations:     hpe.com/docker-volume-name: mysc14-acf736f0-4c94-11e9-9965-6cc21739c360
                 pv.kubernetes.io/provisioned-by: hpe.com/hpe
                 volume.beta.kubernetes.io/storage-class: mysc14
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    mysc14
Status:          Bound
Claim:           default/pvc1
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        20Gi
Node Affinity:   <none>
Message:
Source:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     hpe.com/hpe
    FSType:
    SecretRef:  nil
    ReadOnly:   false
    Options:    map[cpg:SHASHI_SNAP_DEST_CPG name:mysc14-acf736f0-4c94-11e9-9965-6cc21739c360 provisioning:thin size:10 backend:3PAR1]
Events:         <none>
[root@master-node ~]#
[root@master-node ~]# kubectl delete pvc1
error: resource(s) were provided, but no name, label selector, or --all flag specified
[root@master-node ~]# kubectl delete pvc pvc1
persistentvolumeclaim "pvc1" deleted
[root@master-node ~]#
[root@master-node ~]# kubectl describe pv mysc14-acf736f0-4c94-11e9-9965-6cc21739c360
Error from server (NotFound): persistentvolumes "mysc14-acf736f0-4c94-11e9-9965-6cc21739c360" not found

In both cases, when the reclaimPolicy is specified/not specified as Delete , deleting the PVC deletes the corresponding PV and docker volume also

[root@master-node ~]# docker volume ls | grep mysc14-acf736f0-4c94-11e9-9965-6cc21739c360
[root@master-node ~]#

[wdurairaj]

@virendra-sharma did you observe that , if you manually delete the volume (associated with PV), did you face any issue?
Like

1. Dangling LUN entries on 3PAR, which was preventing volume delete ?
2. scsi entries not cleaned up on the initiator side (docker host) ?

Etc. Please update your findings here.

[wdurairaj]

Also, see if my previous update -- successful PV deletion with reclaim policy set to "Delete"

[virendra-sharma]

@wdurairaj as discussed, I have updated bug with delete reclaim policy issue arises for docker UCP setup.
from UCP UI if we delete pv,pvc,sc by selecting all together, created volume doesn't deletes.