Bug 1663145

Summary: [3.10] Recycler refuses to recycle PVs when a pod has a different PV using the same PVC
Product: OpenShift Container Platform Reporter: Tomas Smetana <tsmetana>
Component: StorageAssignee: Tomas Smetana <tsmetana>
Status: CLOSED ERRATA QA Contact: Wenqi He <wehe>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, aos-storage-staff, lxia
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: 1623426
: 1663147 (view as bug list) Environment:
Last Closed: 2019-01-30 15:13:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1663147    
Bug Blocks: 1623426    

Comment 1 Tomas Smetana 2019-01-03 11:34:23 UTC
https://github.com/openshift/origin/pull/21720

Comment 3 Wenqi He 2019-01-09 08:34:44 UTC
Tested on below version:
openshift v3.10.98
kubernetes v1.10.0+b81c8f8

Tried to verify this bug, because "StorageObjectInUseProtection" feature is enabled by default in 3.10, when delete the pvc, it would be in terminating status.
After delete the pod which use the pvc, the pvc can be deleted.
# oc get pvc
NAME          STATUS        VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfscpvc       Terminating   nfspv          1Gi        RWX                           30m
# oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                 STORAGECLASS   REASON    AGE
nfspv          1Gi        RWX            Recycle          Bound       default/nfscpvc                                34m
# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
nfs                        1/1       Running   0          2m
# oc exec -it nfs sh
/ # ls
bin              etc              hello-openshift  mnt              root             sys              usr
dev              hello            home             proc             run              tmp              var
/ # touch /mnt/wehe
/ # ls /mnt/wehe
/mnt/wehe
/ # exit
# oc delete pvc nfscpvc
persistentvolumeclaim "nfscpvc" deleted
# oc get pvc
NAME          STATUS        VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfscpvc       Terminating   nfspv          1Gi        RWX                           5m

Then *Must* delete the pod and create the pvc again, check the pv
# oc delete pods nfs
pod "nfs" deleted
# oc create -f pvc-template.json 
persistentvolumeclaim "nfscpvc" created
# oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                 STORAGECLASS   REASON    AGE
nfspv          1Gi        RWX            Recycle          Bound       default/nfscpvc                                40m
# oc describe pv nfspv
Name:            nfspv
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Bound
Claim:           default/nfscpvc
Reclaim Policy:  Recycle
Access Modes:    RWX
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    XXXXXXXXXXXXX
    Path:      /var/lib/exports/wehe
    ReadOnly:  false
Events:
  Type    Reason          Age               From                         Message
  ----    ------          ----              ----                         -------
  Normal  RecyclerPod     25s (x3 over 6m)  persistentvolume-controller  Recycler pod: Successfully assigned recycler-for-nfspv to ip-172-18-2-59.ec2.internal
  Normal  RecyclerPod     25s (x2 over 6m)  persistentvolume-controller  Recycler pod: pulling image "XXXXXX:443/openshift3/ose-recycler:v3.10.98"
  Normal  RecyclerPod     25s (x2 over 6m)  persistentvolume-controller  Recycler pod: Successfully pulled image "XXXXX:443/openshift3/ose-recycler:v3.10.98"
  Normal  RecyclerPod     22s (x3 over 6m)  persistentvolume-controller  Recycler pod: Created container
  Normal  RecyclerPod     22s (x3 over 6m)  persistentvolume-controller  Recycler pod: Started container
  Normal  RecyclerPod     22s               persistentvolume-controller  Recycler pod: Container image "XXXXXX:443/openshift3/ose-recycler:v3.10.98" already present on machine
  Normal  VolumeRecycled  21s (x2 over 6m)  persistentvolume-controller  Volume recycled

And there is no file under the nfs host folder:
# ls /var/lib/exports/wehe

Comment 5 errata-xmlrpc 2019-01-30 15:13:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0206