Bug 1663145 - [3.10] Recycler refuses to recycle PVs when a pod has a different PV using the same PVC
Summary: [3.10] Recycler refuses to recycle PVs when a pod has a different PV using th...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 3.10.z
Assignee: Tomas Smetana
QA Contact: Wenqi He
URL:
Whiteboard:
Depends On: 1663147
Blocks: 1623426
TreeView+ depends on / blocked
 
Reported: 2019-01-03 10:02 UTC by Tomas Smetana
Modified: 2019-01-30 15:13 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of: 1623426
: 1663147 (view as bug list)
Environment:
Last Closed: 2019-01-30 15:13:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0206 0 None None None 2019-01-30 15:13:25 UTC

Comment 1 Tomas Smetana 2019-01-03 11:34:23 UTC
https://github.com/openshift/origin/pull/21720

Comment 3 Wenqi He 2019-01-09 08:34:44 UTC
Tested on below version:
openshift v3.10.98
kubernetes v1.10.0+b81c8f8

Tried to verify this bug, because "StorageObjectInUseProtection" feature is enabled by default in 3.10, when delete the pvc, it would be in terminating status.
After delete the pod which use the pvc, the pvc can be deleted.
# oc get pvc
NAME          STATUS        VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfscpvc       Terminating   nfspv          1Gi        RWX                           30m
# oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                 STORAGECLASS   REASON    AGE
nfspv          1Gi        RWX            Recycle          Bound       default/nfscpvc                                34m
# oc get pods
NAME                       READY     STATUS    RESTARTS   AGE
nfs                        1/1       Running   0          2m
# oc exec -it nfs sh
/ # ls
bin              etc              hello-openshift  mnt              root             sys              usr
dev              hello            home             proc             run              tmp              var
/ # touch /mnt/wehe
/ # ls /mnt/wehe
/mnt/wehe
/ # exit
# oc delete pvc nfscpvc
persistentvolumeclaim "nfscpvc" deleted
# oc get pvc
NAME          STATUS        VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfscpvc       Terminating   nfspv          1Gi        RWX                           5m

Then *Must* delete the pod and create the pvc again, check the pv
# oc delete pods nfs
pod "nfs" deleted
# oc create -f pvc-template.json 
persistentvolumeclaim "nfscpvc" created
# oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                 STORAGECLASS   REASON    AGE
nfspv          1Gi        RWX            Recycle          Bound       default/nfscpvc                                40m
# oc describe pv nfspv
Name:            nfspv
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Bound
Claim:           default/nfscpvc
Reclaim Policy:  Recycle
Access Modes:    RWX
Capacity:        1Gi
Node Affinity:   <none>
Message:         
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    XXXXXXXXXXXXX
    Path:      /var/lib/exports/wehe
    ReadOnly:  false
Events:
  Type    Reason          Age               From                         Message
  ----    ------          ----              ----                         -------
  Normal  RecyclerPod     25s (x3 over 6m)  persistentvolume-controller  Recycler pod: Successfully assigned recycler-for-nfspv to ip-172-18-2-59.ec2.internal
  Normal  RecyclerPod     25s (x2 over 6m)  persistentvolume-controller  Recycler pod: pulling image "XXXXXX:443/openshift3/ose-recycler:v3.10.98"
  Normal  RecyclerPod     25s (x2 over 6m)  persistentvolume-controller  Recycler pod: Successfully pulled image "XXXXX:443/openshift3/ose-recycler:v3.10.98"
  Normal  RecyclerPod     22s (x3 over 6m)  persistentvolume-controller  Recycler pod: Created container
  Normal  RecyclerPod     22s (x3 over 6m)  persistentvolume-controller  Recycler pod: Started container
  Normal  RecyclerPod     22s               persistentvolume-controller  Recycler pod: Container image "XXXXXX:443/openshift3/ose-recycler:v3.10.98" already present on machine
  Normal  VolumeRecycled  21s (x2 over 6m)  persistentvolume-controller  Volume recycled

And there is no file under the nfs host folder:
# ls /var/lib/exports/wehe

Comment 5 errata-xmlrpc 2019-01-30 15:13:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0206


Note You need to log in before you can comment on or make changes to this bug.