Bug 1570606 - PVCProtection: Pod and PVC are in stuck status when there is a deployment config
Summary: PVCProtection: Pod and PVC are in stuck status when there is a deployment config
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.0
Assignee: Hemant Kumar
QA Contact: Wenqi He
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-23 10:28 UTC by Wenqi He
Modified: 2018-04-23 16:49 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-04-23 16:49:00 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Wenqi He 2018-04-23 10:28:25 UTC
Description of problem:
After enable PVCProtection feature-gate, create a new app like mongodb-persistent

Version-Release number of selected component (if applicable):
openshift v3.9.24
kubernetes v1.9.1+a0ce1bc657

How reproducible:

Steps to Reproduce:
1. oc new-app mongodb-persistent
2. Wait until the first deployment done and pod is running
3. Delete the pvc 
oc delete pvc mongodb
4. Delete the pod
oc delete pods 

Actual results:
Since the PVCProtection is enabled, the pvc is in terminating status after the first deployed pod is deleted, the second pod is created by the second deployment, and both are in stuck status 

$ oc get pvc
NAME      STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb   Terminating   pvc-6c97bf73-46be-11e8-aefb-000d3a11d3cd   1Gi        RWO            azddef         36m

$ oc delete pods mongodb-1-59cfn
pod "mongodb-1-59cfn" deleted

$ oc get pods
NAME              READY     STATUS              RESTARTS   AGE
mongodb-1-s7pdg   0/1       ContainerCreating   0          11m

$ oc describe pod mongodb-1-s7pdg
  Type     Reason                 Age               From                                Message
  ----     ------                 ----              ----                                -------
  Normal   Scheduled              11m               default-scheduler                   Successfully assigned mongodb-1-s7pdg to storage-master-etcd-nfs-1
  Normal   SuccessfulMountVolume  11m               kubelet, storage-master-etcd-nfs-1  MountVolume.SetUp succeeded for volume "default-token-mfvmd"
  Warning  FailedMount            38s (x5 over 9m)  kubelet, storage-master-etcd-nfs-1  Unable to mount volumes for pod "mongodb-1-s7pdg_wehe(54460aaf-46c4-11e8-aefb-000d3a11d3cd)": timeout expired waiting for volumes to attach/mount for pod "wehe"/"mongodb-1-s7pdg". list of unattached/unmounted volumes=[mongodb-data]

Expected results:
PVC could be deleted because the pod which is used this PVC has been deleted already.

Node Log (of failed PODs):
E0423 07:10:43.997974   14719 desired_state_of_world_populator.go:276] Error processing volume "mongodb-data" for pod "mongodb-1-s7pdg_wehe(54460aaf-46c4-11e8-aefb-000d3a11d3cd)": error processing PVC "wehe"/"mongodb": can't start pod because PVC wehe/mongodb is being deleted

Additional info:

Comment 1 Hemant Kumar 2018-04-23 16:49:00 UTC
Not sure if this is a bug. IMO - this is working as intended because we are intentionally creating race condition in code that protects the PVC.

Does scaling down the deployment rather than deleting the pod results in PVC deletion? I am pretty sure that will work without problem. I am closing this bug, but if scaling down deployment does not work please reopen this BZ.

There is no way to guarantee PVC deletion before new pod creation happens and hence the BZ should be considered an user error.

Note You need to log in before you can comment on or make changes to this bug.