Bug 1467800 - PV is not cleaned up after directly delete the project for nfs-provisioner
PV is not cleaned up after directly delete the project for nfs-provisioner
Status: CLOSED NOTABUG
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.6.1
Unspecified Unspecified
medium Severity medium
: ---
: 3.6.z
Assigned To: Bradley Childs
Wenqi He
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-05 04:08 EDT by Wenqi He
Modified: 2017-07-05 11:24 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-05 11:22:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Wenqi He 2017-07-05 04:08:10 EDT
Description of problem:
nfs-provisioner deployment and service are created in an project, if we delete the project directly, the pv cannot be deleted although it is "Delete" reclaim policy

Version-Release number of selected component (if applicable):
openshift v3.6.131
kubernetes v1.6.1+5115d708d7

How reproducible:
Always

Steps to Reproduce:
1. Create a project
2. Create a scc
$ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/deploy/kube-config/openshift-scc.yaml
3. Add user to this scc
$ oadm policy add-scc-to-user nfs-provisioner <user> 
4. Create nfs deployment and service
$ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/auth/deployment-sa.yaml
5. Create storage class 
$ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/class.yaml
6. Create pvc and pod
7. Touch some file on mounted dir
8. Directly delete the project


Actual results:
The PV created by nfs-provisioner have not cleaned up
# oc get pv
NAME                                      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS     CLAIM             STORAGECLASS            REASON    AGE
pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9   1Gi        RWX           Delete          Released   3z64o/nfsdynpvc   nfs-provisioner-3z64o             20m

Expected results:
PV can be deleted in reasonable time 

Master Log:
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859374   12762 pv_controller_base.go:180] enqueued "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" for sync
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859422   12762 pv_controller_base.go:302] volumeWorker[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859440   12762 pv_controller_base.go:558] storeObjectUpdate updating volume "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" with version 60641
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859463   12762 pv_controller.go:407] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: phase: Released, bound to: "3z64o/nfsdynpvc (uid: d63a35a5-6153-11e7-b249-000d3a1a72a9)", boundByController: false
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859471   12762 pv_controller.go:432] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: volume is bound to claim 3z64o/nfsdynpvc
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859484   12762 pv_controller.go:441] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: claim 3z64o/nfsdynpvc not found
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859492   12762 pv_controller.go:956] reclaimVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: policy is Delete
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859498   12762 pv_controller.go:1408] scheduleOperation[delete-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9[d6521c6e-6153-11e7-b249-000d3a1a72a9]]
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859537   12762 pv_controller.go:1058] deleteVolumeOperation [pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9] started
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.863310   12762 handler.go:146] kube-apiserver: GET "/api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" satisfied by gorestful with webservice /api/v1
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.864716   12762 wrap.go:42] GET /api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9: (4.118798ms) 200 [[openshift/v1.6.1+5115d708d7 (linux/amd64) kubernetes/314edd5/system:serviceaccount:kube-system:persistent-volume-binder] 40.121.213.45:57714]
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865296   12762 pv_controller.go:1162] isVolumeReleased[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: volume is released
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865320   12762 pv_controller.go:1171] doDeleteVolume [pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]
Jul  5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865336   12762 pv_controller.go:1180] external deleter for volume "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" requested, ignoring

Node Log (of failed PODs):
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.801491   26973 mount_linux.go:150] Unmounting /var/lib/origin/openshift.local.volumes/pods/d8e494e6-6153-11e7-b249-000d3a1a72a9/volumes/kubernetes.io~nfs/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.808927   26973 operation_generator.go:684] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e494e6-6153-11e7-b249-000d3a1a72a9-default-token-kfl4t" (OuterVolumeSpecName: "default-token-kfl4t") pod "d8e494e6-6153-11e7-b249-000d3a1a72a9" (UID: "d8e494e6-6153-11e7-b249-000d3a1a72a9"). InnerVolumeSpecName "default-token-kfl4t". PluginName "kubernetes.io/secret", VolumeGidValue ""
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.826500   26973 util.go:98] "/var/lib/origin/openshift.local.volumes/pods/d8e494e6-6153-11e7-b249-000d3a1a72a9/volumes/kubernetes.io~nfs/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" is unmounted, deleting the directory
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.826577   26973 operation_generator.go:684] UnmountVolume.TearDown succeeded for volume "kubernetes.io/nfs/d8e494e6-6153-11e7-b249-000d3a1a72a9-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" (OuterVolumeSpecName: "nfsdyn") pod "d8e494e6-6153-11e7-b249-000d3a1a72a9" (UID: "d8e494e6-6153-11e7-b249-000d3a1a72a9"). InnerVolumeSpecName "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9". PluginName "kubernetes.io/nfs", VolumeGidValue ""
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.847689   26973 desired_state_of_world_populator.go:160] Skipping findAndRemoveDeletedPods(). Not permitted until 2017-07-05 07:32:15.747446273 +0000 UTC (getPodStatusRetryDuration 2s).
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.899752   26973 reconciler.go:363] Detached volume "kubernetes.io/secret/d8e494e6-6153-11e7-b249-000d3a1a72a9-default-token-kfl4t" (spec.Name: "default-token-kfl4t") devicePath: ""
Jul  5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.899784   26973 reconciler.go:363] Detached volume "kubernetes.io/nfs/d8e494e6-6153-11e7-b249-000d3a1a72a9-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" (spec.Name: "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9") devicePath: ""


PV Dump:
# oc get pv pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    EXPORT_block: "\nEXPORT\n{\n\tExport_Id = 5;\n\tPath = /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9;\n\tPseudo
      = /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9;\n\tAccess_Type = RW;\n\tSquash
      = no_root_squash;\n\tSecType = sys;\n\tFilesystem_id = 5.5;\n\tFSAL {\n\t\tName
      = VFS;\n\t}\n}\n"
    Export_Id: "5"
    Project_Id: "0"
    Project_block: ""
    Provisioner_Id: d5abc261-5fb7-11e7-8769-0a580a800010
    kubernetes.io/createdby: nfs-dynamic-provisioner
    pv.kubernetes.io/provisioned-by: example.com/nfs
  creationTimestamp: 2017-07-05T07:30:36Z
  name: pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9
  resourceVersion: "60641"
  selfLink: /api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9
  uid: d6521c6e-6153-11e7-b249-000d3a1a72a9
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nfsdynpvc
    namespace: 3z64o
    resourceVersion: "60470"
    uid: d63a35a5-6153-11e7-b249-000d3a1a72a9
  nfs:
    path: /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9
    server: 172.30.206.205
  persistentVolumeReclaimPolicy: Delete
  storageClassName: nfs-provisioner-3z64o
status:
  phase: Released


Additional info:
Comment 1 Matthew Wong 2017-07-05 11:22:50 EDT
This is expected, external provisioners are responsible for cleaning up their own PVs. If they are deleted before all their PVs have been deleted, it is the admin's responsibility to manually delete the PVs since they are in Released state and effectively unusable. This is documented like so: "Deleting the provisioner deployment will cause any outstanding PersistentVolumes to become unusable for as long as the provisioner is gone."

(Additionally, since the provisioner pod is like any other NFS server, unmount will fail indefinitely if the provisioner pod is deleted before all pods using its PVCs are deleted)
Comment 2 Matthew Wong 2017-07-05 11:24:10 EDT
I suppose deleting the project can have unpredictable order of deletion, in which case deleting the PV should always be the last step just in case, since the PV will be useless anyway.

Note You need to log in before you can comment on or make changes to this bug.