Description of problem: nfs-provisioner deployment and service are created in an project, if we delete the project directly, the pv cannot be deleted although it is "Delete" reclaim policy Version-Release number of selected component (if applicable): openshift v3.6.131 kubernetes v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1. Create a project 2. Create a scc $ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/deploy/kube-config/openshift-scc.yaml 3. Add user to this scc $ oadm policy add-scc-to-user nfs-provisioner <user> 4. Create nfs deployment and service $ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/auth/deployment-sa.yaml 5. Create storage class $ oc create -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/class.yaml 6. Create pvc and pod 7. Touch some file on mounted dir 8. Directly delete the project Actual results: The PV created by nfs-provisioner have not cleaned up # oc get pv NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 1Gi RWX Delete Released 3z64o/nfsdynpvc nfs-provisioner-3z64o 20m Expected results: PV can be deleted in reasonable time Master Log: Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859374 12762 pv_controller_base.go:180] enqueued "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" for sync Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859422 12762 pv_controller_base.go:302] volumeWorker[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9] Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859440 12762 pv_controller_base.go:558] storeObjectUpdate updating volume "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" with version 60641 Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859463 12762 pv_controller.go:407] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: phase: Released, bound to: "3z64o/nfsdynpvc (uid: d63a35a5-6153-11e7-b249-000d3a1a72a9)", boundByController: false Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859471 12762 pv_controller.go:432] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: volume is bound to claim 3z64o/nfsdynpvc Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859484 12762 pv_controller.go:441] synchronizing PersistentVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: claim 3z64o/nfsdynpvc not found Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859492 12762 pv_controller.go:956] reclaimVolume[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: policy is Delete Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859498 12762 pv_controller.go:1408] scheduleOperation[delete-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9[d6521c6e-6153-11e7-b249-000d3a1a72a9]] Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.859537 12762 pv_controller.go:1058] deleteVolumeOperation [pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9] started Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.863310 12762 handler.go:146] kube-apiserver: GET "/api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" satisfied by gorestful with webservice /api/v1 Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.864716 12762 wrap.go:42] GET /api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9: (4.118798ms) 200 [[openshift/v1.6.1+5115d708d7 (linux/amd64) kubernetes/314edd5/system:serviceaccount:kube-system:persistent-volume-binder] 40.121.213.45:57714] Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865296 12762 pv_controller.go:1162] isVolumeReleased[pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9]: volume is released Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865320 12762 pv_controller.go:1171] doDeleteVolume [pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9] Jul 5 07:52:33 storage-master atomic-openshift-master: I0705 07:52:33.865336 12762 pv_controller.go:1180] external deleter for volume "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" requested, ignoring Node Log (of failed PODs): Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.801491 26973 mount_linux.go:150] Unmounting /var/lib/origin/openshift.local.volumes/pods/d8e494e6-6153-11e7-b249-000d3a1a72a9/volumes/kubernetes.io~nfs/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.808927 26973 operation_generator.go:684] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e494e6-6153-11e7-b249-000d3a1a72a9-default-token-kfl4t" (OuterVolumeSpecName: "default-token-kfl4t") pod "d8e494e6-6153-11e7-b249-000d3a1a72a9" (UID: "d8e494e6-6153-11e7-b249-000d3a1a72a9"). InnerVolumeSpecName "default-token-kfl4t". PluginName "kubernetes.io/secret", VolumeGidValue "" Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.826500 26973 util.go:98] "/var/lib/origin/openshift.local.volumes/pods/d8e494e6-6153-11e7-b249-000d3a1a72a9/volumes/kubernetes.io~nfs/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" is unmounted, deleting the directory Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.826577 26973 operation_generator.go:684] UnmountVolume.TearDown succeeded for volume "kubernetes.io/nfs/d8e494e6-6153-11e7-b249-000d3a1a72a9-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" (OuterVolumeSpecName: "nfsdyn") pod "d8e494e6-6153-11e7-b249-000d3a1a72a9" (UID: "d8e494e6-6153-11e7-b249-000d3a1a72a9"). InnerVolumeSpecName "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9". PluginName "kubernetes.io/nfs", VolumeGidValue "" Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.847689 26973 desired_state_of_world_populator.go:160] Skipping findAndRemoveDeletedPods(). Not permitted until 2017-07-05 07:32:15.747446273 +0000 UTC (getPodStatusRetryDuration 2s). Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.899752 26973 reconciler.go:363] Detached volume "kubernetes.io/secret/d8e494e6-6153-11e7-b249-000d3a1a72a9-default-token-kfl4t" (spec.Name: "default-token-kfl4t") devicePath: "" Jul 5 07:32:13 storage-node-1 atomic-openshift-node: I0705 07:32:13.899784 26973 reconciler.go:363] Detached volume "kubernetes.io/nfs/d8e494e6-6153-11e7-b249-000d3a1a72a9-pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9" (spec.Name: "pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9") devicePath: "" PV Dump: # oc get pv pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: EXPORT_block: "\nEXPORT\n{\n\tExport_Id = 5;\n\tPath = /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9;\n\tPseudo = /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9;\n\tAccess_Type = RW;\n\tSquash = no_root_squash;\n\tSecType = sys;\n\tFilesystem_id = 5.5;\n\tFSAL {\n\t\tName = VFS;\n\t}\n}\n" Export_Id: "5" Project_Id: "0" Project_block: "" Provisioner_Id: d5abc261-5fb7-11e7-8769-0a580a800010 kubernetes.io/createdby: nfs-dynamic-provisioner pv.kubernetes.io/provisioned-by: example.com/nfs creationTimestamp: 2017-07-05T07:30:36Z name: pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 resourceVersion: "60641" selfLink: /api/v1/persistentvolumes/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 uid: d6521c6e-6153-11e7-b249-000d3a1a72a9 spec: accessModes: - ReadWriteMany capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: nfsdynpvc namespace: 3z64o resourceVersion: "60470" uid: d63a35a5-6153-11e7-b249-000d3a1a72a9 nfs: path: /export/pvc-d63a35a5-6153-11e7-b249-000d3a1a72a9 server: 172.30.206.205 persistentVolumeReclaimPolicy: Delete storageClassName: nfs-provisioner-3z64o status: phase: Released Additional info:
This is expected, external provisioners are responsible for cleaning up their own PVs. If they are deleted before all their PVs have been deleted, it is the admin's responsibility to manually delete the PVs since they are in Released state and effectively unusable. This is documented like so: "Deleting the provisioner deployment will cause any outstanding PersistentVolumes to become unusable for as long as the provisioner is gone." (Additionally, since the provisioner pod is like any other NFS server, unmount will fail indefinitely if the provisioner pod is deleted before all pods using its PVCs are deleted)
I suppose deleting the project can have unpredictable order of deletion, in which case deleting the PV should always be the last step just in case, since the PV will be useless anyway.