+++ This bug was initially created as a clone of Bug #1471630 +++ Description of problem: On containerized OCP 3.6 on vSphere, the vmdk can not be unmounted from node when the Pod is deleted. On RPM installed OCP, this is not reproducible, the unmount was successful and immediate. Version-Release number of selected component (if applicable): openshift v3.6.140 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 How reproducible: Always Steps to Reproduce: 1. Setup containerized OCP 3.6 on vSphere. 2. Ensure each Vm is configured with 'disk.enableUUID=1', configure cloud provider 3. Create StorageClass, PVC, Pod 4. Delete Pod 5. Verify vmdk is unmounted from node Actual results: After step 5, vmdk is still attached to node. [root@ocp36 ~]# mount|grep vsphere /dev/sdb on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk type ext4 (rw,relatime,seclabel,data=ordered) Expected results: Vmdk should be unmounted. Master Log: Node Log (of failed PODs): Jul 17 14:26:52 ocp36 atomic-openshift-node: I0717 14:26:52.741075 24244 nsenter_mount.go:202] IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point Jul 17 14:26:52 ocp36 atomic-openshift-node: W0717 14:26:52.741086 24244 util.go:85] Warning: "/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" is not a mountpoint, deleting Jul 17 14:26:52 ocp36 atomic-openshift-node: E0717 14:26:52.741183 24244 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk\"" failed. No retries permitted until 2017-07-17 14:28:52.741137521 +0800 CST (durationBeforeRetry 2m0s). Error: UnmountDevice failed for volume "kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" (spec.Name: "pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82") with: remove /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: device or resource busy Jul 17 14:26:52 ocp36 journal: I0717 14:26:52.741075 24244 nsenter_mount.go:202] IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point Jul 17 14:26:52 ocp36 journal: W0717 14:26:52.741086 24244 util.go:85] Warning: "/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" is not a mountpoint, deleting Jul 17 14:26:52 ocp36 journal: E0717 14:26:52.741183 24244 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk\"" failed. No retries permitted until 2017-07-17 14:28:52.741137521 +0800 CST (durationBeforeRetry 2m0s). Error: UnmountDevice failed for volume "kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" (spec.Name: "pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82") with: remove /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: device or resource busy PV Dump: apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume creationTimestamp: null name: pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82 spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: c4 namespace: jhou resourceVersion: "105830" uid: 0a6b27db-6ab5-11e7-8f64-0050569f1b82 persistentVolumeReclaimPolicy: Delete storageClassName: vsphere-thin vsphereVolume: fsType: ext4 volumePath: '[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk' PVC Dump: apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-class: vsphere-thin volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume creationTimestamp: null name: c4 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi volumeName: pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82 StorageClass Dump (if StorageClass used by PV/PVC): apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: null name: vsphere-thin parameters: diskformat: thin provisioner: kubernetes.io/vsphere-volume Additional info: --- Additional comment from Jan Safranek on 2017-07-18 04:13:26 EDT --- Digging in the node: * /proc/mounts says that /dev/sdb is mounted: /dev/sdb /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1]\040kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk ext4 rw,seclabel,relatime,data=ordered 0 0 Notice a space (\040) in the mount path. There is a directory named "[datastore1] kubevols". * IsLikelyNotMountPoint think that it's not a mount point: IsLikelyNotMountPoint findmnt output for path /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1]: IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point Notice that findmnt output is cut at the first space -> bug in Kubernetes NsenterMounter. --- Additional comment from Jan Safranek on 2017-07-18 07:56:59 EDT --- posted a PR upstream: https://github.com/kubernetes/kubernetes/pull/49111
origin/release-3.6 PR: https://github.com/openshift/origin/pull/15372
Verified in v3.6.172.0.1 The VMDK could be successfully unmounted after deleting Pod.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716