Bug 1473338

Summary:	[3.6][vSphere][containerized] VMDK not unmounted after deleting Pod
Product:	OpenShift Container Platform	Reporter:	Eric Paris <eparis>
Component:	Storage	Assignee:	Jan Safranek <jsafrane>
Status:	CLOSED ERRATA	QA Contact:	Jianwei Hou <jhou>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.6.1	CC:	aos-bugs, aos-storage-staff, jhou, jsafrane, smunilla
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1471630	Environment:
Last Closed:	2017-08-10 05:31:01 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1471630
Bug Blocks:

Description Eric Paris 2017-07-20 14:13:01 UTC

+++ This bug was initially created as a clone of Bug #1471630 +++

Description of problem:
On containerized OCP 3.6 on vSphere, the vmdk can not be unmounted from node when the Pod is deleted. 
On RPM installed OCP, this is not reproducible, the unmount was successful and immediate.

Version-Release number of selected component (if applicable):
openshift v3.6.140
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
Always

Steps to Reproduce:
1. Setup containerized OCP 3.6 on vSphere.
2. Ensure each Vm is configured with 'disk.enableUUID=1', configure cloud provider
3. Create StorageClass, PVC, Pod
4. Delete Pod
5. Verify vmdk is unmounted from node

Actual results:
After step 5, vmdk is still attached to node.
[root@ocp36 ~]# mount|grep vsphere
/dev/sdb on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk type ext4 (rw,relatime,seclabel,data=ordered)

Expected results:
Vmdk should be unmounted.

Master Log:

Node Log (of failed PODs):
Jul 17 14:26:52 ocp36 atomic-openshift-node: I0717 14:26:52.741075   24244 nsenter_mount.go:202] IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point
Jul 17 14:26:52 ocp36 atomic-openshift-node: W0717 14:26:52.741086   24244 util.go:85] Warning: "/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" is not a mountpoint, deleting
Jul 17 14:26:52 ocp36 atomic-openshift-node: E0717 14:26:52.741183   24244 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk\"" failed. No retries permitted until 2017-07-17 14:28:52.741137521 +0800 CST (durationBeforeRetry 2m0s). Error: UnmountDevice failed for volume "kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" (spec.Name: "pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82") with: remove /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: device or resource busy
Jul 17 14:26:52 ocp36 journal: I0717 14:26:52.741075   24244 nsenter_mount.go:202] IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point
Jul 17 14:26:52 ocp36 journal: W0717 14:26:52.741086   24244 util.go:85] Warning: "/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" is not a mountpoint, deleting
Jul 17 14:26:52 ocp36 journal: E0717 14:26:52.741183   24244 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk\"" failed. No retries permitted until 2017-07-17 14:28:52.741137521 +0800 CST (durationBeforeRetry 2m0s). Error: UnmountDevice failed for volume "kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk" (spec.Name: "pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82") with: remove /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: device or resource busy



PV Dump:
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
  creationTimestamp: null
  name: pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: c4
    namespace: jhou
    resourceVersion: "105830"
    uid: 0a6b27db-6ab5-11e7-8f64-0050569f1b82
  persistentVolumeReclaimPolicy: Delete
  storageClassName: vsphere-thin
  vsphereVolume:
    fsType: ext4
    volumePath: '[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk'

PVC Dump:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-class: vsphere-thin
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
  creationTimestamp: null
  name: c4
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82

StorageClass Dump (if StorageClass used by PV/PVC):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  name: vsphere-thin
parameters:
  diskformat: thin
provisioner: kubernetes.io/vsphere-volume

Additional info:

--- Additional comment from Jan Safranek on 2017-07-18 04:13:26 EDT ---

Digging in the node:

* /proc/mounts says that /dev/sdb is mounted:

/dev/sdb /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1]\040kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk ext4 rw,seclabel,relatime,data=ordered 0 0

Notice a space (\040) in the mount path. There is a directory named "[datastore1] kubevols".


* IsLikelyNotMountPoint think that it's not a mount point:

IsLikelyNotMountPoint findmnt output for path /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1]:

IsLikelyNotMountPoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/[datastore1] kubevols/kubernetes-dynamic-pvc-0a6b27db-6ab5-11e7-8f64-0050569f1b82.vmdk is not a mount point


Notice that findmnt output is cut at the first space -> bug in Kubernetes NsenterMounter.

--- Additional comment from Jan Safranek on 2017-07-18 07:56:59 EDT ---

posted a PR upstream: https://github.com/kubernetes/kubernetes/pull/49111

Comment 1 Jan Safranek 2017-07-20 14:33:44 UTC

origin/release-3.6 PR: https://github.com/openshift/origin/pull/15372

Comment 3 Jianwei Hou 2017-07-31 07:14:27 UTC

Verified in v3.6.172.0.1
The VMDK could be successfully unmounted after deleting Pod.

Comment 5 errata-xmlrpc 2017-08-10 05:31:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716