Created attachment 1639770 [details] Vsphere snapshot Description of problem: After upgrade from 4.2 to 4.3 nightly build, PV cannot mount to node. It reports volume does not exist but it can be found in vmware portal. Also new create pod with pvc/pv doesn't work. Version-Release number of selected component (if applicable): upgrade from 4.2 to 4.3.0-0.nightly-2019-11-25-153929 How reproducible: Sometimes during upgrade Steps to Reproduce: 1. Pod with pvc/pv created and work fine before upgrade. 2. upgrade from 4.2 to 4.3 nightly build 3. previous pod with pvc/pv doesn't work anymore. $ oc describe pod elasticsearch-cdm-1fnn2uw8-3-6669dc877d-zfdfk Name: elasticsearch-cdm-1fnn2uw8-3-6669dc877d-zfdfk Namespace: openshift-logging Priority: 0 PriorityClassName: <none> Node: compute-3/139.178.76.25 Start Time: Tue, 26 Nov 2019 16:04:23 +0800 Labels: cluster-name=elasticsearch component=elasticsearch es-node-client=true es-node-data=true es-node-master=true node-name=elasticsearch-cdm-1fnn2uw8-3 pod-template-hash=6669dc877d tuned.openshift.io/elasticsearch=true Annotations: openshift.io/scc: restricted Status: Pending IP: Controlled By: ReplicaSet/elasticsearch-cdm-1fnn2uw8-3-6669dc877d Containers: elasticsearch: Container ID: Image: image-registry.openshift-image-registry.svc:5000/openshift/ose-logging-elasticsearch5@sha256:0029f94d663d374a4f86c318bc1fb483a0958e22ae09f7723b2c20aab5d6c41c Image ID: Ports: 9300/TCP, 9200/TCP Host Ports: 0/TCP, 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 2Gi Requests: cpu: 200m memory: 2Gi Readiness: exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3 Environment: DC_NAME: elasticsearch-cdm-1fnn2uw8-3 NAMESPACE: openshift-logging (v1:metadata.namespace) KUBERNETES_TRUST_CERT: true SERVICE_DNS: elasticsearch-cluster CLUSTER_NAME: elasticsearch INSTANCE_RAM: 2Gi HEAP_DUMP_LOCATION: /elasticsearch/persistent/heapdump.hprof RECOVER_AFTER_TIME: 5m READINESS_PROBE_TIMEOUT: 30 POD_LABEL: cluster=elasticsearch IS_MASTER: true HAS_DATA: true Mounts: /elasticsearch/persistent from elasticsearch-storage (rw) /etc/openshift/elasticsearch/secret from certificates (rw) /usr/share/java/elasticsearch/config from elasticsearch-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-mkwf9 (ro) proxy: Container ID: Image: image-registry.openshift-image-registry.svc:5000/openshift/ose-oauth-proxy@sha256:741ad9d77dd96da36a1b1df51747eff477eb33889391aacd8ca82445022afcc4 Image ID: Port: 60000/TCP Host Port: 0/TCP Args: --https-address=:60000 --provider=openshift --upstream=https://127.0.0.1:9200 --tls-cert=/etc/proxy/secrets/tls.crt --tls-key=/etc/proxy/secrets/tls.key --upstream-ca=/etc/proxy/elasticsearch/admin-ca --openshift-service-account=elasticsearch -openshift-sar={"resource": "namespaces", "verb": "get"} -openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}} --pass-user-bearer-token --cookie-secret=buqCSQ4QJymiNNmSz1/Yug== State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 64Mi Requests: cpu: 100m memory: 64Mi Environment: <none> Mounts: /etc/proxy/elasticsearch from certificates (rw) /etc/proxy/secrets from elasticsearch-metrics (rw) /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-mkwf9 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: elasticsearch-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: elasticsearch Optional: false elasticsearch-storage: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: elasticsearch-elasticsearch-cdm-1fnn2uw8-3 ReadOnly: false certificates: Type: Secret (a volume populated by a Secret) SecretName: elasticsearch Optional: false elasticsearch-metrics: Type: Secret (a volume populated by a Secret) SecretName: elasticsearch-metrics Optional: false elasticsearch-token-mkwf9: Type: Secret (a volume populated by a Secret) SecretName: elasticsearch-token-mkwf9 Optional: false QoS Class: Burstable Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling <unknown> default-scheduler 0/7 nodes are available: 1 Insufficient cpu, 2 node(s) were unschedulable, 5 Insufficient memory. Warning FailedScheduling <unknown> default-scheduler 0/7 nodes are available: 1 Insufficient cpu, 2 node(s) were unschedulable, 5 Insufficient memory. Warning FailedScheduling <unknown> default-scheduler 0/7 nodes are available: 1 Insufficient cpu, 1 node(s) had taints that the pod didn't tolerate, 1 node(s) were unschedulable, 5 Insufficient memory. Warning FailedScheduling <unknown> default-scheduler 0/7 nodes are available: 1 Insufficient cpu, 2 node(s) had taints that the pod didn't tolerate, 5 Insufficient memory. Normal Scheduled <unknown> default-scheduler Successfully assigned openshift-logging/elasticsearch-cdm-1fnn2uw8-3-6669dc877d-zfdfk to compute-3 Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r79fa1f4acbdb4cc493872caf3a1673c4.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Normal SuccessfulAttachVolume 121m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r1a2a31af30874d3ea2205a9bba48d391.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r5b09f2fc63f64c3c9cf3be67ae31ead0.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r16b7800cbcb040938e1e71a25536ccf4.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-rc5d0aceaf4484676a1adfab75e5f7ab5.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 121m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r1708b677305f44939f06ad327e6f89a9.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 120m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r99ecdf02405643e2b3c1b5bd7746db01.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 120m kubelet, compute-3 MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-rfc5058769d7d43709106f5e1aa665a73.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Warning FailedMount 101m kubelet, compute-3 Unable to attach or mount volumes: unmounted volumes=[elasticsearch-storage], unattached volumes=[elasticsearch-token-mkwf9 elasticsearch-metrics elasticsearch-storage elasticsearch-config certificates]: timed out waiting for the condition Warning FailedMount 76m (x3 over 119m) kubelet, compute-3 Unable to attach or mount volumes: unmounted volumes=[elasticsearch-storage], unattached volumes=[elasticsearch-config certificates elasticsearch-token-mkwf9 elasticsearch-metrics elasticsearch-storage]: timed out waiting for the condition Warning FailedMount 30m (x3 over 55m) kubelet, compute-3 Unable to attach or mount volumes: unmounted volumes=[elasticsearch-storage], unattached volumes=[elasticsearch-storage elasticsearch-config certificates elasticsearch-token-mkwf9 elasticsearch-metrics]: timed out waiting for the condition Warning FailedMount 68s (x87 over 119m) kubelet, compute-3 (combined from similar events): MountVolume.SetUp failed for volume "pvc-1aec1a38-101c-11ea-91dc-0050568b94af" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af Output: Running scope as unit: run-r2b3d6075f1b241e0bd87b236011df39e.scope mount: /var/lib/kubelet/pods/1bc67be5-5c6f-44e7-ae96-73bc93fc198c/volumes/kubernetes.io~vsphere-volume/pvc-1aec1a38-101c-11ea-91dc-0050568b94af: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk does not exist. Actual results: Pod doesn't work after upgrade. Expected results: Pod should work after upgrade. Master Log: Node Log (of failed PODs): PV Dump: $ oc describe pvc elasticsearch-elasticsearch-cdm-1fnn2uw8-1 Name: elasticsearch-elasticsearch-cdm-1fnn2uw8-1 Namespace: openshift-logging StorageClass: thin Status: Bound Volume: pvc-1ae9daea-101c-11ea-91dc-0050568b94af Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pvc-protection] Capacity: 9537Mi Access Modes: RWO VolumeMode: Filesystem Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ProvisioningSucceeded 3h35m persistentvolume-controller Successfully provisioned volume pvc-1ae9daea-101c-11ea-91dc-0050568b94af using kubernetes.io/vsphere-volume Mounted By: elasticsearch-cdm-1fnn2uw8-1-65957d5b44-m6tr2 [wduan@dhcp-140-40 01_general]$ oc describe pv pvc-1ae9daea-101c-11ea-91dc-0050568b94af Name: pvc-1ae9daea-101c-11ea-91dc-0050568b94af Labels: <none> Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner pv.kubernetes.io/bound-by-controller: yes pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume Finalizers: [kubernetes.io/pv-protection] StorageClass: thin Status: Bound Claim: openshift-logging/elasticsearch-elasticsearch-cdm-1fnn2uw8-1 Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 9537Mi Node Affinity: <none> Message: Source: Type: vSphereVolume (a Persistent Disk resource in vSphere) VolumePath: [nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1ae9daea-101c-11ea-91dc-0050568b94af.vmdk FSType: ext4 StoragePolicyName: Events: <none> PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): $ oc describe sc thin Name: thin IsDefaultClass: Yes Annotations: storageclass.kubernetes.io/is-default-class=true Provisioner: kubernetes.io/vsphere-volume Parameters: diskformat=thin AllowVolumeExpansion: <unset> MountOptions: <none> ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none> [wduan@dhcp-140-40 01_general]$ oc get sc thin -oyaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: "2019-11-26T06:24:13Z" name: thin ownerReferences: - apiVersion: v1 kind: clusteroperator name: storage uid: c49c0a62-1013-11ea-b211-0050568ba4b9 resourceVersion: "11461" selfLink: /apis/storage.k8s.io/v1/storageclasses/thin uid: 5d6041e6-1015-11ea-ba6e-0050568b89a7 parameters: diskformat: thin provisioner: kubernetes.io/vsphere-volume reclaimPolicy: Delete volumeBindingMode: Immediate Additional info: 1. See volume existed in snapshot. 2. Check on the work node, not sure if something missing in the below configuration. sh-4.4# ps -eaf | grep kubelet root 1586 1 3 07:49 ? 00:06:50 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --node-labels=node-role.kubernetes.io/worker,node.openshift.io/os_id=rhcos --minimum-container-ttl-duration=6m0s --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec --cloud-provider= --v=3
Some more info about one of the PV/volume. $ oc get pv pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 1Gi RWO Delete Bound wduan/sc-resourcegroup-04 thin 166m $ oc get pv pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 -o yaml | grep volumePath volumePath: '[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902.vmdk' In vSphere events, Reconfigured compute-3 on vsphere-qe.vmware.devcluster.openshift.com in dc1. Modified: config.hardware.device(1000).device: (2000, 2002, 2001) -> (2000, 2002, 2001, 2003); Added: config.hardware.device(2003): (key = 2003, deviceInfo = (label = "Hard disk 4", summary = "1,048,576 KB"), backing = (fileName = "ds:///vmfs/volumes/5c9ce559-d9430ec0-e8d5-506b4bb49f6a/kubevols/qe-minmli-428-xzwsj-dynamic-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902.vmdk", datastore = 'vim.Datastore:c95eb2db-783e-4b6a-b867-01da64d6716e:datastore-266', backingObjectId = "", diskMode = "independent_persistent", split = false, writeThrough = false, thinProvisioned = true, eagerlyScrub = <unset>, uuid = "6000C299-4015-7f24-5b5a-7a735746b2d5", contentId = "6fee0501c61825b198c93044fffffffe", changeId = <unset>, parent = null, deltaDiskFormat = <unset>, digestEnabled = false, deltaGrainSize = <unset>, deltaDiskFormatVariant = <unset>, sharing = "sharingNone", keyId = null), connectable = null, slotInfo = null, controllerKey = 1000, unitNumber = 3, capacityInKB = 1048576, capacityInBytes = 1073741824, shares = (shares = 1000, level = "normal"), storageIOAllocation = (limit = -1, shares = (shares = 1000, level = "normal"), reservation = 0), diskObjectId = "3889-2003", vFlashCacheConfigInfo = null, iofilter = <unset>, vDiskId = null); config.extraConfig("scsi0:3.redo"): (key = "scsi0:3.redo", value = ""); Deleted: $ oc get pod -n wduan NAME READY STATUS RESTARTS AGE sc-resourcegroup-04 0/1 ContainerCreating 0 170m $ oc get pod sc-resourcegroup-04 -n wduan -o yaml | grep nodeName nodeName: compute-3 $ oc get node compute-3 -o yaml | tail -11 volumesAttached: - devicePath: /dev/disk/by-id/wwn-0x6000c2994512ce66e7773f4366e9bb8a name: kubernetes.io/vsphere-volume/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1ae9daea-101c-11ea-91dc-0050568b94af.vmdk - devicePath: /dev/disk/by-id/wwn-0x6000c29ac292917e9d030724cb6b45e4 name: kubernetes.io/vsphere-volume/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-1aec1a38-101c-11ea-91dc-0050568b94af.vmdk - devicePath: /dev/disk/by-id/wwn-0x6000c29940157f245b5a7a735746b2d5 name: kubernetes.io/vsphere-volume/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902.vmdk volumesInUse: - kubernetes.io/vsphere-volume/1bc67be5-5c6f-44e7-ae96-73bc93fc198c-pvc-1aec1a38-101c-11ea-91dc-0050568b94af - kubernetes.io/vsphere-volume/d6e9b986-807b-4b44-b077-50a8d1103255-pvc-1ae9daea-101c-11ea-91dc-0050568b94af - kubernetes.io/vsphere-volume/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 On the node compute-3, sh-4.4# /usr/lib/udev/scsi_id -g -u -d /dev/sdd 36000c29940157f245b5a7a735746b2d5 sh-4.4# mount | grep sdd sh-4.4# df -h | grep sdd sh-4.4# ls -lh /dev/sdd brw-rw----. 1 root disk 8, 48 Nov 26 10:00 /dev/sdd sh-4.4# fdisk -l /dev/sdd Disk /dev/sdd: 1 GiB, 1073741824 bytes, 2097152 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes sh-4.4# ls -lh /dev/disk/by-uuid/ total 0 lrwxrwxrwx. 1 root root 10 Nov 26 08:02 477c3d77-20c6-4ff3-8bb3-dc2543eedfbd -> ../../sda3 lrwxrwxrwx. 1 root root 9 Nov 26 09:58 544b95be-c9f0-4fc9-92f3-3942ea0fc81d -> ../../sdb lrwxrwxrwx. 1 root root 10 Nov 26 08:03 91de875e-af22-4585-91cb-e74437f6af68 -> ../../sda2 lrwxrwxrwx. 1 root root 9 Nov 26 08:04 d9ebc209-83dc-4bac-88dd-cb3eaeabbdda -> ../../sdc $ oc get pods -n wduan -o yaml | grep -w uid uid: e36bc1da-d27b-4caf-ae1b-e4fbfeec0808 sh-4.4# mount | grep e36bc1da-d27b-4caf-ae1b-e4fbfeec0808 tmpfs on /var/lib/kubelet/pods/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808/volumes/kubernetes.io~secret/default-token-5vkxf type tmpfs (rw,relatime,seclabel) sh-4.4# df -h | grep e36bc1da-d27b-4caf-ae1b-e4fbfeec0808 tmpfs 3.9G 24K 3.9G 1% /var/lib/kubelet/pods/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808/volumes/kubernetes.io~secret/default-token-5vkxf
$ oc describe pod -n wduan Name: sc-resourcegroup-04 Namespace: wduan Priority: 0 PriorityClassName: <none> Node: compute-3/139.178.76.25 Start Time: Tue, 26 Nov 2019 18:00:10 +0800 Labels: name=frontendhttp Annotations: openshift.io/scc: anyuid Status: Pending IP: Containers: myfrontend: Container ID: Image: docker.io/aosqe/hello-openshift Image ID: Port: 80/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /mnt/local from local (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-5vkxf (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: local: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: sc-resourcegroup-04 ReadOnly: false default-token-5vkxf: Type: Secret (a volume populated by a Secret) SecretName: default-token-5vkxf Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 29m (x10 over 151m) kubelet, compute-3 Unable to attach or mount volumes: unmounted volumes=[local], unattached volumes=[default-token-5vkxf local]: timed out waiting for the condition Warning FailedMount 9m48s (x92 over 3h2m) kubelet, compute-3 (combined from similar events): MountVolume.SetUp failed for volume "pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808/volumes/kubernetes.io~vsphere-volume/pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902.vmdk /var/lib/kubelet/pods/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808/volumes/kubernetes.io~vsphere-volume/pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902 Output: Running scope as unit: run-rd4c939ccc1564752b1995d2c19c3affb.scope mount: /var/lib/kubelet/pods/e36bc1da-d27b-4caf-ae1b-e4fbfeec0808/volumes/kubernetes.io~vsphere-volume/pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[nvme-ds1] kubevols/qe-minmli-428-xzwsj-dynamic-pvc-91bd1cbc-9e92-41d0-ae14-960084f3d902.vmdk does not exist. Warning FailedMount 4m9s (x61 over 3h2m) kubelet, compute-3 Unable to attach or mount volumes: unmounted volumes=[local], unattached volumes=[local default-token-5vkxf]: timed out waiting for the condition
*** Bug 1775685 has been marked as a duplicate of this bug. ***
*** Bug 1777195 has been marked as a duplicate of this bug. ***
For an update, we had some CI issues blocking merging this PR. We have resolved that problem and should be able to merge the fix into 4.3 as soon as within a couple hours: https://github.com/openshift/machine-config-operator/pull/1293
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062