Description of problem: bug related to https://bugzilla.redhat.com/show_bug.cgi?id=1339295 https://bugzilla.redhat.com/show_bug.cgi?id=1359720 But occurs on openshift 3.4 with a difference. The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. Jun 19 17:58:28 node atomic-openshift-node: I0619 17:58:28.218779 47287 kubelet_volumes.go:158] Orphaned pod "282fce99-510e-11e7-8a64-fa163e3a6dec" found, but volumes are still mounted; err: <nil>, volumes: [pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24] ... Jun 19 17:58:36 node atomic-openshift-node: W0619 17:58:36.509788 47287 mount.go:125] could not determine device for path: "/var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24" Jun 19 17:58:36 node atomic-openshift-node: I0619 17:58:36.509820 47287 mount.go:179] Directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted Jun 19 17:58:36 node atomic-openshift-node: E0619 17:58:36.509830 47287 reconciler.go:408] Could not construct volume information: directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted Version-Release number of selected component (if applicable): OpenShift Conatiner Platform 3.4.0 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
> The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. Do you mean volume is unmounted but still attached to the node? Dynamically created PVCs are always in bound state and should have no bearing on use in different node.
I have tried reproducing this locally and I can not reproduce it. I also tried to reproduce with atomic-openshift-node stopped on node machine and then I deleted the pod. But still volume seems to be getting detached correctly. @Vlad - I need some more help in figuring this out. 1. Can you post output of `lsblk` on node where volume is still attached? If it is still attached. 2. Can you post controller logs of the same duration when volume failed to detach from the node? The logs you posted above are atomic-openshift-node process log. We would also need logs from the controller.