Bug 1464477 - cinder disk doesn't get detached
cinder disk doesn't get detached
Status: CLOSED DEFERRED
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.4.0
Unspecified Unspecified
unspecified Severity medium
: ---
: 3.7.0
Assigned To: Hemant Kumar
Jianwei Hou
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 10:07 EDT by Vladislav Walek
Modified: 2017-10-27 08:32 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-07 15:38:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vladislav Walek 2017-06-23 10:07:29 EDT
Description of problem:

bug related to
https://bugzilla.redhat.com/show_bug.cgi?id=1339295
https://bugzilla.redhat.com/show_bug.cgi?id=1359720

But occurs on openshift 3.4 with a difference. The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. 

Jun 19 17:58:28 node atomic-openshift-node: I0619 17:58:28.218779   47287 kubelet_volumes.go:158] Orphaned pod "282fce99-510e-11e7-8a64-fa163e3a6dec" found, but volumes are still mounted; err: <nil>, volumes: [pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24]
...
Jun 19 17:58:36 node atomic-openshift-node: W0619 17:58:36.509788   47287 mount.go:125] could not determine device for path: "/var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24"
Jun 19 17:58:36 node atomic-openshift-node: I0619 17:58:36.509820   47287 mount.go:179] Directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted
Jun 19 17:58:36 node atomic-openshift-node: E0619 17:58:36.509830   47287 reconciler.go:408] Could not construct volume information: directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted

Version-Release number of selected component (if applicable):
OpenShift Conatiner Platform 3.4.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:


Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Comment 2 Hemant Kumar 2017-06-27 21:17:07 EDT
> The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. 

Do you mean volume is unmounted but still attached to the node? Dynamically created PVCs are always in bound state and should have no bearing on use in different node.
Comment 4 Hemant Kumar 2017-06-27 22:21:44 EDT
I have tried reproducing this locally and I can not reproduce it. I also tried to reproduce with atomic-openshift-node stopped on node machine and then I deleted the pod. But still volume seems to be getting detached correctly.

@Vlad - I need some more help in figuring this out.

1. Can you post output of `lsblk` on node where volume is still attached? If it is still attached.
2. Can you post controller logs of the same duration when volume failed to detach from the node? The logs you posted above are atomic-openshift-node process log. We would also need logs from the controller.

Note You need to log in before you can comment on or make changes to this bug.