Bug 1464477 - cinder disk doesn't get detached
Summary: cinder disk doesn't get detached
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.7.0
Assignee: Hemant Kumar
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-23 14:07 UTC by Vladislav Walek
Modified: 2020-08-13 09:27 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-07 19:38:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Vladislav Walek 2017-06-23 14:07:29 UTC
Description of problem:

bug related to
https://bugzilla.redhat.com/show_bug.cgi?id=1339295
https://bugzilla.redhat.com/show_bug.cgi?id=1359720

But occurs on openshift 3.4 with a difference. The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. 

Jun 19 17:58:28 node atomic-openshift-node: I0619 17:58:28.218779   47287 kubelet_volumes.go:158] Orphaned pod "282fce99-510e-11e7-8a64-fa163e3a6dec" found, but volumes are still mounted; err: <nil>, volumes: [pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24]
...
Jun 19 17:58:36 node atomic-openshift-node: W0619 17:58:36.509788   47287 mount.go:125] could not determine device for path: "/var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24"
Jun 19 17:58:36 node atomic-openshift-node: I0619 17:58:36.509820   47287 mount.go:179] Directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted
Jun 19 17:58:36 node atomic-openshift-node: E0619 17:58:36.509830   47287 reconciler.go:408] Could not construct volume information: directory /var/lib/origin/openshift.local.volumes/pods/282fce99-510e-11e7-8a64-fa163e3a6dec/volumes/kubernetes.io~cinder/pvc-266b22f6-510e-11e7-a1ea-fa163e52ca24 is not mounted

Version-Release number of selected component (if applicable):
OpenShift Conatiner Platform 3.4.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:


Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 2 Hemant Kumar 2017-06-28 01:17:07 UTC
> The volume is detached from the pod, but the pvc is still bound. Therefore the pod can't be rescheduled to the different node. 

Do you mean volume is unmounted but still attached to the node? Dynamically created PVCs are always in bound state and should have no bearing on use in different node.

Comment 4 Hemant Kumar 2017-06-28 02:21:44 UTC
I have tried reproducing this locally and I can not reproduce it. I also tried to reproduce with atomic-openshift-node stopped on node machine and then I deleted the pod. But still volume seems to be getting detached correctly.

@Vlad - I need some more help in figuring this out.

1. Can you post output of `lsblk` on node where volume is still attached? If it is still attached.
2. Can you post controller logs of the same duration when volume failed to detach from the node? The logs you posted above are atomic-openshift-node process log. We would also need logs from the controller.


Note You need to log in before you can comment on or make changes to this bug.