Bug 1308919 - unable to detach volume from the instance(HP 3PAR)
Summary: unable to detach volume from the instance(HP 3PAR)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 6.0 (Juno)
Hardware: All
OS: Linux
high
medium
Target Milestone: async
: 6.0 (Juno)
Assignee: Eric Harney
QA Contact: nlevinki
URL:
Whiteboard:
: 1309316 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-16 13:18 UTC by Pratik Pravin Bandarkar
Modified: 2019-10-10 11:13 UTC (History)
8 users (show)

Fixed In Version: openstack-cinder-2014.2.4-6.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-10 17:06:48 UTC
Target Upstream Version:
Embargoed:
vaggarwa: needinfo+


Attachments (Terms of Use)
3par_volume_hostinfo (113.10 KB, image/png)
2016-02-22 07:14 UTC, Pratik Pravin Bandarkar
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0436 0 normal SHIPPED_LIVE openstack-cinder bug fix advisory 2016-03-10 22:05:37 UTC

Description Pratik Pravin Bandarkar 2016-02-16 13:18:28 UTC
Description of problem:

- We have an instance named "feb3-selinux" which has a 5GB volume attached to it.
=======================================
[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk /data1
[root@feb3-selinux data1]# date
Tue Feb 16 05:35:13 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
====================================

- Next, we try to detach the volume from the Horizon dashboard. The volume is detached from the instance, but the state on the horizon dashboard keeps as detaching even after very long time.

- But, the volume is not attached to the instance any longer.

[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  20G  0 disk
└─vda1 253:1    0  20G  0 part /

[root@feb3-selinux data1]# date
Tue Feb 16 05:42:34 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
[root@feb3-selinux data1]# cd /data1
[root@feb3-selinux data1]#
[root@feb3-selinux data1]# ls -lrth
ls: reading directory .: Input/output error
total 0

- The volume is not shown in the multipath O/p on the compute node.

- When the volume is in detaching state, i manually go and update the status of the volume as available. But in the attached to column , it still shows "attached to feb3-selinux", even when it is actually not.

- After this, surprisingly when i do a hard reboot of the isntance, it actually attaches that volume back to the instance.
Find the below O/p for your reference.
============================
[gaurav.MCKD-4041] ➤ ssh root.xx.xx
X11 forwarding request failed on channel 0
Last login: Tue Feb 16 05:11:03 2016 from xx.xx.xx.xx
[root@feb3-selinux ~]#
[root@feb3-selinux ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk
[root@feb3-selinux ~]#

[root@comp3 ~]# multipath -ll | grep 360002ac0000000000000006400008044
360002ac0000000000000006400008044 dm-13 3PARdata,VV


Version-Release number of selected component (if applicable):
- RHOS6(non-ha)
- HP 3PAR backend for cinder(cinder.volume.drivers.san.hp.hp_3par_fc.HP3PARFCDriver)

How reproducible:
100%


Steps to Reproduce:
1.
2.
3.

Actual results:
unable to detach volume from instance.

Expected results:
volume should be detached from the instance.

Additional info:

Comment 10 Sergey Gotliv 2016-02-17 09:00:38 UTC
HP 3PAR driver fails to delete a vlun, most probably because volume doesn't exist on the storage anymore. We have to escalate this issue to HP.

2016-02-16 16:09:07.501 55809 ERROR cinder.volume.drivers.san.hp.hp_3par_common [req-7aff09d7-2a6f-4c4f-aa78-da3d43876568 0691b74100f64062a160f986269f2ed5 767f5371e9d94a38b1169b4870d73daf - - -] Not found (HTTP 404) 17 - host does not exist

Comment 16 Sergey Gotliv 2016-02-17 13:29:31 UTC
It seems that others also experience very similar problems trying detach cinder volume backed by the HP 3PAR driver. Its not necessarily exactly the same problem, but the error in the log would look the same.

https://bugs.launchpad.net/cinder/+bug/1546453

Comment 17 Lee Yarwood 2016-02-17 13:46:33 UTC
*** Bug 1309316 has been marked as a duplicate of this bug. ***

Comment 19 Pratik Pravin Bandarkar 2016-02-22 07:14:13 UTC
Created attachment 1129166 [details]
3par_volume_hostinfo

Comment 35 errata-xmlrpc 2016-03-10 17:06:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0436.html


Note You need to log in before you can comment on or make changes to this bug.