Bug 1308919 - unable to detach volume from the instance(HP 3PAR)
unable to detach volume from the instance(HP 3PAR)
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
6.0 (Juno)
All Linux
high Severity medium
: async
: 6.0 (Juno)
Assigned To: Eric Harney
nlevinki
: Unconfirmed, ZStream
: 1309316 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-16 08:18 EST by Pratik Pravin Bandarkar
Modified: 2017-02-20 00:48 EST (History)
8 users (show)

See Also:
Fixed In Version: openstack-cinder-2014.2.4-6.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-03-10 12:06:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
vaggarwa: needinfo+


Attachments (Terms of Use)
3par_volume_hostinfo (113.10 KB, image/png)
2016-02-22 02:14 EST, Pratik Pravin Bandarkar
no flags Details

  None (edit)
Description Pratik Pravin Bandarkar 2016-02-16 08:18:28 EST
Description of problem:

- We have an instance named "feb3-selinux" which has a 5GB volume attached to it.
=======================================
[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk /data1
[root@feb3-selinux data1]# date
Tue Feb 16 05:35:13 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
====================================

- Next, we try to detach the volume from the Horizon dashboard. The volume is detached from the instance, but the state on the horizon dashboard keeps as detaching even after very long time.

- But, the volume is not attached to the instance any longer.

[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  20G  0 disk
└─vda1 253:1    0  20G  0 part /

[root@feb3-selinux data1]# date
Tue Feb 16 05:42:34 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
[root@feb3-selinux data1]# cd /data1
[root@feb3-selinux data1]#
[root@feb3-selinux data1]# ls -lrth
ls: reading directory .: Input/output error
total 0

- The volume is not shown in the multipath O/p on the compute node.

- When the volume is in detaching state, i manually go and update the status of the volume as available. But in the attached to column , it still shows "attached to feb3-selinux", even when it is actually not.

- After this, surprisingly when i do a hard reboot of the isntance, it actually attaches that volume back to the instance.
Find the below O/p for your reference.
============================
[gaurav.MCKD-4041] ➤ ssh root@xx.xx.xx.xx
X11 forwarding request failed on channel 0
Last login: Tue Feb 16 05:11:03 2016 from xx.xx.xx.xx
[root@feb3-selinux ~]#
[root@feb3-selinux ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk
[root@feb3-selinux ~]#

[root@comp3 ~]# multipath -ll | grep 360002ac0000000000000006400008044
360002ac0000000000000006400008044 dm-13 3PARdata,VV


Version-Release number of selected component (if applicable):
- RHOS6(non-ha)
- HP 3PAR backend for cinder(cinder.volume.drivers.san.hp.hp_3par_fc.HP3PARFCDriver)

How reproducible:
100%


Steps to Reproduce:
1.
2.
3.

Actual results:
unable to detach volume from instance.

Expected results:
volume should be detached from the instance.

Additional info:
Comment 10 Sergey Gotliv 2016-02-17 04:00:38 EST
HP 3PAR driver fails to delete a vlun, most probably because volume doesn't exist on the storage anymore. We have to escalate this issue to HP.

2016-02-16 16:09:07.501 55809 ERROR cinder.volume.drivers.san.hp.hp_3par_common [req-7aff09d7-2a6f-4c4f-aa78-da3d43876568 0691b74100f64062a160f986269f2ed5 767f5371e9d94a38b1169b4870d73daf - - -] Not found (HTTP 404) 17 - host does not exist
Comment 16 Sergey Gotliv 2016-02-17 08:29:31 EST
It seems that others also experience very similar problems trying detach cinder volume backed by the HP 3PAR driver. Its not necessarily exactly the same problem, but the error in the log would look the same.

https://bugs.launchpad.net/cinder/+bug/1546453
Comment 17 Lee Yarwood 2016-02-17 08:46:33 EST
*** Bug 1309316 has been marked as a duplicate of this bug. ***
Comment 19 Pratik Pravin Bandarkar 2016-02-22 02:14 EST
Created attachment 1129166 [details]
3par_volume_hostinfo
Comment 35 errata-xmlrpc 2016-03-10 12:06:48 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0436.html

Note You need to log in before you can comment on or make changes to this bug.