Bug 1308919

Summary: unable to detach volume from the instance(HP 3PAR)
Product: Red Hat OpenStack Reporter: Pratik Pravin Bandarkar <pbandark>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: asv, eharney, geguileo, jraju, pbandark, sgotliv, vaggarwa, yeylon
Target Milestone: asyncKeywords: Unconfirmed, ZStream
Target Release: 6.0 (Juno)Flags: vaggarwa: needinfo+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-cinder-2014.2.4-6.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-10 17:06:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
3par_volume_hostinfo none

Description Pratik Pravin Bandarkar 2016-02-16 13:18:28 UTC
Description of problem:

- We have an instance named "feb3-selinux" which has a 5GB volume attached to it.
=======================================
[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk /data1
[root@feb3-selinux data1]# date
Tue Feb 16 05:35:13 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
====================================

- Next, we try to detach the volume from the Horizon dashboard. The volume is detached from the instance, but the state on the horizon dashboard keeps as detaching even after very long time.

- But, the volume is not attached to the instance any longer.

[root@feb3-selinux data1]# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  20G  0 disk
└─vda1 253:1    0  20G  0 part /

[root@feb3-selinux data1]# date
Tue Feb 16 05:42:34 EST 2016
[root@feb3-selinux data1]# df -kh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  859M   20G   5% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   17M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb        4.6G   19M  4.3G   1% /data1
[root@feb3-selinux data1]# cd /data1
[root@feb3-selinux data1]#
[root@feb3-selinux data1]# ls -lrth
ls: reading directory .: Input/output error
total 0

- The volume is not shown in the multipath O/p on the compute node.

- When the volume is in detaching state, i manually go and update the status of the volume as available. But in the attached to column , it still shows "attached to feb3-selinux", even when it is actually not.

- After this, surprisingly when i do a hard reboot of the isntance, it actually attaches that volume back to the instance.
Find the below O/p for your reference.
============================
[gaurav.MCKD-4041] ➤ ssh root.xx.xx
X11 forwarding request failed on channel 0
Last login: Tue Feb 16 05:11:03 2016 from xx.xx.xx.xx
[root@feb3-selinux ~]#
[root@feb3-selinux ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G  0 disk
└─vda1 253:1    0   20G  0 part /
vdb    253:16   0  4.8G  0 disk
[root@feb3-selinux ~]#

[root@comp3 ~]# multipath -ll | grep 360002ac0000000000000006400008044
360002ac0000000000000006400008044 dm-13 3PARdata,VV


Version-Release number of selected component (if applicable):
- RHOS6(non-ha)
- HP 3PAR backend for cinder(cinder.volume.drivers.san.hp.hp_3par_fc.HP3PARFCDriver)

How reproducible:
100%


Steps to Reproduce:
1.
2.
3.

Actual results:
unable to detach volume from instance.

Expected results:
volume should be detached from the instance.

Additional info:

Comment 10 Sergey Gotliv 2016-02-17 09:00:38 UTC
HP 3PAR driver fails to delete a vlun, most probably because volume doesn't exist on the storage anymore. We have to escalate this issue to HP.

2016-02-16 16:09:07.501 55809 ERROR cinder.volume.drivers.san.hp.hp_3par_common [req-7aff09d7-2a6f-4c4f-aa78-da3d43876568 0691b74100f64062a160f986269f2ed5 767f5371e9d94a38b1169b4870d73daf - - -] Not found (HTTP 404) 17 - host does not exist

Comment 16 Sergey Gotliv 2016-02-17 13:29:31 UTC
It seems that others also experience very similar problems trying detach cinder volume backed by the HP 3PAR driver. Its not necessarily exactly the same problem, but the error in the log would look the same.

https://bugs.launchpad.net/cinder/+bug/1546453

Comment 17 Lee Yarwood 2016-02-17 13:46:33 UTC
*** Bug 1309316 has been marked as a duplicate of this bug. ***

Comment 19 Pratik Pravin Bandarkar 2016-02-22 07:14:13 UTC
Created attachment 1129166 [details]
3par_volume_hostinfo

Comment 35 errata-xmlrpc 2016-03-10 17:06:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0436.html