Bug 1275937

Summary: nova searches for wrong device unexpectedly when multipath device has faulty lun
Product: Red Hat OpenStack Reporter: Chen <cchen>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: Prasanth Anbalagan <panbalag>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: berrange, cchen, dasmith, eglynn, kchamart, lyarwood, ndipanov, panbalag, pbrady, rbiba, sbauza, sferdjao, sgordon, vromanso, yeylon
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-nova-2015.1.2-6.el7ost Doc Type: Bug Fix
Doc Text:
Previously, Nova failed to parse the output from the "multipath -l ${device}" command when errors were present. Consequently, the attaching and detaching of volumes could fail. This update corrects the find_multipath_device method to ensure that any errors present in the output from the aforementioned command are ignored. As a result, both the attaching and detaching of volumes will now succeed even if errors occur.
 Story Points: ---
Clone Of:
: 1280359 1281762 (view as bug list) Environment:
Last Closed: 2015-12-21 17:08:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1278209, 1280359, 1281762    
Attachments:
Description Flags
nova-compute.log none

Description Chen 2015-10-28 07:03:56 UTC 
Description of problem:

nova searches for wrong device unexpectedly when multipath device has faulty lun

The error message is something following:

TRACE nova.compute.manager [instance: d9db6ec9-0758-4565-acd8-e06bd6a29c3a] libvirtError: Cannot access storage file '/dev/mapper/Oct' (as uid:107, gid:107): No such file or directory

As the above we can see that nova-compute is looking for a device called /dev/mapper/Oct.

Version-Release number of selected component (if applicable):

OSP 6.0 on RHEL 7

How reproducible:



Steps to Reproduce: ( according to the customer )
1. Make one lun faulty in multipath device
2. Start a nova instance
3. Check the nova-compute.log

Actual results:

A wrong device ( /dev/mapper/Oct ) is used

Expected results:

The instance should boot normally.

Additional info:
Comment 2 Chen 2015-10-28 07:05:23 UTC 
Created attachment 1087152 [details]
nova-compute.log
Comment 3 Lee Yarwood 2015-10-29 14:18:00 UTC 
Thanks for the report Chen.Can you confirm what the output of `multipath -l ${device}` is with these faulty LUNs? I think we need to port the following change from cinder prior to the os-brick split :

Multipath commands with error messages in stdout fail to parse
https://review.openstack.org/#/c/128755/

I did try to get this into nova sometime ago but with the move to os-brick this was rejected :

Multipath commands with error messages in stdout fail to parse
https://review.openstack.org/#/c/172660/
Comment 4 Lee Yarwood 2015-10-30 16:47:21 UTC 
*** Bug 1275936 has been marked as a duplicate of this bug. ***
Comment 5 Lee Yarwood 2015-10-30 16:47:24 UTC 
*** Bug 1275935 has been marked as a duplicate of this bug. ***
Comment 14 Prasanth Anbalagan 2015-12-09 20:13:14 UTC 
Verified as follows - attach/detach volume to instance works even when there is error seen in multipath -ll output.

************
Version
************

[root@seal17 ~(keystone_admin)]# yum list installed | grep  nova
openstack-nova-api.noarch            2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-cert.noarch           2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-common.noarch         2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-compute.noarch        2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-conductor.noarch      2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-console.noarch        2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-novncproxy.noarch     2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-scheduler.noarch      2015.1.2-7.el7ost       @rhelosp-7.0-puddle

*************
Logs
*************

[root@seal17 etc]# multipath -ll
Dec 09 22:03:26 | missing value for option 'break-multipath' on line 1 of /etc/multipath.conf
36001405efa90fa3e42047ba886c7656e dm-1 LIO-ORG ,IBLOCK          
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 10:0:0:0 sdb 8:16 active ready  running
[root@seal17 etc]# nova list
[root@seal17 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID                                   | Name | Status | Task State | Power State | Networks         |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6e56ec06-3287-4d02-90db-69905ecda71f | vm1  | ACTIVE | -          | Running     | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
[root@seal17 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |   Display Name   | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume |  1   |     LUKS    |  false   |                                      |
| 156fdd02-00ca-427e-a7be-4ca245f352d5 |   in-use  |       vol1       |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
| 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| 899809dd-5415-4f17-9fba-d0cc2846d838 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| e4a9778e-ccac-49c1-8bc2-095f4f99af0f | available |     nfsvol1      |  1   |      -      |  false   |                                      |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]# nova volume-attach vm1 e4a9778e-ccac-49c1-8bc2-095f4f99af0f
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | e4a9778e-ccac-49c1-8bc2-095f4f99af0f |
| serverId | 6e56ec06-3287-4d02-90db-69905ecda71f |
| volumeId | e4a9778e-ccac-49c1-8bc2-095f4f99af0f |
+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |   Display Name   | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume |  1   |     LUKS    |  false   |                                      |
| 156fdd02-00ca-427e-a7be-4ca245f352d5 |   in-use  |       vol1       |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
| 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| 899809dd-5415-4f17-9fba-d0cc2846d838 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| e4a9778e-ccac-49c1-8bc2-095f4f99af0f |   in-use  |     nfsvol1      |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]#
Comment 16 errata-xmlrpc 2015-12-21 17:08:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2673