This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1275937 - nova searches for wrong device unexpectedly when multipath device has faulty lun
nova searches for wrong device unexpectedly when multipath device has faulty lun
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
6.0 (Juno)
Unspecified Unspecified
medium Severity medium
: z3
: 7.0 (Kilo)
Assigned To: Lee Yarwood
Prasanth Anbalagan
: Triaged, ZStream
: 1275935 1275936 (view as bug list)
Depends On:
Blocks: 1278209 1280359 1281762
  Show dependency treegraph
 
Reported: 2015-10-28 03:03 EDT by Chen
Modified: 2015-12-21 12:08 EST (History)
17 users (show)

See Also:
Fixed In Version: openstack-nova-2015.1.2-6.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, Nova failed to parse the output from the "multipath -l ${device}" command when errors were present. Consequently, the attaching and detaching of volumes could fail. This update corrects the find_multipath_device method to ensure that any errors present in the output from the aforementioned command are ignored. As a result, both the attaching and detaching of volumes will now succeed even if errors occur.
Story Points: ---
Clone Of:
: 1280359 1281762 (view as bug list)
Environment:
Last Closed: 2015-12-21 12:08:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
nova-compute.log (424.25 KB, application/zip)
2015-10-28 03:05 EDT, Chen
no flags Details

  None (edit)
Description Chen 2015-10-28 03:03:56 EDT
Description of problem:

nova searches for wrong device unexpectedly when multipath device has faulty lun

The error message is something following:

TRACE nova.compute.manager [instance: d9db6ec9-0758-4565-acd8-e06bd6a29c3a] libvirtError: Cannot access storage file '/dev/mapper/Oct' (as uid:107, gid:107): No such file or directory

As the above we can see that nova-compute is looking for a device called /dev/mapper/Oct.

Version-Release number of selected component (if applicable):

OSP 6.0 on RHEL 7

How reproducible:



Steps to Reproduce: ( according to the customer )
1. Make one lun faulty in multipath device
2. Start a nova instance
3. Check the nova-compute.log

Actual results:

A wrong device ( /dev/mapper/Oct ) is used

Expected results:

The instance should boot normally.

Additional info:
Comment 2 Chen 2015-10-28 03:05 EDT
Created attachment 1087152 [details]
nova-compute.log
Comment 3 Lee Yarwood 2015-10-29 10:18:00 EDT
Thanks for the report Chen.Can you confirm what the output of `multipath -l ${device}` is with these faulty LUNs? I think we need to port the following change from cinder prior to the os-brick split :

Multipath commands with error messages in stdout fail to parse
https://review.openstack.org/#/c/128755/

I did try to get this into nova sometime ago but with the move to os-brick this was rejected :

Multipath commands with error messages in stdout fail to parse
https://review.openstack.org/#/c/172660/
Comment 4 Lee Yarwood 2015-10-30 12:47:21 EDT
*** Bug 1275936 has been marked as a duplicate of this bug. ***
Comment 5 Lee Yarwood 2015-10-30 12:47:24 EDT
*** Bug 1275935 has been marked as a duplicate of this bug. ***
Comment 14 Prasanth Anbalagan 2015-12-09 15:13:14 EST
Verified as follows - attach/detach volume to instance works even when there is error seen in multipath -ll output.

************
Version
************

[root@seal17 ~(keystone_admin)]# yum list installed | grep  nova
openstack-nova-api.noarch            2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-cert.noarch           2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-common.noarch         2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-compute.noarch        2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-conductor.noarch      2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-console.noarch        2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-novncproxy.noarch     2015.1.2-7.el7ost       @rhelosp-7.0-puddle
openstack-nova-scheduler.noarch      2015.1.2-7.el7ost       @rhelosp-7.0-puddle

*************
Logs
*************

[root@seal17 etc]# multipath -ll
Dec 09 22:03:26 | missing value for option 'break-multipath' on line 1 of /etc/multipath.conf
36001405efa90fa3e42047ba886c7656e dm-1 LIO-ORG ,IBLOCK          
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 10:0:0:0 sdb 8:16 active ready  running
[root@seal17 etc]# nova list
[root@seal17 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID                                   | Name | Status | Task State | Power State | Networks         |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6e56ec06-3287-4d02-90db-69905ecda71f | vm1  | ACTIVE | -          | Running     | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
[root@seal17 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |   Display Name   | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume |  1   |     LUKS    |  false   |                                      |
| 156fdd02-00ca-427e-a7be-4ca245f352d5 |   in-use  |       vol1       |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
| 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| 899809dd-5415-4f17-9fba-d0cc2846d838 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| e4a9778e-ccac-49c1-8bc2-095f4f99af0f | available |     nfsvol1      |  1   |      -      |  false   |                                      |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]# nova volume-attach vm1 e4a9778e-ccac-49c1-8bc2-095f4f99af0f
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | e4a9778e-ccac-49c1-8bc2-095f4f99af0f |
| serverId | 6e56ec06-3287-4d02-90db-69905ecda71f |
| volumeId | e4a9778e-ccac-49c1-8bc2-095f4f99af0f |
+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# 
[root@seal17 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |   Display Name   | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
| 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume |  1   |     LUKS    |  false   |                                      |
| 156fdd02-00ca-427e-a7be-4ca245f352d5 |   in-use  |       vol1       |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
| 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| 899809dd-5415-4f17-9fba-d0cc2846d838 |   error   |     nfsvol1      |  1   |     nfs     |  false   |                                      |
| e4a9778e-ccac-49c1-8bc2-095f4f99af0f |   in-use  |     nfsvol1      |  1   |      -      |  false   | 6e56ec06-3287-4d02-90db-69905ecda71f |
+--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+
[root@seal17 ~(keystone_admin)]#
Comment 16 errata-xmlrpc 2015-12-21 12:08:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2673

Note You need to log in before you can comment on or make changes to this bug.