Description of problem: nova searches for wrong device unexpectedly when multipath device has faulty lun The error message is something following: TRACE nova.compute.manager [instance: d9db6ec9-0758-4565-acd8-e06bd6a29c3a] libvirtError: Cannot access storage file '/dev/mapper/Oct' (as uid:107, gid:107): No such file or directory As the above we can see that nova-compute is looking for a device called /dev/mapper/Oct. Version-Release number of selected component (if applicable): OSP 6.0 on RHEL 7 How reproducible: Steps to Reproduce: ( according to the customer ) 1. Make one lun faulty in multipath device 2. Start a nova instance 3. Check the nova-compute.log Actual results: A wrong device ( /dev/mapper/Oct ) is used Expected results: The instance should boot normally. Additional info:
Created attachment 1087152 [details] nova-compute.log
Thanks for the report Chen.Can you confirm what the output of `multipath -l ${device}` is with these faulty LUNs? I think we need to port the following change from cinder prior to the os-brick split : Multipath commands with error messages in stdout fail to parse https://review.openstack.org/#/c/128755/ I did try to get this into nova sometime ago but with the move to os-brick this was rejected : Multipath commands with error messages in stdout fail to parse https://review.openstack.org/#/c/172660/
*** Bug 1275936 has been marked as a duplicate of this bug. ***
*** Bug 1275935 has been marked as a duplicate of this bug. ***
Verified as follows - attach/detach volume to instance works even when there is error seen in multipath -ll output. ************ Version ************ [root@seal17 ~(keystone_admin)]# yum list installed | grep nova openstack-nova-api.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-cert.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-common.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-compute.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-conductor.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-console.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-novncproxy.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle openstack-nova-scheduler.noarch 2015.1.2-7.el7ost @rhelosp-7.0-puddle ************* Logs ************* [root@seal17 etc]# multipath -ll Dec 09 22:03:26 | missing value for option 'break-multipath' on line 1 of /etc/multipath.conf 36001405efa90fa3e42047ba886c7656e dm-1 LIO-ORG ,IBLOCK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active `- 10:0:0:0 sdb 8:16 active ready running [root@seal17 etc]# nova list [root@seal17 ~(keystone_admin)]# nova list +--------------------------------------+------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------+--------+------------+-------------+------------------+ | 6e56ec06-3287-4d02-90db-69905ecda71f | vm1 | ACTIVE | - | Running | private=10.0.0.4 | +--------------------------------------+------+--------+------------+-------------+------------------+ [root@seal17 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ | 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume | 1 | LUKS | false | | | 156fdd02-00ca-427e-a7be-4ca245f352d5 | in-use | vol1 | 1 | - | false | 6e56ec06-3287-4d02-90db-69905ecda71f | | 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 | error | nfsvol1 | 1 | nfs | false | | | 899809dd-5415-4f17-9fba-d0cc2846d838 | error | nfsvol1 | 1 | nfs | false | | | e4a9778e-ccac-49c1-8bc2-095f4f99af0f | available | nfsvol1 | 1 | - | false | | +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ [root@seal17 ~(keystone_admin)]# nova volume-attach vm1 e4a9778e-ccac-49c1-8bc2-095f4f99af0f +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdc | | id | e4a9778e-ccac-49c1-8bc2-095f4f99af0f | | serverId | 6e56ec06-3287-4d02-90db-69905ecda71f | | volumeId | e4a9778e-ccac-49c1-8bc2-095f4f99af0f | +----------+--------------------------------------+ [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# [root@seal17 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ | 001efa06-fbda-4c7f-bdfd-999b5b533923 | available | encrypted volume | 1 | LUKS | false | | | 156fdd02-00ca-427e-a7be-4ca245f352d5 | in-use | vol1 | 1 | - | false | 6e56ec06-3287-4d02-90db-69905ecda71f | | 2dde0a96-8d53-4a9f-8ffa-8ae04da9b1a0 | error | nfsvol1 | 1 | nfs | false | | | 899809dd-5415-4f17-9fba-d0cc2846d838 | error | nfsvol1 | 1 | nfs | false | | | e4a9778e-ccac-49c1-8bc2-095f4f99af0f | in-use | nfsvol1 | 1 | - | false | 6e56ec06-3287-4d02-90db-69905ecda71f | +--------------------------------------+-----------+------------------+------+-------------+----------+--------------------------------------+ [root@seal17 ~(keystone_admin)]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2673