Bug 1235786 - multipath errors being parsed as device names.
Summary: multipath errors being parsed as device names.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 6.0 (Juno)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 8.0 (Liberty)
Assignee: Eric Harney
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-25 18:28 UTC by Lee Yarwood
Modified: 2019-08-15 04:46 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-23 14:48:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1468864 0 None None None Never
OpenStack gerrit 165459 0 None None None Never
OpenStack gerrit 165460 0 None None None Never

Description Lee Yarwood 2015-06-25 18:28:28 UTC
Description of problem:
The following utility method incorrectly parses multipath errors as a device name :

cinder/brick/initiator/connector.py

~~~
473     def _get_multipath_device_name(self, single_path_device):
474         device = os.path.realpath(single_path_device)
475         out = self._run_multipath(['-ll',
476                                   device],
477                                   check_exit_code=[0, 1])[0]
478         mpath_line = [line for line in out.splitlines()
479                       if "scsi_id" not in line]  # ignore udev errors
480         if len(mpath_line) > 0 and len(mpath_line[0]) > 0:
481             return "/dev/mapper/%s" % mpath_line[0].split(" ")[0]
482 
483         return None
~~~

This results in the following error, this later escalates into an odd encoding exception slightly masking this original failure :

~~~
12701 2015-06-18 17:00:06.656 1228 DEBUG cinder.brick.initiator.connector [req-c5ed9a90-c0de-44bb-a602-7d7168c51180 916f3ee5132143dbb723231472118a22 7b1de946cd814605aaca2082ffc650c7 - - -] multipath ['-ll', u'/dev/sdv']: stdout=Jun 18 17:00:06 | vda: No fc_host device for 'host-1'
12702 Jun 18 17:00:06 | vda: No fc_host device for 'host-1'
12703 Jun 18 17:00:06 | vda: No fc_remote_port device for 'rport--1:-1-0'
12704 mpathe (3624a9370f5786ee930acb2bc00011075) dm-5 PURE    ,FlashArray
12705 size=20G features='0' hwhandler='0' wp=rw
12706 |-+- policy='service-time 0' prio=1 status=active
12707 | `- 7:0:0:6  sdv 65:80  active ready running
12708 |-+- policy='service-time 0' prio=1 status=enabled
12709 | `- 10:0:0:6 sdu 65:64  active ready running
12710 |-+- policy='service-time 0' prio=1 status=enabled
12711 | `- 9:0:0:6  sdw 65:96  active ready running
12712 `-+- policy='service-time 0' prio=1 status=enabled
12713   `- 8:0:0:6  sdx 65:112 active ready running
12714  stderr= _run_multipath /usr/lib/python2.7/site-packages/cinder/brick/initiator/connector.py:607
12715 2015-06-18 17:00:06.656 1228 DEBUG cinder.openstack.common.processutils [req-c5ed9a90-c0de-44bb-a602-7d7168c51180 916f3ee5132143dbb723231472118a22 7b1de946cd814605aaca2082ffc650c7 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf dd if=/dev/mapper/Jun of=/dev/null count=1 execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:158
~~~

Version-Release number of selected component (if applicable):
openstack-cinder-2014.2.3-5.el7ost.noarch

How reproducible:
Always.

Steps to Reproduce:
1. Use a multipath enabled backend such as the currently uncertified `PureStorage` backend.
2. Ensure `multipath -ll` prints some form of error or warning on the first line.
3. Attempt to attach a volume using the above backend.

Actual results:
Attaching the volume fails as the error string is parsed as the device name.

Expected results:
Error string ignored and the volume is attached.

Additional info:

Simple workaround is to remove errors such as the above by blacklisting local devices etc.

I have a simple patch in mind for this to reuse the linuxscsi utility methods I've been working on, I'll push this upstream to os-brick shortly and into cinder for stable/kilo.

Comment 3 Eric Harney 2015-06-25 18:39:01 UTC
This is probably fixed by https://review.openstack.org/#/c/165459/ .

Comment 4 Lee Yarwood 2015-06-26 09:07:42 UTC
(In reply to Eric Harney from comment #3)
> This is probably fixed by https://review.openstack.org/#/c/165459/ .

Yup, still it would be nice to remove the bulk of this method and just call self._linuxscsi.find_multipath_device(). As it stands the above patch would still fail if any action was being carried out on the multipath device (LP#1466444).

Comment 5 Sergey Gotliv 2015-10-19 05:53:13 UTC
Lee,

As you confirmed here [1], the issue is resolved in the latest os-brick version.
I am closing that bug. 

[1] https://bugs.launchpad.net/os-brick/+bug/1468864/comments/2


Note You need to log in before you can comment on or make changes to this bug.