Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1900595

Summary: multipath should print a warning if /etc/multipath.conf includes options with invalid values.
Product: Red Hat Enterprise Linux 8 Reporter: Lin Li <lilin>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED ERRATA QA Contact: Lin Li <lilin>
Severity: low Docs Contact:
Priority: low    
Version: 8.4CC: agk, bmarzins, borgan, chetjain, heinzm, iranna.ankad, jbrassow, msnitzer, prajnoha, zkabelac
Target Milestone: rcKeywords: Triaged
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: device-mapper-multipath-0.8.4-20.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2017969 (view as bug list) Environment:
Last Closed: 2022-05-10 15:21:50 UTC Type: Enhancement
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2017969    

Description Lin Li 2020-11-23 12:37:31 UTC
Description of problem:
[ppc64le]  multipath doesn't report error while setting path_selector to wrong value

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.8.4-5.el8.ppc64le

How reproducible:
always

Steps to Reproduce:
1.[root@ibm-p9z-20-lp15 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.4-5.el8.ppc64le
device-mapper-multipath-0.8.4-5.el8.ppc64le
2. Edit /etc/multipath.conf
devices {
        device {
                vendor "^DGC"
                product "^(RAID|DISK|VRAID)"
                path_grouping_policy "group_by_prio"
                user_friendly_names "yes"
                path_selector "round-robin 0"
        }
}


3.[root@ibm-p9z-20-lp15 ~]# multipath -ll
mpatha (3600601605e203f008e22d0221f07eb11) dm-0 DGC,VRAID
size=8.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:1:0 sdb 8:16 active ready running
| `- 2:0:1:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0  active ready running
  `- 2:0:0:0 sdc 8:32 active ready running



4.Edit /etc/multipath.conf
devices {
        device {
                vendor "^DGC"
                product "^(RAID|DISK|VRAID)"
                path_grouping_policy "group_by_prio"
                user_friendly_names "yes"
                path_selector "path_selector 0"  <-------set wrong value

        }
}


5.[root@ibm-p9z-20-lp15 ~]# service multipathd reload
Redirecting to /bin/systemctl reload multipathd.service

6.[root@ibm-p9z-20-lp15 ~]# multipath -ll
mpatha (3600601605e203f008e22d0221f07eb11) dm-0 DGC,VRAID
size=8.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:1:0 sdb 8:16 active ready running
| `- 2:0:1:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0  active ready running
  `- 2:0:0:0 sdc 8:32 active ready running


Actual results:
multipathd doesn't report error while setting path_selector to wrong value 

Expected results:
multipathd should report an error while setting path_selector to wrong value

Additional info:
This is a ppc64le power server

Comment 1 Lin Li 2020-11-23 12:42:34 UTC
"unknown path selector type" can be seen.

[root@ibm-p9z-20-lp15 ~]# dmesg
[  509.463393] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[  949.357893] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[ 1035.160452] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[ 1205.676710] device-mapper: multipath round-robin: version 1.2.0 loaded
[ 1205.855430] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[ 1264.981102] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1264.981111] device-mapper: ioctl: error adding target to table
[ 1264.997627] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1264.997632] device-mapper: ioctl: error adding target to table
[ 1265.014143] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1265.014147] device-mapper: ioctl: error adding target to table
[ 1265.030584] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1265.030589] device-mapper: ioctl: error adding target to table
[ 1306.954637] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1306.954645] device-mapper: ioctl: error adding target to table
[ 1306.971042] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1306.971047] device-mapper: ioctl: error adding target to table
[ 1306.987459] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1306.987464] device-mapper: ioctl: error adding target to table
[ 1307.003884] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1307.003889] device-mapper: ioctl: error adding target to table
[ 1431.632820] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1431.632827] device-mapper: ioctl: error adding target to table
[ 1431.649284] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1431.649289] device-mapper: ioctl: error adding target to table
[ 1431.665681] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1431.665686] device-mapper: ioctl: error adding target to table
[ 1431.682016] device-mapper: table: 253:0: multipath: unknown path selector type
[ 1431.682020] device-mapper: ioctl: error adding target to table
[ 2214.163339] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2214.163347] device-mapper: ioctl: error adding target to table
[ 2214.179843] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2214.179847] device-mapper: ioctl: error adding target to table
[ 2214.196233] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2214.196237] device-mapper: ioctl: error adding target to table
[ 2214.212597] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2214.212601] device-mapper: ioctl: error adding target to table
[ 2364.197866] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2364.197875] device-mapper: ioctl: error adding target to table
[ 2364.214278] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2364.214283] device-mapper: ioctl: error adding target to table
[ 2364.230656] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2364.230660] device-mapper: ioctl: error adding target to table
[ 2364.247064] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2364.247068] device-mapper: ioctl: error adding target to table
[ 2851.516323] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[ 2931.655755] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2931.655763] device-mapper: ioctl: error adding target to table
[ 2931.672330] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2931.672336] device-mapper: ioctl: error adding target to table
[ 2931.688920] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2931.688926] device-mapper: ioctl: error adding target to table
[ 2931.705644] device-mapper: table: 253:0: multipath: unknown path selector type
[ 2931.705652] device-mapper: ioctl: error adding target to table
[ 3116.218749] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3116.218758] device-mapper: ioctl: error adding target to table
[ 3116.235189] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3116.235193] device-mapper: ioctl: error adding target to table
[ 3116.251550] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3116.251555] device-mapper: ioctl: error adding target to table
[ 3116.267930] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3116.267934] device-mapper: ioctl: error adding target to table
[ 3206.748331] sd 1:0:1:0: alua: port group 02 state A preferred supports tolUsNA
[ 3274.560381] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3274.560389] device-mapper: ioctl: error adding target to table
[ 3274.577485] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3274.577493] device-mapper: ioctl: error adding target to table
[ 3274.594349] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3274.594357] device-mapper: ioctl: error adding target to table
[ 3274.611289] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3274.611297] device-mapper: ioctl: error adding target to table
[ 3359.510443] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3359.510451] device-mapper: ioctl: error adding target to table
[ 3359.526818] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3359.526821] device-mapper: ioctl: error adding target to table
[ 3359.543217] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3359.543221] device-mapper: ioctl: error adding target to table
[ 3359.559572] device-mapper: table: 253:0: multipath: unknown path selector type
[ 3359.559576] device-mapper: ioctl: error adding target to table

Comment 2 Lin Li 2020-11-23 12:44:47 UTC
This is a ppc64le power server. You can login 
ibm-p9z-20-lp15.fs.lab.eng.bos.redhat.com/redhat to check it.
Thanks!

Comment 3 Lin Li 2020-11-23 13:07:23 UTC
Hi Ben,
I also tested it on NETAPP x84_64 server. multipathd doesn't report error while setting path_selector to unknown path selector type on NETAPP x84_64 server too. But the output of "multipath -ll" is different. The output of "multipath -ll" is empty on NETAPP x84_64 server.
Do I need to open another bug independently?
Thanks in advance!

Comment 4 Ben Marzinski 2020-11-24 18:13:11 UTC
(In reply to Lin Li from comment #3)
> Hi Ben,
> I also tested it on NETAPP x84_64 server. multipathd doesn't report error
> while setting path_selector to unknown path selector type on NETAPP x84_64
> server too. But the output of "multipath -ll" is different. The output of
> "multipath -ll" is empty on NETAPP x84_64 server.
> Do I need to open another bug independently?
> Thanks in advance!

I don't think that there is a difference between different architectures here. If I had to guess, I would guess that the difference is that on the ppc64le machines, the
multipath device already existed before you changed the path_selector, while on the x86_64 machines, either the devices didn't exist, or you ran multipathd reconfigure after changing the configuration.  If that's not the case, let me know.

But there are two issues here. One is that multipath doesn't really give warnings for ANY invalid parameter value. It only warns if the parameter itself is invalid, or if the value can't be stored (for instance, a number that is too big to fit in the variable). This is more of a feature request, since nothing is really broken.  The other issue (which I will use this bug for) is that when I use a garbage value for the path_selector, multipathd can crash on startup or reconfigure.  It's fine for multipathd to use a default value when an invalid value is given. But it should never crash.

Comment 5 Ben Marzinski 2020-11-24 19:59:40 UTC
Actually, never mind the second paragraph of the last comment. The issue I was seeing with the crash is because I was accidentally running with some patches that I was reviewing for upstream. They were causing the crash in this instance. I can't reproduce it on the rhel-8 code. So we only need one bugzilla, for multipath not issuing warnings in general on invalid values. Sorry for the confusion.

Comment 6 Jonathan Earl Brassow 2021-01-06 16:17:06 UTC
stripping devel_ack due to ITR removal.

Comment 7 Ben Marzinski 2021-01-21 02:17:06 UTC
*** Bug 1918191 has been marked as a duplicate of this bug. ***

Comment 8 Ben Marzinski 2021-01-21 02:30:39 UTC
*** Bug 1918263 has been marked as a duplicate of this bug. ***

Comment 9 Ben Marzinski 2021-01-25 21:18:00 UTC
*** Bug 1919077 has been marked as a duplicate of this bug. ***

Comment 11 Ben Marzinski 2021-11-23 17:10:32 UTC
Multipath now has improved checking of multipath.conf values.  However, this error checking is still more limited than what you are testing for.  For numeric values with ranges, multipath.conf will now cap the values to the range. For values with keywords, multipath will now print a  message if the keyword is invalid. There are a couple of other checks.  However, for values that are arbitrary strings, multipath does not do any checking, since it doesn't know which values will be accepted. This depends on things like which kernel modules exist on the system. For instance, if the service-time path selector module exists, then

path_selector "service-time 0"

is valid. If that module doesn't exist, it isn't.

This amount of error checking is what is acceptable upstream.

Comment 15 Lin Li 2021-11-25 15:27:57 UTC
Verified on device-mapper-multipath-0.8.4-20.el8
[root@storageqe-05 ~]# rpm -qa | grep multipath
device-mapper-multipath-libs-0.8.4-20.el8.x86_64
device-mapper-multipath-0.8.4-20.el8.x86_64

1.[root@storageqe-05 ~]# man multipath.conf
verbosity        Default  verbosity.  Higher values increase the verbosity level. Valid
                        levels are between 0 and 6.<-----------

                        The default is: 2


[root@storageqe-05 ~]# cat /etc/multipath.conf
defaults {
	user_friendly_names yes
	find_multipaths yes
	enable_foreign "^$"
        #path_selector "service-time 1009"
        verbosity 7   <------------------------------------
}

[root@storageqe-05 ~]# multipath -ll
Nov 25 10:19:44 | /etc/multipath.conf line 16, value for verbosity too large, capping at 4  <--------------------------------------

Test result: For numeric values with ranges, multipath.conf will now cap the values to the range.


2.[root@storageqe-05 ~]# cat /etc/multipath.conf
defaults {
	user_friendly_names yes
	find_multipaths yes
	enable_foreign "^$"
        #path_selector "service-time 1009"
        #verbosity 7
        foo 2   <----------------------------
}

[root@storageqe-05 ~]# service multipathd reload
Redirecting to /bin/systemctl reload multipathd.service

[root@storageqe-05 ~]# multipath -ll
Nov 25 10:23:31 | /etc/multipath.conf line 17, invalid keyword in the defaults section: foo <-------------------------------

Test result:For values with keywords, multipath will now print a  message if the keyword is invalid.

Comment 17 errata-xmlrpc 2022-05-10 15:21:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (device-mapper-multipath bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2036