Bug 924924

Summary: multipathd does not config file defaults
Product: Red Hat Enterprise Linux 6 Reporter: Jon Magrini <jmagrini>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED ERRATA QA Contact: yanfu,wang <yanwang>
Severity: low Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, agk, bdonahue, bmarzins, dwysocha, h7sunny, heinzm, msnitzer, prajnoha, prockai, slevine, wduffee, yanwang, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: device-mapper-multipath-0.4.9-66.el6 Doc Type: Enhancement
Doc Text:
Feature: There is a new multipath.conf default parameter "replace_wwid_whitespace". If set to "yes", multipath will add "--replace-whitespace" to the default getuid_callout. This parameter defaults to "no". Reason: To make it easier for users to configure all of their devices to use wwids with the whitespace removed Result (if any): If this parameter is set, all dveices that use scsi_id get their wwid will now return wwids with the whitespace replaced by a single underscore.
Story Points: ---
Clone Of:
: 987144 (view as bug list) Environment:
Last Closed: 2013-11-21 07:48:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 987144    

Description Jon Magrini 2013-03-22 20:29:18 UTC
Description of problem:
multipathd does not apply config file defaults

Version-Release number of selected component (if applicable):
# rpm -qa *multipath*
device-mapper-multipath-libs-0.4.9-56.el6_3.1.x86_64
device-mapper-multipath-0.4.9-56.el6_3.1.x86_64

How reproducible:
very

Steps to Reproduce:
1. change the getuid_callout within the config file defaults 
defaults {
        getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
        user_friendly_names yes
}

2. reload multipathd
# mutlipathd -v3 -d
or: 
# multipathd -k'reconfigure'
3. changes are not applied
  
Actual results:

mpath-testlunA (360014380125989a100004000007f0000) dm-4 HP,HSV300
size=10G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 4:0:0:2  sdb  8:16   active ready running
| `- 4:0:3:2  sdaf 65:240 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 4:0:1:2  sdl  8:176  active ready running
  `- 4:0:2:2  sdv  65:80  active ready running

Mar 22 15:49:24 cs-rh6-4 multipathd: sdb: getuid = /lib/udev/scsi_id --whitelisted --device=/dev/%n (controller setting)

Mar 22 15:49:24 cs-rh6-4 multipathd: sdaf: getuid = /lib/udev/scsi_id --whitelisted --device=/dev/%n (controller setting)

I thought the above would just be applying the DEFAULT_GETUID from libmultipath/defaults.h.  However, if a device is blacklisted from multipath, the output shows as expected: 
---
 wwn-0x600508e00000000031e759188bad4807 -> ../../sdao

blacklist { 
        wwid "3600508e00000000031e759188bad4807"
}

Mar 22 15:49:24 cs-rh6-4 multipathd: sdao: getuid = /lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n (config file default)

Expected results:
Apply entries from defaults{} within multipath.conf

Comment 1 Ben Marzinski 2013-03-25 17:30:23 UTC
The configuration in the defaults section is only applied to devices if there if that parameter is not already configured in the devices or multipaths sections. The precedence order goes

multipaths > devices > defaults

If you run:

# multipathd show config

(you don't need to use -k"<command>" in RHEL6)

you will see the entire configuration, including the built-in defaults.

For your device, there is a built-in default

devices {
        ...
        device {
                vendor "(COMPAQ|HP)"
                product "HSV1[01]1|HSV2[01]0|HSV300|HSV4[05]0"
                path_grouping_policy group_by_prio
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "round-robin 0"
                path_checker tur
                features "0"
                hardware_handler "0"
                prio alua
                failback immediate
                rr_weight uniform
                no_path_retry 12
                rr_min_io 100
                rr_min_io_rq 1
        }
        ...
}

To get this change to work, you want the following multipath.conf

####
defaults {
	user_friendly_names yes
	hwtable_regex_match yes
}

devices {
	device {
		vendor "HP"
		product "HSV300"
		getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
	}
}
####

If you don't put "hwtable_regex_match" in the defaults section, you must match the vendor and product string exactly from built-in configuration to overwrite it, like so:

                vendor "(COMPAQ|HP)"
                product "HSV1[01]1|HSV2[01]0|HSV300|HSV4[05]0"

With "hwtable_regex_match", multipath will do the same regex matching that it does on your devices, to figure out which config you want to modify.  Either way works fine.

Then you can make multipathd reload the config with

# service multipathd reload

and check that it overwrote it correctly with

# multipathd show config

Does this fix it for you?

Comment 2 Jon Magrini 2013-04-11 13:38:04 UTC
The goal is to overwrite the devices{} without having to enter a devices{} stanza.  

Here is my multipath.conf: 

defaults {
        getuid_callout "/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n"
        user_friendly_names yes
        hwtable_regex_match yes
}

blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
}

I am attempting to add the --replace-whitespace via defaults{} for *any* storage enclosure without having to add a separate devices{} stanza for each.  Since a devices{} does not exist in multipath.conf, the defaults{} should apply.   

With the above config, you can see the defaults{} is not being applied though a devices{} does not exist.  So it looks by default multipathd is performing a regex_match and only applying the hwtable entry even if a devices{} is not added to config file.  

Apr 11 09:00:36 | sdai: getuid = /lib/udev/scsi_id --whitelisted --device=/dev/%n (controller setting)

Apr 11 09:00:36 | sdv: getuid = /lib/udev/scsi_id --whitelisted --device=/dev/%n (controller setting)

Comment 3 Ben Marzinski 2013-04-11 22:26:27 UTC
Unfortunately, there is no way to do this for devices that have a built-in config.  Every built-in config explicitly sets the getuid_callout, so that setting it in
the defaults section will have no effect on those devices, since the devices section has priority over the defaults section.

Comment 5 Ben Marzinski 2013-05-20 22:52:02 UTC
While there is no way to change how the multipath precedence order works, for getuid_callout we can probably fix things, as described in Bug #961829.

Comment 6 Ben Marzinski 2013-06-24 22:56:14 UTC
I actually fixed this issue differently.  Instead of removing all of the default getuid settings in the devices section, which still had the possibility of causing problems for existing users on upgrade, I added a new multipath.conf keyword.

If you add

replace_wwid_whitespace yes

In the defaults section of multipath.conf, then the default getuid callout for all devices that use it will be changed to include --replace-whitespace.  This still means that users only need to change one line to get all the devices to replace whitespaces, but it can't effect any existing customers unless they add that line.

Comment 7 Ben Marzinski 2013-06-24 23:05:10 UTC
*** Bug 961829 has been marked as a duplicate of this bug. ***

Comment 10 Steven J. Levine 2013-07-22 19:38:47 UTC
I will need to add this new keyword to the Multipath document, so I'm cloning this as a documentation bug.

Comment 11 yanfu,wang 2013-10-10 07:54:46 UTC
Set up a multipath device on top of an iscsi device served by tgtd. By default, these have uuids with spaces in them.
Reproduced like below mpathb shown.
[root@storageqe-17 ~]# multipath -ll
mpathb (1IET     00010001) dm-5 IET,VIRTUAL-DISK
size=500M features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 7:0:0:1 sde 8:64 active ready running
mpatha (360060160a32611005a800a3b568ce211) dm-0 DGC,RAID 10
size=50G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| |- 1:0:0:0 sda 8:0  active ready running
| `- 2:0:1:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
  |- 1:0:1:0 sdb 8:16 active ready running
  `- 2:0:0:0 sdc 8:32 active ready running

Verified on device-mapper-multipath-0.4.9-71.el6 with "replace_wwid_whitespace" set to "yes" in the defaults section of multipath.conf, the uuids that multipath gets will have their whitespace replaced by an "_". 
[root@storageqe-17 ~]# cat /etc/multipath.conf 
# multipath.conf written by anaconda

defaults {
	user_friendly_names yes
	replace_wwid_whitespace yes
}

[root@storageqe-17 ~]# multipath -l
mpathc (1IET_00010001) dm-11 IET,VIRTUAL-DISK
size=500M features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- 8:0:0:1 sdd 8:48 active undef running
...

Comment 13 errata-xmlrpc 2013-11-21 07:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1574.html