Bug 174084

Summary: multipathd incorrectly handles devnode blacklist on runtime
Product: Red Hat Enterprise Linux 4 Reporter: Kari Hautio <kari.hautio>
Component: device-mapper-multipathAssignee: Alasdair Kergon <agk>
Status: CLOSED ERRATA QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 4.0CC: agk, bmarzins, christophe.varoqui, dmo, egoggin, jbrassow, junichi.nomura, kanderso, kueda, lmb, poelstra, rkenna, tranlan
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: RHEA-2006-0084 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-03-07 18:41:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 168429, 171044    

Description Kari Hautio 2005-11-24 11:31:03 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7

Description of problem:
Multipathd seems to ignore blacklisting if device is created after the multipathd has been started.

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.5-6.0.RHEL4

How reproducible:
Always

Steps to Reproduce:
1. Ensure that md devices are blacklisted in config
ie.
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
}
2. start at least one raid array eg. /dev/md0
3. start multipathd
4. Check that the already started array is ignored eg. 
fuser /dev/md0
5. start up one more raid eg. /dev/md10


Actual Results:  1) According to syslog multipathd starts to monitor /dev/md10
Nov 24 13:04:12 src@CLA-1 kernel: raid1: raid set md10 active with 2 out of 2 mirrors
Nov 24 13:04:12 src@CLA-1 kernel: md: syncing RAID array md10
Nov 24 13:04:12 src@CLA-1 multipathd: error calling out /rbin/fc_getloopid.sh md10
Nov 24 13:04:12 src@CLA-1 multipathd: md10: path checker registered
Nov 24 13:04:21 src@CLA-1 kernel: md: md10: sync done.
Nov 24 13:06:41 src@CLA-1 kernel: md: md10 still in use.
Nov 24 13:07:03 src@CLA-1 kernel: md: md10 stopped.
2) /dev/md10 is opened (and kept open by multipathd)
# fuser /dev/md10
/dev/md10:            1583
[root@CLA-1(EMC-R_FSPR4_1.9-1) /root]
# ps ax | grep 1583
 1583 ?        SLl    0:00 /sbin/multipathd
 3237 pts/0    S+     0:00 grep 1583
[root@CLA-1(EMC-R_FSPR4_1.9-1) /root]


Expected Results:  md10 should have been ignored by multipathd (according to blacklist)


Additional info:

This behaviour makes it problematic to stop the array since it's kept open by a process (=multipathd), the workaround is to restart multipathd after creating new arrays however this causes problems in recovering the paths in some corner cases.

Comment 1 Bastien Nocera 2005-11-24 15:21:10 UTC
Could you run multipathd with the "-v 10" option?
We should see a message like:
/dev/md10 blacklisted
in the logs.

Comment 2 Kari Hautio 2005-11-24 16:00:09 UTC
Here's a new run, somewhow it seems to renember /dev/md10 now so I'm creating
md11 while it's running.

Nov 24 17:56:11 src@CLA-0 multipathd: --------start up--------
Nov 24 17:56:11 src@CLA-0 multipathd: read /etc/multipath.conf
Nov 24 17:56:11 src@CLA-0 multipathd: dm-0 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-10 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-11 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-12 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-13 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-14 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-15 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-16 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-17 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-18 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-19 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-1 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-20 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-21 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-22 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-23 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-24 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-25 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-26 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-27 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-28 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-29 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-2 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-30 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-31 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-32 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-33 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-34 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-35 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-36 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-3 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-4 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-5 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-6 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-7 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-8 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: dm-9 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop0 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop1 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop2 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop3 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop4 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop5 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop6 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: loop7 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: md0 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: md10 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: md1 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: md9 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram0 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram10 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram11 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram12 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram13 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram14 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram15 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram1 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram2 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram3 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram4 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram5 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram6 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram7 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram8 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: ram9 blacklisted
Nov 24 17:56:11 src@CLA-0 multipathd: path sda not found in pathvec
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sda =====
Nov 24 17:56:11 src@CLA-0 multipathd: bus = 1
Nov 24 17:56:11 src@CLA-0 multipathd: dev_t = 8:0
Nov 24 17:56:11 src@CLA-0 multipathd: size = 286749610
Nov 24 17:56:11 src@CLA-0 multipathd: vendor = IBM
Nov 24 17:56:11 src@CLA-0 multipathd: product = IC35L146F2DY10-0
Nov 24 17:56:11 src@CLA-0 multipathd: rev = F56A
Nov 24 17:56:11 src@CLA-0 multipathd: h:b:t:l = 0:0:0:0
Nov 24 17:56:11 src@CLA-0 multipathd: tgt_node_name = 0x500507671903f255
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getuid = /rbin/fc_getloopid.sh %n
(internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (callout)
Nov 24 17:56:11 src@CLA-0 multipathd: path sdb not found in pathvec
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdb =====
Nov 24 17:56:11 src@CLA-0 multipathd: bus = 1
Nov 24 17:56:11 src@CLA-0 multipathd: dev_t = 8:16
Nov 24 17:56:11 src@CLA-0 multipathd: size = 286749610
Nov 24 17:56:11 src@CLA-0 multipathd: vendor = IBM
Nov 24 17:56:11 src@CLA-0 multipathd: product = IC35L146F2DY10-0
Nov 24 17:56:11 src@CLA-0 multipathd: rev = F56A
Nov 24 17:56:11 src@CLA-0 multipathd: h:b:t:l = 0:0:1:0
Nov 24 17:56:11 src@CLA-0 multipathd: tgt_node_name = 0x500507671903fd43
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getuid = /rbin/fc_getloopid.sh %n
(internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (callout)
Nov 24 17:56:11 src@CLA-0 multipathd: path sdc not found in pathvec
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdc =====
Nov 24 17:56:11 src@CLA-0 multipathd: bus = 1
Nov 24 17:56:11 src@CLA-0 multipathd: dev_t = 8:32
Nov 24 17:56:11 src@CLA-0 multipathd: size = 286749610
Nov 24 17:56:11 src@CLA-0 multipathd: vendor = IBM
Nov 24 17:56:11 src@CLA-0 multipathd: product = IC35L146F2DY10-0
Nov 24 17:56:11 src@CLA-0 multipathd: rev = F56A
Nov 24 17:56:11 src@CLA-0 multipathd: h:b:t:l = 1:0:0:0
Nov 24 17:56:11 src@CLA-0 multipathd: tgt_node_name = 0x500507671903f255
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getuid = /rbin/fc_getloopid.sh %n
(internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (callout)
Nov 24 17:56:11 src@CLA-0 multipathd: path sdd not found in pathvec
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdd =====
Nov 24 17:56:11 src@CLA-0 multipathd: bus = 1
Nov 24 17:56:11 src@CLA-0 multipathd: dev_t = 8:48
Nov 24 17:56:11 src@CLA-0 multipathd: size = 286749610
Nov 24 17:56:11 src@CLA-0 multipathd: vendor = IBM
Nov 24 17:56:11 src@CLA-0 multipathd: product = IC35L146F2DY10-0
Nov 24 17:56:11 src@CLA-0 multipathd: rev = F56A
Nov 24 17:56:11 src@CLA-0 multipathd: h:b:t:l = 1:0:1:0
Nov 24 17:56:11 src@CLA-0 multipathd: tgt_node_name = 0x500507671903fd43
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getuid = /rbin/fc_getloopid.sh %n
(internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (callout)
Nov 24 17:56:11 src@CLA-0 multipathd: pgfailback = -2 (config file default)
Nov 24 17:56:11 src@CLA-0 multipathd: HDF_62: event checker started
Nov 24 17:56:11 src@CLA-0 multipathd: pgfailback = -2 (config file default)
Nov 24 17:56:11 src@CLA-0 multipathd: HDF_63: event checker started
Nov 24 17:56:11 src@CLA-0 multipathd: path checkers start up
Nov 24 17:56:11 src@CLA-0 multipathd: 8:0: readsector0 checker reports path is up
Nov 24 17:56:11 src@CLA-0 multipathd: 8:0: reinstated
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sda =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdc =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: 8:16: readsector0 checker reports path is up
Nov 24 17:56:11 src@CLA-0 multipathd: 8:16: reinstated
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdb =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdd =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: 8:32: readsector0 checker reports path is up
Nov 24 17:56:11 src@CLA-0 multipathd: 8:32: reinstated
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sda =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdc =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 62 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: 8:48: readsector0 checker reports path is up
Nov 24 17:56:11 src@CLA-0 multipathd: 8:48: reinstated
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdb =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (cache)
Nov 24 17:56:11 src@CLA-0 multipathd: ===== path sdd =====
Nov 24 17:56:11 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: getprio = /rbin/true (internal default)
Nov 24 17:56:11 src@CLA-0 multipathd: prio = 0
Nov 24 17:56:11 src@CLA-0 multipathd: uid = 63 (cache)
Nov 24 17:56:12 src@CLA-0 multipathd: multipathd startup succeeded
Nov 24 17:57:01 src@CLA-0 kernel: md: md11 stopped.
Nov 24 17:57:01 src@CLA-0 kernel: md: bind<loop1>
Nov 24 17:57:01 src@CLA-0 kernel: md: bind<loop0>
Nov 24 17:57:01 src@CLA-0 kernel: raid1: raid set md11 active with 2 out of 2
mirrors
Nov 24 17:57:01 src@CLA-0 multipathd: path md11 not found in pathvec
Nov 24 17:57:01 src@CLA-0 multipathd: ===== path md11 =====
Nov 24 17:57:01 src@CLA-0 multipathd: bus = 0
Nov 24 17:57:01 src@CLA-0 multipathd: dev_t = 9:11
Nov 24 17:57:01 src@CLA-0 multipathd: size = 19840
Nov 24 17:57:01 src@CLA-0 multipathd: path checker = readsector0 (internal default)
Nov 24 17:57:01 src@CLA-0 multipathd: getuid = /rbin/fc_getloopid.sh %n
(internal default)
Nov 24 17:57:01 src@CLA-0 multipathd: error calling out /rbin/fc_getloopid.sh md11
Nov 24 17:57:01 src@CLA-0 multipathd: uid =  (callout)
Nov 24 17:57:01 src@CLA-0 multipathd: md11: path checker registered


Comment 5 Ben Marzinski 2005-11-28 20:50:47 UTC
I can recreate this bug with the U2 code. It is already fixed in the U3 code. I
will have a U3 device-mapper-multipath package available shortly.

Comment 9 Ben Marzinski 2005-12-22 19:33:59 UTC
The kernel doesn't handle the blacklisting of devices. It simply works with the
devices that the userspace tools supply.  This problem was in userspace, and the
fix did not involve any kernel code.  The fix is in the U3
device-mapper-multipath package that is being tested by QA.

Comment 16 Red Hat Bugzilla 2006-03-07 18:41:49 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2006-0084.html