Bug 675366

(In reply to comment #0)
> Description of problem:
> 
> Kernel reported the following messages while starting multipathd service even
> no 253:3 device present in /proc/partitions.
> 
> kernel: device-mapper: table: 253:3: multipath: error getting device
> kernel: device-mapper: ioctl: error adding target to table
> 
> Device major:minor always equal to the 253:<present max minor number in
> /proc/partitions + 1 > 

You're not reading the error properly (admittedly not the easiest to parse).  The DM device that is being created (253:3) is failing to get a device during table load -- as such the DM device is never created (that is why you never see 253:3 in /proc/partitions).

Though the error could be improved by sharing which device DM couldn't get,
e.g.:
 multipath: error getting device $NAME 
  or
 multipath: error getting device $MAJ:$MIN

> Additional info:
> 
> Workaround is to blacklist the local disk in multipath.conf 
> 
> Here is the local disk info 
> 
> # df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/vg_dhcp20934-lv_root
>                       3.9G  2.2G  1.5G  61% /
> tmpfs                 250M   19M  231M   8% /dev/shm
> /dev/sda1             485M   46M  414M  10% /boot
> 
> # pvs
>   PV         VG           Fmt  Attr PSize PFree
>   /dev/sda2  vg_dhcp20934 lvm2 a-   4.88g    0 
> 
> # grep -v ^# /etc/multipath.conf 
> 
> 
> defaults {
>  user_friendly_names yes
> }
> blacklist {
>  devnode "^sda$"
> }
> 
> # /etc/init.d/multipathd start
> Starting multipathd daemon:                                [  OK  ]
> 
> /var/log/messages 
> 
> multipathd: mpathb: load table [0 10444800 multipath 0 0 2 1 round-robin 0 1 1
> 8:16 1000 round-robin 0 1 1 8:32 1000]
> multipathd: mpathb: event checker started
> multipathd: path checkers start up
> multipathd: dm-2: add map (uevent)
> multipathd: dm-2: devmap already registered

AFAIK if a device is already open multipathd shouldn't attempt to add it to an mpath device.  So we need some clarification from Ben on this.

> multipathd: dm-2: add map (uevent)
> multipathd: dm-2: devmap already registered

These messages can be safely ignored.  They will be going away in 6.1.  Multipathd attempts to add new devices on change uevents.  Unfortunately, these change uevents can happen with stunning frequency, for things as mundane as doing IO to the device.  In 6.1 multipath doesn't print out any messages at the default verbosity when it gets extraneous change uevents.


These messages:

kernel: device-mapper: table: 253:3: multipath: error getting device
kernel: device-mapper: ioctl: error adding target to table
kernel: device-mapper: table: 253:3: multipath: error getting device
kernel: device-mapper: ioctl: error adding target to table

Are most likely due to multipathd attempting to create a multipath device on top of an already opened device.  This is usually because there are device that should be blacklisted which are not.

Running:

# multipath -v3

Will show you all of the device that multipath is trying to create (along with a lot of other debugging information). You can use this to see which device is not getting created.

You should see a line like:

Feb 10 03:45:21 | mpatha: set ACT_CREATE (map does not exist)
Feb 10 03:45:21 | mpatha: domap (0) failure for create/reload map

This tells you that mpatha failed to get created. Then you can look for lines like:

Feb 10 03:45:21 | sda: ownership set to mpatha

This tells you which devices belong to mpatha (in this case it is sda, which is actually my root device, and should not be multipathd)

Then you can look for a line like

Feb 10 03:45:21 | sda: uid = 1ATA     WDC WD800JD-75MSA3                        
   WD-WMAM9HH14567 (callout)

This tells you that the wwid of sda is
1ATA     WDC WD800JD-75MSA3                           WD-WMAM9HH14567

so you should add

wwid "1ATA     WDC WD800JD-75MSA3                           WD-WMAM9HH14567"

to the blacklists section of /etc/multipath.conf
For blacklisting individual devices, its always best to use the wwid, since it
is guaranteed to always be the same, while the devnode can change on reboot.

Ben, thanks for such a wonderful explanation . Yes, I missed the option to blacklist using wwid. 

- dominic