Bug 869253 - /dev/mapper/mpathX is block device not symbolic link
/dev/mapper/mpathX is block device not symbolic link
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity medium
: rc
: ---
Assigned To: Ben Marzinski
Gris Ge
:
Depends On: 864368 869254 872174
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-23 08:10 EDT by Peter Rajnoha
Modified: 2013-02-21 05:57 EST (History)
14 users (show)

See Also:
Fixed In Version: device-mapper-multipath-0.4.9-62.el6
Doc Type: Bug Fix
Doc Text:
Cause: multipathd and kpartx were not telling libdevmapper to skip doing device creation, and let udev do it. Consequence: Sometimes libdevmapper would create a block device in /dev/mapper, and sometimes udev would create a symlink in /dev/mapper Fix: multipathd and kpartx now tell libdevmapper to skip doing device creation Result: udev will always create a symlink in /dev/mapper as expected.
Story Points: ---
Clone Of: 864368
Environment:
Last Closed: 2013-02-21 05:57:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Peter Rajnoha 2012-10-23 08:10:57 EDT
Cloning this bug for 6.4 as this might cause problems with tools reading udev database (like LVM) - the nodes/symlinks should now all be processed by udev, direct node creation without udev should be used for debug only.

+++ This bug was initially created as a clone of Bug #864368 +++

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
# rpm -q device-mapper-multipath
device-mapper-multipath-0.4.9-29.fc18.x86_64
# uname -a
Linux storageqe-02-iscsi1.lab.bos.redhat.com 3.6.0-3.fc18.x86_64 #1 SMP Wed Oct 3 13:29:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
#ls -l /dev/mapper/
total 0
crw-------. 1 root root  10, 236 Oct  7 20:46 control
brw-rw----. 1 root disk 253,   0 Oct  7 20:46 mpathb
brw-------. 1 root root 253,   1 Oct  7 20:46 mpathbp1
brw-------. 1 root root 253,   2 Oct  7 20:46 mpathbp2
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 vg_storageqe--02--iscsi1-lv_root -> ../dm-4
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 vg_storageqe--02--iscsi1-lv_swap -> ../dm-3
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 VolGroup00-LogVol00 -> ../dm-5
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 VolGroup00-LogVol01 -> ../dm-6

  
Actual results:


Expected results:


Additional info:

--- Additional comment from xiaoli@redhat.com on 2012-10-09 11:03:42 CEST ---

close it if it's by design.

--- Additional comment from prajnoha@redhat.com on 2012-10-09 11:44:00 CEST ---

If there are nodes instead of symlinks in /dev/mapper, it means that libdevmapper was used in "non-udev" mode and there was a fallback to original direct node creation in libdevmapper, bypassing the udev.

Mpath should work now with udev support/udev synchronisation enabled by default so the situation described here is a fallback in case there's something wrong with udev setup.

As for kpartx and mappings created for partitions defined on top of an mpath device, it has an extra "-s" parameter to enable udev synchronisation (kpartx is called within udev rules directly to automatically create these mappings).

Ben, I've noticed there's still "kpartx" call without "-s" in 62-multipath.rules: 

  RUN+="$env{MPATH_SBIN_PATH}/kpartx -a -p p $tempnode"

We should change that to use the "-s" as well... Otherwise the device is not visible from udev database accessible from libudev etc. and that one could be used as a primary source of information for the list of existing devices in the system - LVM uses that for example and if udev is bypassed, the device is not seen by LVM then (or any other subsystem using udev DB).

That's for kpartx, though I'm not quite sure now why the non-partition mpath mappings bypassed udev.

--- Additional comment from bmarzins@redhat.com on 2012-10-23 05:31:24 CEST ---

Is there any way to tell libdevmapper that I want udev to create the device, but I don't want to wait for that to get done? I'm really leery of having kpartx, which is called from udev, wait on udev.

--- Additional comment from prajnoha@redhat.com on 2012-10-23 13:37:22 CEST ---

(In reply to comment #3)
> Is there any way to tell libdevmapper that I want udev to create the device,
> but I don't want to wait for that to get done? 

Yup... However, I see another part of the problem now! The libdm part was not correctly passing flags that control udev supporting code in case udev_sync is not used! The DM_UDEV_DISABLE_LIBRARY_FALLBACK was not propagated internally while udev_sync is off, it should be fixed with this patch (I'll include that in next 6.4 build):

  http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=a820a68619c10108666f8093545837233c954046

Another part of the fix is for kpartx that should use the above-mentioned flag to control libdm to not try fixing any nodes and symlinks (in case udev_sync is not used - actually, this is the preferred way as, otherwise, there's a race between udev and libdm with respect to node creation without udev_sync. So we need to gracefully let udev do the work and disable any further checks and fallbacks to direct node creation).

--- Additional comment from prajnoha@redhat.com on 2012-10-23 13:40:07 CEST ---

Created attachment 632040 [details]
kpartx patch to use DM_UDEV_DISABLE_LIBRARY_FLAG

Patch for kpartx attached... (we should probably make this a default setting in libdm)

--- Additional comment from prajnoha@redhat.com on 2012-10-23 14:00:17 CEST ---

The use of DM_UDEV_DISABLE_LIBRARY_FLAG will cause libdm to rely on udev completely to create nodes and symlinks, libdm itself will not try any checks and it won't do any corrections with this flag used.

Ben, it's up to you - you may use this flag conditionally (e.g. dmsetup uses "--verifyudev" to not use this flag and do any corrections if udev did not create the nodes/symlinks for some reason, but by default, it makes use of it). Though, the library fallback should be just for debug, I think - if udev works correctly, there's no reason to have missing nodes/symlinks even if completely relying on udev.

--- Additional comment from prajnoha@redhat.com on 2012-10-23 14:04:59 CEST ---

Created attachment 632052 [details]
complete patch to use DM_UDEV_DISABLE_LIBRARY_FALLBACK in mpath tools

...attaching complete patch for mpath tools

--- Additional comment from prajnoha@redhat.com on 2012-10-23 14:07:20 CEST ---

Created attachment 632055 [details]
complete patch to use DM_UDEV_DISABLE_LIBRARY_FALLBACK in mpath tools
Comment 1 Ben Marzinski 2012-10-23 17:37:44 EDT
To test this, you need to make sure that multipathd creates the device through libdevmapper before udev can.  You may simply be able to see this by running kpartx to create some partitions without using the "-s".  It will be a lot easier to hit will selinux completely disabled. The other way to hit it is to start up multipathd, and then add a lot of devices all at once. You can probably use scsi_debug.  You want to set up multipath devices with only one path, since
otherwise adding the extra paths to the tables will give udev a chance to catch up. Again, disabling selinux will make this a lot easier to hit.

You've reproduced it if you've got multipath creating devnodes instead of links in /dev/mapper/
Comment 2 Xiaowei Li 2012-10-23 23:14:24 EDT
steps to reproduce accordingly Ben's comments.

    1. # dd if=/dev/zero of=disk.img bs=1M count=100
    2. use "fdisk disk.img" to create a partition
    3. # kpartx -av disk01 
    add map loop0p1 (253:1): 0 202752 linear /dev/loop0 2048
    4. # ls /dev/mapper/ -l
    ===
    brw-r--r-- 1 root root 253,   1 Oct 23 23:12 loop0p1
    ===

    check if the loop0p1 is a node device.

    5. if use "kpartx -asv disk01" will create the symbolic link.
Comment 3 Ben Marzinski 2012-10-25 15:45:51 EDT
Patch ported.  multipathd and kpartx now use the DM_UDEV_DISABLE_LIBRARY_FLAG flag.
Comment 5 Xiaowei Li 2013-01-15 03:42:54 EST
(In reply to comment #2)
> steps to reproduce accordingly Ben's comments.
> 
>     1. # dd if=/dev/zero of=disk.img bs=1M count=100
>     2. # losetup /dev/loop10 disk.img
>     3. # fdisk /dev/loop10 to create a partition
>     4. # kpartx -av disk.img
>     add map loop0p1 (253:1): 0 202752 linear /dev/loop0 2048
>     5. # ls /dev/mapper/ -l
>     ===
>     brw-r--r-- 1 root root 253,   1 Oct 23 23:12 loop0p1
>     ===
> 
>     check if the loop0p1 is a node device.
> 
>     5. if use "kpartx -asv disk01" will create the symbolic link.

This above steps are for RHEL7.0. updated them for RHEl 6.4
Comment 6 Gris Ge 2013-01-15 04:53:37 EST
Test case created for this bug:
/kernel/storage/multipath/symbolic_link_mpathx_BZ_869253


In RHEL 6.3, there are some block file in /dev/mapper folder for a short while. After udev finished, all file changed to link.

In RHEL 6.4 (device-mapper-multipath-0.4.9-63.el6.x86_64) , no block file in /dev/mapper any more.

VERIFY.
Comment 8 errata-xmlrpc 2013-02-21 05:57:42 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0458.html

Note You need to log in before you can comment on or make changes to this bug.