RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 869253 - /dev/mapper/mpathX is block device not symbolic link
Summary: /dev/mapper/mpathX is block device not symbolic link
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: device-mapper-multipath
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Ben Marzinski
QA Contact: Gris Ge
URL:
Whiteboard:
Depends On: 864368 869254 872174
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-10-23 12:10 UTC by Peter Rajnoha
Modified: 2013-02-21 10:57 UTC (History)
14 users (show)

Fixed In Version: device-mapper-multipath-0.4.9-62.el6
Doc Type: Bug Fix
Doc Text:
Cause: multipathd and kpartx were not telling libdevmapper to skip doing device creation, and let udev do it. Consequence: Sometimes libdevmapper would create a block device in /dev/mapper, and sometimes udev would create a symlink in /dev/mapper Fix: multipathd and kpartx now tell libdevmapper to skip doing device creation Result: udev will always create a symlink in /dev/mapper as expected.
Clone Of: 864368
Environment:
Last Closed: 2013-02-21 10:57:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:0458 0 normal SHIPPED_LIVE device-mapper-multipath bug fix and enhancement update 2013-02-20 21:07:30 UTC

Description Peter Rajnoha 2012-10-23 12:10:57 UTC
Cloning this bug for 6.4 as this might cause problems with tools reading udev database (like LVM) - the nodes/symlinks should now all be processed by udev, direct node creation without udev should be used for debug only.

+++ This bug was initially created as a clone of Bug #864368 +++

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
# rpm -q device-mapper-multipath
device-mapper-multipath-0.4.9-29.fc18.x86_64
# uname -a
Linux storageqe-02-iscsi1.lab.bos.redhat.com 3.6.0-3.fc18.x86_64 #1 SMP Wed Oct 3 13:29:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
#ls -l /dev/mapper/
total 0
crw-------. 1 root root  10, 236 Oct  7 20:46 control
brw-rw----. 1 root disk 253,   0 Oct  7 20:46 mpathb
brw-------. 1 root root 253,   1 Oct  7 20:46 mpathbp1
brw-------. 1 root root 253,   2 Oct  7 20:46 mpathbp2
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 vg_storageqe--02--iscsi1-lv_root -> ../dm-4
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 vg_storageqe--02--iscsi1-lv_swap -> ../dm-3
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 VolGroup00-LogVol00 -> ../dm-5
lrwxrwxrwx. 1 root root        7 Oct  7 20:46 VolGroup00-LogVol01 -> ../dm-6

  
Actual results:


Expected results:


Additional info:

--- Additional comment from xiaoli on 2012-10-09 11:03:42 CEST ---

close it if it's by design.

--- Additional comment from prajnoha on 2012-10-09 11:44:00 CEST ---

If there are nodes instead of symlinks in /dev/mapper, it means that libdevmapper was used in "non-udev" mode and there was a fallback to original direct node creation in libdevmapper, bypassing the udev.

Mpath should work now with udev support/udev synchronisation enabled by default so the situation described here is a fallback in case there's something wrong with udev setup.

As for kpartx and mappings created for partitions defined on top of an mpath device, it has an extra "-s" parameter to enable udev synchronisation (kpartx is called within udev rules directly to automatically create these mappings).

Ben, I've noticed there's still "kpartx" call without "-s" in 62-multipath.rules: 

  RUN+="$env{MPATH_SBIN_PATH}/kpartx -a -p p $tempnode"

We should change that to use the "-s" as well... Otherwise the device is not visible from udev database accessible from libudev etc. and that one could be used as a primary source of information for the list of existing devices in the system - LVM uses that for example and if udev is bypassed, the device is not seen by LVM then (or any other subsystem using udev DB).

That's for kpartx, though I'm not quite sure now why the non-partition mpath mappings bypassed udev.

--- Additional comment from bmarzins on 2012-10-23 05:31:24 CEST ---

Is there any way to tell libdevmapper that I want udev to create the device, but I don't want to wait for that to get done? I'm really leery of having kpartx, which is called from udev, wait on udev.

--- Additional comment from prajnoha on 2012-10-23 13:37:22 CEST ---

(In reply to comment #3)
> Is there any way to tell libdevmapper that I want udev to create the device,
> but I don't want to wait for that to get done? 

Yup... However, I see another part of the problem now! The libdm part was not correctly passing flags that control udev supporting code in case udev_sync is not used! The DM_UDEV_DISABLE_LIBRARY_FALLBACK was not propagated internally while udev_sync is off, it should be fixed with this patch (I'll include that in next 6.4 build):

  http://git.fedorahosted.org/cgit/lvm2.git/commit/?id=a820a68619c10108666f8093545837233c954046

Another part of the fix is for kpartx that should use the above-mentioned flag to control libdm to not try fixing any nodes and symlinks (in case udev_sync is not used - actually, this is the preferred way as, otherwise, there's a race between udev and libdm with respect to node creation without udev_sync. So we need to gracefully let udev do the work and disable any further checks and fallbacks to direct node creation).

--- Additional comment from prajnoha on 2012-10-23 13:40:07 CEST ---

Created attachment 632040 [details]
kpartx patch to use DM_UDEV_DISABLE_LIBRARY_FLAG

Patch for kpartx attached... (we should probably make this a default setting in libdm)

--- Additional comment from prajnoha on 2012-10-23 14:00:17 CEST ---

The use of DM_UDEV_DISABLE_LIBRARY_FLAG will cause libdm to rely on udev completely to create nodes and symlinks, libdm itself will not try any checks and it won't do any corrections with this flag used.

Ben, it's up to you - you may use this flag conditionally (e.g. dmsetup uses "--verifyudev" to not use this flag and do any corrections if udev did not create the nodes/symlinks for some reason, but by default, it makes use of it). Though, the library fallback should be just for debug, I think - if udev works correctly, there's no reason to have missing nodes/symlinks even if completely relying on udev.

--- Additional comment from prajnoha on 2012-10-23 14:04:59 CEST ---

Created attachment 632052 [details]
complete patch to use DM_UDEV_DISABLE_LIBRARY_FALLBACK in mpath tools

...attaching complete patch for mpath tools

--- Additional comment from prajnoha on 2012-10-23 14:07:20 CEST ---

Created attachment 632055 [details]
complete patch to use DM_UDEV_DISABLE_LIBRARY_FALLBACK in mpath tools

Comment 1 Ben Marzinski 2012-10-23 21:37:44 UTC
To test this, you need to make sure that multipathd creates the device through libdevmapper before udev can.  You may simply be able to see this by running kpartx to create some partitions without using the "-s".  It will be a lot easier to hit will selinux completely disabled. The other way to hit it is to start up multipathd, and then add a lot of devices all at once. You can probably use scsi_debug.  You want to set up multipath devices with only one path, since
otherwise adding the extra paths to the tables will give udev a chance to catch up. Again, disabling selinux will make this a lot easier to hit.

You've reproduced it if you've got multipath creating devnodes instead of links in /dev/mapper/

Comment 2 Xiaowei Li 2012-10-24 03:14:24 UTC
steps to reproduce accordingly Ben's comments.

    1. # dd if=/dev/zero of=disk.img bs=1M count=100
    2. use "fdisk disk.img" to create a partition
    3. # kpartx -av disk01 
    add map loop0p1 (253:1): 0 202752 linear /dev/loop0 2048
    4. # ls /dev/mapper/ -l
    ===
    brw-r--r-- 1 root root 253,   1 Oct 23 23:12 loop0p1
    ===

    check if the loop0p1 is a node device.

    5. if use "kpartx -asv disk01" will create the symbolic link.

Comment 3 Ben Marzinski 2012-10-25 19:45:51 UTC
Patch ported.  multipathd and kpartx now use the DM_UDEV_DISABLE_LIBRARY_FLAG flag.

Comment 5 Xiaowei Li 2013-01-15 08:42:54 UTC
(In reply to comment #2)
> steps to reproduce accordingly Ben's comments.
> 
>     1. # dd if=/dev/zero of=disk.img bs=1M count=100
>     2. # losetup /dev/loop10 disk.img
>     3. # fdisk /dev/loop10 to create a partition
>     4. # kpartx -av disk.img
>     add map loop0p1 (253:1): 0 202752 linear /dev/loop0 2048
>     5. # ls /dev/mapper/ -l
>     ===
>     brw-r--r-- 1 root root 253,   1 Oct 23 23:12 loop0p1
>     ===
> 
>     check if the loop0p1 is a node device.
> 
>     5. if use "kpartx -asv disk01" will create the symbolic link.

This above steps are for RHEL7.0. updated them for RHEl 6.4

Comment 6 Gris Ge 2013-01-15 09:53:37 UTC
Test case created for this bug:
/kernel/storage/multipath/symbolic_link_mpathx_BZ_869253


In RHEL 6.3, there are some block file in /dev/mapper folder for a short while. After udev finished, all file changed to link.

In RHEL 6.4 (device-mapper-multipath-0.4.9-63.el6.x86_64) , no block file in /dev/mapper any more.

VERIFY.

Comment 8 errata-xmlrpc 2013-02-21 10:57:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0458.html


Note You need to log in before you can comment on or make changes to this bug.