Bug 735292 - Multipath overwrites permissions set via udev RUN+=
Summary: Multipath overwrites permissions set via udev RUN+=
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath
Version: 5.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Ben Marzinski
QA Contact: Storage QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-09-02 07:50 UTC by Garrett Ellis
Modified: 2014-02-06 23:03 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-06 23:03:29 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Garrett Ellis 2011-09-02 07:50:24 UTC
Description of problem:
When udev rules are used to set /dev/mapper/mpathX permissions via RUN+="chown foo...", Multipath overwrites some or all of them. This is highly problematic for RHEL 5 systems running Oracle RAC 11 and Multipath, largely because partition 1 requires one set of permissions while partition 5 requires another. The logical solution appears to be the use of "udev" rules, however any uid/gid/mode setting in multipath.conf *might* override those set by udev rules. As well, because udev cannot match KERNEL=="mpath16", the USER/GROUP/MODE settings within udev are not available, so one must queue a chown/chmod via RUN+=. This queuing mechanism appears to race with multipath on system boot.

Version-Release number of selected component (if applicable):
2.6.18-194.11.3.el5
device-mapper-multipath-0.4.7-34.el5_5.5

How reproducible:
On Dell R910, I had this constantly. Assuming 40 /dev/mapper/mpathX volumes, each with a p1 and p5, a system boot will yield 38 disks with the udev ownership and 2 with multipath's applied (or default) ownership. The disks impacted change each time which causes me to suspect a race.

Steps to Reproduce:
1. Configure RHEL5 with 40 /dev/mapper/mpathX volumes. Follow the standard practice of blacklisting local devices and aliasing your devices. Each should have a primary partition 1 and a logical partition 5. 
2. Configure udev rules (40-multipath.rules) to apply permissions as follows:
After /dev/mpath/%c symlink created:
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p5", RUN+="/bin/chmod 0640 /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p5", RUN+="/bin/chown oracle:dba /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p1", RUN+="/bin/chmod 0660 /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p1", RUN+="/bin/chown oracle:dba /dev/mapper/%c"
After kpartx:
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*", RUN+="/bin/chown oracle:dba /dev/mapper/%c"

3. Edit /etc/multipath.conf and assign uid/gid other than oracle/dba to each alias.
4. Boot the system a few times and observe /dev/mapper with ls -l. After one or two reboots you should see a seemingly random pattern emerge where one or more of the disks gets the multipath permission while most retain the udev permission.
  
Additional info:

Being that this results from a combination of actions, I'm not certain the bug is in multipath. I seek discussion with you based on what I believe is a bug *somewhere* in this series of interactions, and also offer thoughts on remedy. Based on what I've read, this problem is pervasive to all Oracle RAC installations where RHEL 5 multipathing is used, so I believe it will provide value to many of your users to analyze and respond to this issue.

Workaround:
1) Use "udev" rules and uid/gid settings within /etc/multipath.conf to apply proper ownership to /dev/mapper/mpathX nodes. This suggests that applying permissions only via multipath.conf is acceptable. Our requirements call for varied permissions from p1 to p5. This renders multipath.conf unsuitable since it only supports one uid/gid/mode and then applies it to all partitions of that device.

Suggestions for remedy:
1) Add a setting to /etc/multipath.conf which disables the application of any permission. This would permit udev rules, even those relying on RUN+=chmod to set and maintain permissions for the /dev/mapper/mpathX files created.
2) Enable multipath.conf to specify different uid/gid/mode settings for partitions of multipath devices in /dev/mapper/mpathX. This would eliminate the need for udev permissions modification altogether.

Comment 1 RHEL Product and Program Management 2014-01-29 10:37:37 UTC
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.

Comment 2 Ben Marzinski 2014-02-06 23:03:29 UTC
(In reply to Garrett Ellis from comment #0)
> Suggestions for remedy:
> 1) Add a setting to /etc/multipath.conf which disables the application of
> any permission. This would permit udev rules, even those relying on
> RUN+=chmod to set and maintain permissions for the /dev/mapper/mpathX files
> created.
> 2) Enable multipath.conf to specify different uid/gid/mode settings for
> partitions of multipath devices in /dev/mapper/mpathX. This would eliminate
> the need for udev permissions modification altogether.

kpartx creates the partitions and it doesn't use a configuration file.  Making kpartx use multipath.conf is non-trivial, and this issue is already solved in RHEL6, by udev having total control over creating the device nodes.

Since device-mapper and not udev creates the device nodes in RHEL5, I'm not sure that a race is avoidable. The only tested way to set permissions in RHEL5 is to use multipath.conf, which like you mentioned, doesn't work with kpartx.

I'm willing to troubleshoot workarounds, but there's no way to make device-mapper device creation go through udev in RHEL5, and adding a configuration file for kpartx it a significant amount of development for this late in RHEL5's lifetime.

If you're still experiencing this issue, let me know, and I'll see if something can be changed in the udev rules files to deal with it.


Note You need to log in before you can comment on or make changes to this bug.