Bug 735292

Description of problem:
When udev rules are used to set /dev/mapper/mpathX permissions via RUN+="chown foo...", Multipath overwrites some or all of them. This is highly problematic for RHEL 5 systems running Oracle RAC 11 and Multipath, largely because partition 1 requires one set of permissions while partition 5 requires another. The logical solution appears to be the use of "udev" rules, however any uid/gid/mode setting in multipath.conf *might* override those set by udev rules. As well, because udev cannot match KERNEL=="mpath16", the USER/GROUP/MODE settings within udev are not available, so one must queue a chown/chmod via RUN+=. This queuing mechanism appears to race with multipath on system boot.

Version-Release number of selected component (if applicable):
2.6.18-194.11.3.el5
device-mapper-multipath-0.4.7-34.el5_5.5

How reproducible:
On Dell R910, I had this constantly. Assuming 40 /dev/mapper/mpathX volumes, each with a p1 and p5, a system boot will yield 38 disks with the udev ownership and 2 with multipath's applied (or default) ownership. The disks impacted change each time which causes me to suspect a race.

Steps to Reproduce:
1. Configure RHEL5 with 40 /dev/mapper/mpathX volumes. Follow the standard practice of blacklisting local devices and aliasing your devices. Each should have a primary partition 1 and a logical partition 5. 
2. Configure udev rules (40-multipath.rules) to apply permissions as follows:
After /dev/mpath/%c symlink created:
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p5", RUN+="/bin/chmod 0640 /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p5", RUN+="/bin/chown oracle:dba /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p1", RUN+="/bin/chmod 0660 /dev/mapper/%c"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*p1", RUN+="/bin/chown oracle:dba /dev/mapper/%c"
After kpartx:
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="mpath[0-9]*", RUN+="/bin/chown oracle:dba /dev/mapper/%c"

3. Edit /etc/multipath.conf and assign uid/gid other than oracle/dba to each alias.
4. Boot the system a few times and observe /dev/mapper with ls -l. After one or two reboots you should see a seemingly random pattern emerge where one or more of the disks gets the multipath permission while most retain the udev permission.
  
Additional info:

Being that this results from a combination of actions, I'm not certain the bug is in multipath. I seek discussion with you based on what I believe is a bug *somewhere* in this series of interactions, and also offer thoughts on remedy. Based on what I've read, this problem is pervasive to all Oracle RAC installations where RHEL 5 multipathing is used, so I believe it will provide value to many of your users to analyze and respond to this issue.

Workaround:
1) Use "udev" rules and uid/gid settings within /etc/multipath.conf to apply proper ownership to /dev/mapper/mpathX nodes. This suggests that applying permissions only via multipath.conf is acceptable. Our requirements call for varied permissions from p1 to p5. This renders multipath.conf unsuitable since it only supports one uid/gid/mode and then applies it to all partitions of that device.

Suggestions for remedy:
1) Add a setting to /etc/multipath.conf which disables the application of any permission. This would permit udev rules, even those relying on RUN+=chmod to set and maintain permissions for the /dev/mapper/mpathX files created.
2) Enable multipath.conf to specify different uid/gid/mode settings for partitions of multipath devices in /dev/mapper/mpathX. This would eliminate the need for udev permissions modification altogether.