Bug 409741

This is most likely an issue with the initrd.  If you multiphath paths are
loaded in the initrd, any changes that were made to the multipath configuration
file will not be reflected in the devices after reboot until the initrd is
refreshed.

To solve this, set up the devices like you would like them, and then remake the
initrd with mkinitrd.

I tested this and still have the same issue. I also was able to replicate this 
problem on another system running kernel 2.6.18-8.el5

device-mapper-1.02.20-1.el5
device-mapper-multipath-0.4.7-12.el5
device-mapper-1.02.20-1.el5

This appears to have started after I upgraded to the 0.4.7-12 multipath tools.

Do I need to build the initrd without multipath modules maybe?

I can't reproduce this.  But, this could happen if /etc/ was on a sperate file
system from /, and you started your multipath devices before you mounted /etc/.
Do you have /etc/ on a seperate file system?

Otherwise, can you attach the following files:

/etc/multipath.conf
/etc/sysconfig/mkinitrd/multipath (if you have it)
/etc/redhat-release
/etc/fstab

Created attachment 290582 [details]
fstab

Sorry for the slow reply, I was on vacation over Christmas. 

/etc is not a seperate filesystem so this isn't the problem. However during 
boot I do get the following error which may be having some effect.

"Cannot make directory [/var/lib] : Read-only file system

I get this message many times but then the system boots normally. Once 
booted /var/lib is there and permissions are set to root:root 755. The actual 
bindings file is set to 600. This is a new build of RHEL5 and I have always 
seen this message. I have always assumed this was a timing issue at boot and 
up until now have seen no issues caused by this. This may not be causing the 
multipath issue but since the bindings file is located in /var/lib maybe it is.

Created attachment 290583 [details]
multipath

Created attachment 290584 [details]
multipath.conf

Created attachment 290585 [details]
redhat-release

I'm pretty sure that I understand what your problem is now.  Looking at your
/etc/sysconfig/mkinitrd/multipath file from comment #7, multipath is probably
not being run by your initrd. However, it is being run before /var is mounted or
root is remounted read/write.  Since multipath cannot read the bindings file or
create one, it simply gives up on the user friendly names.  I've never seen this
exact error before, but I've seen similar problems.  There is already a fix for
this that will be in RHEL 5.2. You will be able to change your bindings file
location in /etc/multipath.conf

In the mean time, there are some workarounds, but none of them are great.
Probably the easiest one is to copy /var/lib/multipath/bindings to a temporary
location, unmount the device that is currently mounted on /var, make a
/var/lib/multipath/bindings file on the root filesystem, and copy you regular
bindings file to that.  That way, before /var is mounted, the machine will still
see the same bindings file.

I don't understand why upgrading the device-mapper-multipath package would have
anything to do with this, or why you didn't see this problem before, if you've
were seeing the same error message before. But regardless, I'm pretty sure from
the message in Comment #6, that this is your problem, and in will be fixed in
RHEL 5.2

*** This bug has been marked as a duplicate of 357331 ***

This happens if you have / and /var on separate partitions. When you boot your
system with no disks over FC/iSCSI attached everything goes fine. When you
attache  those disks and do scsi rescan and after that execute 'multipath -v2'
the /var/lib/multipath/bindings is being created considering you have previously
edited /etc/multipath.conf and added 'user_friendly_names yes' option. After you
reboot multipath routine from rc.sysinit comes after / partition was mounted and
just before any other partition has been mounted. Because there is no /var/lib
(hence no /var/lib/multipath/bindings) at this point, you get multipath devices
being made with their wwid names instead of friendly ones.

There are two more or less solid solutions to this problem. One is to make an
alias in /etc/multipath.conf for every single multipath wwid that you have
attached. And second is to move /var/lib/multipath/bindings to let say
/etc/multipath.bindings (my choice) and make a soft link
/var/lib/multipath/bindings that point to it. You then have to boot into single
user mode executing 'init 1', unmount /var (most probably you will have to
unmount /var/lib/nfs/rpc_pipefs before), go into empty /var, make lib/multipath/
subdirectories and also make there soft link /var/lib/multipath/bindings that
point to /etc/multipath.bindings.

These two solutions are only valid and show results if multipath routine isn't
integrated into your initramfs image. If you don't have / partition on multipath
device, you don't need multipath routine in your initramfs.