Red Hat Bugzilla – Bug 241189
[NetApp-S 5.1 bug] multipath unable to read the user_friendly_names bindings file during boot
Last modified: 2010-01-11 21:37:44 EST
Description of problem:
Automounting of dm-mp devices using the mpath softlink entries present in the
/dev/mpath/ directory in the fstab fails on a RHEL5 host.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Configure dm-mp devices on a RHEL5 host with "user_friendly_names" set to
"yes" in the multipath.conf file. This would put mpath soft links in the
/dev/mpath/ directory which point to the actual dm-mp block devices.
2. Now add a /dev/mpath/ entry to the fstab file to enable automounting of the
dm-mp device as shown below:
/dev/mpath/mpath1 /mnt/tmp ext3 defaults 0 0
3. Run a mount command on the above mpath entry which is successful i.e. the
corresponding dm-mp device is mounted at /mnt/tmp/.
4. Now reboot the host and verify whether the dm-mp device is still mounted on
The automount of the above dm-mp device fails during bootup.
The above automount should have succeeded.
1. Automounting using /dev/mapper/ entries is still successful. But the same
is not true with /dev/mpath/ entries as described above.
2. This is not seen on RHEL4 hosts. So this does look like a regression on RHEL5.
3. Apparently setting user_friendly_names to "yes" in the multipath.conf file on
a RHEL5 host causes issues as is evident from the following message thrown on
the console during bootup:
"Cannot open bindings file [/var/lib/multipath/bindings]: Read-only filesystem"
So no mpath softlinks are available during bootup which would obviously cause
the automount to fail.
This is probably do to changes in how multipath interacts with udev between
RHEL4 and RHEL5. I'll take a look into it. However, there really is no reason to
use the /dev/mpath path over the /dev/mapper one. It is useful to have your
multipath devices all in one directory, but the /dev/mapper names are just as
unique and persistent, and they are guaranteed to exist as soon as device mapper
has created the device, which is not true of the /dev/mpath/* names or the
/dev/dm-* names (which are created later by udev)
I've fixed the cause of the "Cannot open bindings file
[/var/lib/multipath/bindings]: Read-only filesystem" message, so if that was the
only thing keeping you from using the /dev/mpath symlinks, it should work know.
However, it is still better to use the /dev/mapper names during bootup, since
you can't guarantee that udev will always have the symlink created in time, for
I've tried automounting using the /dev/mpath/ symlinks on RHEL 5.1 Beta1 and it
works fine. But as recommended, we would be using /dev/mapper/ entries alone.
A fix for this issue has been included in the packages contained in the beta
(RHN channel) or most recent snapshot (partners.redhat.com) for RHEL5.1. Please
verify that your issue is fixed.
After you (Red Hat Partner) have verified that this issue has been addressed,
please perform the following:
1) Change the *status* of this bug to VERIFIED.
2) Add *keyword* of PartnerVerified (leaving the existing keywords unmodified)
If this issue is not fixed, please add a comment describing the most recent
symptoms of the problem you are having and change the status of the bug to ASSIGNED.
I have verifed the fix on RHEL 5.1 Beta1. Making the appropriate changes to this
I tried verifying the same on the latest RHEL 5.1 Snapshot3 x86_64 kernel. But
unfortunately there seems to be a problem here. During reboot, the filesystem
fsck of the corresponding mpath entry fails with the following error:
fsck.ext3: No such file or directory while trying to open /dev/mpath/mpath0"
One is then dropped to a shell for repair. At this point, I can see the
/dev/mpath/mpath0 entry pointing to a corresponding dm device, and a manual
mount of this device is successful. This proves that the device is available.
Strangely, I can't edit any files now as the root partition is mounted read-only.
The only way now to successfully reboot is:
1) Remount the root partition in read-write.
2) Edit the /etc/fstab file by removing the corresponding mpath entry in it.
3) Then run a reboot.
Reopening this bugzilla.
Created attachment 173621 [details]
Console log displaying fsck error during reboot.
Note: Automount of dm-mp devices work fine while using /dev/mapper/ entries
instead of /dev/mpath/ entries (as reported earlier).
Setting to FAILS_QA to flag for further analysis.
Like I said in Comment #4, it is impossible to make sure that udev will have the
link set up in time. The reason that/dev/mpath/* entries exist is that before
multipath had the user_friendly_names option, it was easier to seperate the
multipath devices out into their own directory for scripting. Now those
symlinks are mostly useless. You should always use the /dev/mapper/ names. On
some systems udev might always get the symlinks up on time. On some systems it
might never, and on some setups it might vary by boot, but that race will always
I trust that you are not seeing the "Cannot open bindings file
[/var/lib/multipath/bindings]: Read-only filesystem" message anymore. That is
all I fixed, related to this bugzilla. If error is fixed, please mark the bug
I should probably change the subject line to reflect what actually got fixed.
Yes, the error message is no longer seen in RHEL 5.1.
Changing the bug status to VERIFIED.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.