Bug 241189 - [NetApp-S 5.1 bug] multipath unable to read the user_friendly_names bindings file during boot
Summary: [NetApp-S 5.1 bug] multipath unable to read the user_friendly_names bindings ...
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath   
(Show other bugs)
Version: 5.0
Hardware: All
OS: Linux
Target Milestone: ---
: ---
Assignee: Ben Marzinski
QA Contact: Corey Marthaler
Keywords: OtherQA, Regression
Depends On:
Blocks: 217207
TreeView+ depends on / blocked
Reported: 2007-05-24 12:16 UTC by Martin George
Modified: 2010-01-12 02:37 UTC (History)
17 users (show)

Fixed In Version: RHEA-2007-0531
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-11-07 16:47:29 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Console log (215.43 KB, application/zip)
2007-08-27 14:37 UTC, Martin George
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2007:0531 normal SHIPPED_LIVE device-mapper-multipath enhancement update 2007-10-30 16:03:59 UTC

Description Martin George 2007-05-24 12:16:23 UTC
Description of problem:
Automounting of dm-mp devices using the mpath softlink entries present in the
/dev/mpath/ directory in the fstab fails on a RHEL5 host.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Configure dm-mp devices on a RHEL5 host with "user_friendly_names" set to 
"yes" in the multipath.conf file. This would put mpath soft links in the 
/dev/mpath/ directory which point to the actual dm-mp block devices.

2. Now add a /dev/mpath/ entry to the fstab file to enable automounting of the 
dm-mp device as shown below:
/dev/mpath/mpath1       /mnt/tmp                ext3    defaults        0 0

3. Run a mount command on the above mpath entry which is successful i.e. the 
corresponding dm-mp device is mounted at /mnt/tmp/.

4. Now reboot the host and verify whether the dm-mp device is still mounted on 
Actual results:
The automount of the above dm-mp device fails during bootup.

Expected results:
The above automount should have succeeded.

Additional info:

1. Automounting using /dev/mapper/ entries is still successful. But the same 
is not true with /dev/mpath/ entries as described above.

2. This is not seen on RHEL4 hosts. So this does look like a regression on RHEL5.

3. Apparently setting user_friendly_names to "yes" in the multipath.conf file on
a RHEL5 host causes issues as is evident from the following message thrown on
the console during bootup:
"Cannot open bindings file [/var/lib/multipath/bindings]: Read-only filesystem"

So no mpath softlinks are available during bootup which would obviously cause
the automount to fail.

Comment 2 Ben Marzinski 2007-05-31 19:05:41 UTC
This is probably do to changes in how multipath interacts with udev between
RHEL4 and RHEL5.  I'll take a look into it. However, there really is no reason to
use the /dev/mpath path over the /dev/mapper one.  It is useful to have your
multipath devices all in one directory, but the /dev/mapper names are just as
unique and persistent, and they are guaranteed to exist as soon as device mapper
has created the device, which is not true of the /dev/mpath/* names or the
/dev/dm-* names (which are created later by udev)

Comment 4 Ben Marzinski 2007-06-13 15:16:05 UTC
I've fixed the cause of the "Cannot open bindings file
[/var/lib/multipath/bindings]: Read-only filesystem" message, so if that was the
only thing keeping you from using the /dev/mpath symlinks, it should work know.
However, it is still better to use the /dev/mapper names during bootup, since
you can't guarantee that udev will always have the symlink created in time, for
all setups.

Comment 6 Martin George 2007-08-04 20:01:07 UTC
I've tried automounting using the /dev/mpath/ symlinks on RHEL 5.1 Beta1 and it
works fine. But as recommended, we would be using /dev/mapper/ entries alone.

Comment 7 John Poelstra 2007-08-14 19:42:09 UTC
A fix for this issue has been included in the packages contained in the beta
(RHN channel) or most recent snapshot (partners.redhat.com) for RHEL5.1.  Please
verify that your issue is fixed.

After you (Red Hat Partner) have verified that this issue has been addressed,
please perform the following:
1) Change the *status* of this bug to VERIFIED.
2) Add *keyword* of PartnerVerified (leaving the existing keywords unmodified)

If this issue is not fixed, please add a comment describing the most recent
symptoms of the problem you are having and change the status of the bug to ASSIGNED.

Comment 8 Martin George 2007-08-16 07:22:17 UTC
I have verifed the fix on RHEL 5.1 Beta1. Making the appropriate changes to this

Comment 9 Martin George 2007-08-27 14:32:28 UTC
I tried verifying the same on the latest RHEL 5.1 Snapshot3 x86_64 kernel. But
unfortunately there seems to be a problem here. During reboot, the filesystem
fsck of the corresponding mpath entry fails with the following error:

"Checking filesystems
fsck.ext3: No such file or directory while trying to open /dev/mpath/mpath0"

One is then dropped to a shell for repair. At this point, I can see the 
/dev/mpath/mpath0 entry pointing to a corresponding dm device, and a manual
mount of this device is successful. This proves that the device is available.
Strangely, I can't edit any files now as the root partition is mounted read-only. 

The only way now to successfully reboot is:
1) Remount the root partition in read-write.
2) Edit the /etc/fstab file by removing the corresponding mpath entry in it.
3) Then run a reboot. 

Reopening this bugzilla.

Comment 10 Martin George 2007-08-27 14:37:46 UTC
Created attachment 173621 [details]
Console log

Console log displaying fsck error during reboot. 

Note: Automount of dm-mp devices work fine while using /dev/mapper/ entries
instead of /dev/mpath/ entries (as reported earlier).

Comment 11 Andrius Benokraitis 2007-08-27 17:41:41 UTC
Setting to FAILS_QA to flag for further analysis.

Comment 12 Ben Marzinski 2007-08-28 16:49:25 UTC
Like I said in Comment #4, it is impossible to make sure that udev will have the
link set up in time. The reason that/dev/mpath/* entries exist is that before
multipath had the user_friendly_names option, it was easier to seperate the
multipath devices out into their own directory for scripting.  Now those
symlinks are mostly useless.  You should always use the /dev/mapper/ names. On
some systems udev might always get the symlinks up on time.  On some systems it
might never, and on some setups it might vary by boot, but that race will always
be there.

I trust that you are not seeing the "Cannot open bindings file
[/var/lib/multipath/bindings]: Read-only filesystem" message anymore.  That is
all I fixed, related to this bugzilla.  If error is fixed, please mark the bug
as verified.

I should probably change the subject line to reflect what actually got fixed.

Comment 13 NetApp filed bugzillas 2007-08-29 12:36:02 UTC
Yes, the error message is no longer seen in RHEL 5.1.

Changing the bug status to VERIFIED.

Comment 15 errata-xmlrpc 2007-11-07 16:47:29 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.