Red Hat Bugzilla – Bug 241189
[NetApp-S 5.1 bug] multipath unable to read the user_friendly_names bindings file during boot
Last modified: 2010-01-11 21:37:44 EST
Description of problem: Automounting of dm-mp devices using the mpath softlink entries present in the /dev/mpath/ directory in the fstab fails on a RHEL5 host. Version-Release number of selected component (if applicable): device-mapper-multipath-0.4.7-8.el5 How reproducible: Always Steps to Reproduce: 1. Configure dm-mp devices on a RHEL5 host with "user_friendly_names" set to "yes" in the multipath.conf file. This would put mpath soft links in the /dev/mpath/ directory which point to the actual dm-mp block devices. 2. Now add a /dev/mpath/ entry to the fstab file to enable automounting of the dm-mp device as shown below: /dev/mpath/mpath1 /mnt/tmp ext3 defaults 0 0 3. Run a mount command on the above mpath entry which is successful i.e. the corresponding dm-mp device is mounted at /mnt/tmp/. 4. Now reboot the host and verify whether the dm-mp device is still mounted on /mnt/tmp/. Actual results: The automount of the above dm-mp device fails during bootup. Expected results: The above automount should have succeeded. Additional info: 1. Automounting using /dev/mapper/ entries is still successful. But the same is not true with /dev/mpath/ entries as described above. 2. This is not seen on RHEL4 hosts. So this does look like a regression on RHEL5. 3. Apparently setting user_friendly_names to "yes" in the multipath.conf file on a RHEL5 host causes issues as is evident from the following message thrown on the console during bootup: "Cannot open bindings file [/var/lib/multipath/bindings]: Read-only filesystem" So no mpath softlinks are available during bootup which would obviously cause the automount to fail.
This is probably do to changes in how multipath interacts with udev between RHEL4 and RHEL5. I'll take a look into it. However, there really is no reason to use the /dev/mpath path over the /dev/mapper one. It is useful to have your multipath devices all in one directory, but the /dev/mapper names are just as unique and persistent, and they are guaranteed to exist as soon as device mapper has created the device, which is not true of the /dev/mpath/* names or the /dev/dm-* names (which are created later by udev)
I've fixed the cause of the "Cannot open bindings file [/var/lib/multipath/bindings]: Read-only filesystem" message, so if that was the only thing keeping you from using the /dev/mpath symlinks, it should work know. However, it is still better to use the /dev/mapper names during bootup, since you can't guarantee that udev will always have the symlink created in time, for all setups.
I've tried automounting using the /dev/mpath/ symlinks on RHEL 5.1 Beta1 and it works fine. But as recommended, we would be using /dev/mapper/ entries alone.
A fix for this issue has been included in the packages contained in the beta (RHN channel) or most recent snapshot (partners.redhat.com) for RHEL5.1. Please verify that your issue is fixed. After you (Red Hat Partner) have verified that this issue has been addressed, please perform the following: 1) Change the *status* of this bug to VERIFIED. 2) Add *keyword* of PartnerVerified (leaving the existing keywords unmodified) If this issue is not fixed, please add a comment describing the most recent symptoms of the problem you are having and change the status of the bug to ASSIGNED.
I have verifed the fix on RHEL 5.1 Beta1. Making the appropriate changes to this bugzilla.
I tried verifying the same on the latest RHEL 5.1 Snapshot3 x86_64 kernel. But unfortunately there seems to be a problem here. During reboot, the filesystem fsck of the corresponding mpath entry fails with the following error: "Checking filesystems fsck.ext3: No such file or directory while trying to open /dev/mpath/mpath0" One is then dropped to a shell for repair. At this point, I can see the /dev/mpath/mpath0 entry pointing to a corresponding dm device, and a manual mount of this device is successful. This proves that the device is available. Strangely, I can't edit any files now as the root partition is mounted read-only. The only way now to successfully reboot is: 1) Remount the root partition in read-write. 2) Edit the /etc/fstab file by removing the corresponding mpath entry in it. 3) Then run a reboot. Reopening this bugzilla.
Created attachment 173621 [details] Console log Console log displaying fsck error during reboot. Note: Automount of dm-mp devices work fine while using /dev/mapper/ entries instead of /dev/mpath/ entries (as reported earlier).
Setting to FAILS_QA to flag for further analysis.
Like I said in Comment #4, it is impossible to make sure that udev will have the link set up in time. The reason that/dev/mpath/* entries exist is that before multipath had the user_friendly_names option, it was easier to seperate the multipath devices out into their own directory for scripting. Now those symlinks are mostly useless. You should always use the /dev/mapper/ names. On some systems udev might always get the symlinks up on time. On some systems it might never, and on some setups it might vary by boot, but that race will always be there. I trust that you are not seeing the "Cannot open bindings file [/var/lib/multipath/bindings]: Read-only filesystem" message anymore. That is all I fixed, related to this bugzilla. If error is fixed, please mark the bug as verified. I should probably change the subject line to reflect what actually got fixed.
Yes, the error message is no longer seen in RHEL 5.1. Changing the bug status to VERIFIED.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2007-0531.html