Bug 446144 - Multipath devices not ready to mount during boot
Multipath devices not ready to mount during boot
Status: CLOSED DUPLICATE of bug 559852
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Ben Marzinski
Corey Marthaler
Depends On:
  Show dependency treegraph
Reported: 2008-05-12 20:27 EDT by Jacob Luna Lundberg
Modified: 2010-11-09 07:43 EST (History)
18 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2010-09-27 12:05:44 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Messages log from boot with error (46.60 KB, text/plain)
2008-05-12 20:27 EDT, Jacob Luna Lundberg
no flags Details

  None (edit)
Description Jacob Luna Lundberg 2008-05-12 20:27:43 EDT
Description of problem:
During boot, multipath fails to create devices, causing mount to fail and wait
for user input.  This appears to be a new problem within the past release or two
of multipath.  It reminds me of Bug 232849 but this is RHEL5.1.

Version-Release number of selected component (if applicable):

How reproducible:
Every time we reboot one of our production servers.  I will try to start
reproducing it on a test server soon.

Steps to Reproduce:
1. Use multipath
2. Put a mpath device in /etc/fstab
3. Reboot

Actual results:
Log entries note multipathd failing to access files (in the initrd?).  Boot
waits for user intervention because /dev/mapper/25L_66p1 does not exist.

Expected results:
Successful boot and mounted device.

Additional info:
I will attach the output of /var/log/messages from a boot where I entered the
root password, remounted / as rw, and continued booting without the missing
multipath device.
Comment 1 Jacob Luna Lundberg 2008-05-12 20:27:43 EDT
Created attachment 305188 [details]
Messages log from boot with error
Comment 2 Jacob Luna Lundberg 2008-08-04 16:20:08 EDT
This happened again with a RHEL 5.2 box and I had more time to investigate.  The problem is that /sbin/mpath_prio_hds_modular tries to create a file in /tmp while / (and therefore /tmp) is read-only during bootup.

This happens because device names have changed and now an mpath target exists on a device name (such as sdb) which was previously not an mpath target.  Otherwise /sbin/mpath_prio_hds_modular is able to use the file it already created in /tmp during a time when /tmp was not read-only.

Upon reaching the recovery console and logging on with the root password, the problem can be fixed (until the next time a device changes names) by doing:

mount -o remount,rw
Comment 3 Jacob Luna Lundberg 2008-08-04 16:30:57 EDT
mount -o remount,rw /

Bugzilla ate my comment the first time and when I retyped it I left the / off.
Comment 4 Pete Philips 2009-11-12 07:02:35 EST
I am seeing the same on a RHEL 5.3 system . I note that this was first reported on RHEL 5.1 . Has it been fixed in RHEL 5.4?
Comment 5 Pete Philips 2009-11-12 09:13:51 EST
Apologies for the slightly content light comment above. Here's some more detail:

The SAN is an HP EVA4400
The HBA is an Emulex LP10000
The OS is RHEL 5.3 with kernel kernel-PAE-2.6.18-164 (i386).

I have one multipath device, mpath1. This is mounted in fstab thus:

/dev/mapper/mpath1      /x                      ext3    defaults,noatime     0 0

On reboot, mounting of that filesystem fails with

"/dev/mapper/mpath1 does not exist"

After boot is complete I can mount it with

# mount /dev/mapper/mpath1 /x

Changing the fstab entry to

/dev/dm-0      /x                      ext3    defaults,noatime     0 0

boots fine but quoting from http://kbase.redhat.com/faq/docs/DOC-5551

"The /dev/dm-N devices are internal to device mapper and should never be used. These devices are not persistent. Starting with Red Hat Enterprise Linux 5, these devices are no longer created by udev."

So I'm a bit stumped. Can anyone recommend a course of action?
Comment 6 Alasdair Kergon 2009-11-12 09:28:22 EST
Looking at the date on this, assuming you have a support agreement, I suggest opening a ticket here (and mention this bugzilla number): https://www.redhat.com/apps/support/
Comment 8 Ben Marzinski 2010-05-12 19:07:41 EDT
Fixing the original bug should be no problem, but I am not able to reproduce the newer instance from Comment #5, and it's pretty unlikely that they share the same cause.  In RHEL5 multipath still does create /dev/dm-* files in udev, but this shouldn't happen unless the /dev/mapper/* file is also created, and that device node should be created before multipath returns, so It should definitely be there when the filesystem is mounted.

Are you still able to see this problem?  If so, can I see your /etc/multipath.conf file?
Comment 9 Mark Wu 2010-07-30 02:42:14 EDT
 I think /dev is a better location to create temporarily device node for mpath_prio_hds_modular, since it can also work when /tmp is mounted read-only during boot.
Comment 10 Mark Wu 2010-07-30 02:57:54 EDT
This issue happens on DF600 array which use priority callout mpath_prio_hds_modular
Comment 11 Mark Wu 2010-07-30 03:03:02 EDT
Currently, the customer add "_netdev" in the fstab entry which uses the multipath device, and it works fine with this around.
Comment 12 Mark Wu 2010-07-30 03:06:14 EDT
On server reboot the following messages are displayed on the console:
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:0
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:16
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:32
Comment 13 Ben Marzinski 2010-07-30 12:22:46 EDT
Actually, it shouldn't need to create a temporary file at all.  The device file for that scsi device must exist, if you already got this far.  The problem is that mpath_prio_hds_modular doesn't use it.  Most of the prio callouts
take a device name and simply open that device.  mpath_prio_hds_modular takes
a major:minor number, and creates a temporary device to use. The solution is to make it accept device names like /dev/sda, etc.  And then change the default configuration to call it with that.
Comment 14 Ben Marzinski 2010-09-27 12:05:44 EDT

*** This bug has been marked as a duplicate of bug 559852 ***

Note You need to log in before you can comment on or make changes to this bug.