Bug 446144

Summary: Multipath devices not ready to mount during boot
Product: Red Hat Enterprise Linux 5 Reporter: Jacob Luna Lundberg <jacob>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED DUPLICATE QA Contact: Corey Marthaler <cmarthal>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.1CC: agk, bmarzins, bmr, christophe.varoqui, dwu, dwysocha, egoggin, heinzm, iannis, james.hofmeister, junichi.nomura, kueda, lmb, mbroz, pete.philips, prockai, tao, tranlan
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-09-27 16:05:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Messages log from boot with error none

Description Jacob Luna Lundberg 2008-05-13 00:27:43 UTC
Description of problem:
During boot, multipath fails to create devices, causing mount to fail and wait
for user input.  This appears to be a new problem within the past release or two
of multipath.  It reminds me of Bug 232849 but this is RHEL5.1.


Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.7-12.el5_1.3


How reproducible:
Every time we reboot one of our production servers.  I will try to start
reproducing it on a test server soon.


Steps to Reproduce:
1. Use multipath
2. Put a mpath device in /etc/fstab
3. Reboot


Actual results:
Log entries note multipathd failing to access files (in the initrd?).  Boot
waits for user intervention because /dev/mapper/25L_66p1 does not exist.


Expected results:
Successful boot and mounted device.


Additional info:
I will attach the output of /var/log/messages from a boot where I entered the
root password, remounted / as rw, and continued booting without the missing
multipath device.

Comment 1 Jacob Luna Lundberg 2008-05-13 00:27:43 UTC
Created attachment 305188 [details]
Messages log from boot with error

Comment 2 Jacob Luna Lundberg 2008-08-04 20:20:08 UTC
This happened again with a RHEL 5.2 box and I had more time to investigate.  The problem is that /sbin/mpath_prio_hds_modular tries to create a file in /tmp while / (and therefore /tmp) is read-only during bootup.

This happens because device names have changed and now an mpath target exists on a device name (such as sdb) which was previously not an mpath target.  Otherwise /sbin/mpath_prio_hds_modular is able to use the file it already created in /tmp during a time when /tmp was not read-only.

Upon reaching the recovery console and logging on with the root password, the problem can be fixed (until the next time a device changes names) by doing:

mount -o remount,rw
multipath
^D

Comment 3 Jacob Luna Lundberg 2008-08-04 20:30:57 UTC
mount -o remount,rw /

Bugzilla ate my comment the first time and when I retyped it I left the / off.

Comment 4 Pete Philips 2009-11-12 12:02:35 UTC
I am seeing the same on a RHEL 5.3 system . I note that this was first reported on RHEL 5.1 . Has it been fixed in RHEL 5.4?

Comment 5 Pete Philips 2009-11-12 14:13:51 UTC
Apologies for the slightly content light comment above. Here's some more detail:

The SAN is an HP EVA4400
The HBA is an Emulex LP10000
The OS is RHEL 5.3 with kernel kernel-PAE-2.6.18-164 (i386).

I have one multipath device, mpath1. This is mounted in fstab thus:

/dev/mapper/mpath1      /x                      ext3    defaults,noatime     0 0

On reboot, mounting of that filesystem fails with

"/dev/mapper/mpath1 does not exist"

After boot is complete I can mount it with

# mount /dev/mapper/mpath1 /x

Changing the fstab entry to

/dev/dm-0      /x                      ext3    defaults,noatime     0 0

boots fine but quoting from http://kbase.redhat.com/faq/docs/DOC-5551

"The /dev/dm-N devices are internal to device mapper and should never be used. These devices are not persistent. Starting with Red Hat Enterprise Linux 5, these devices are no longer created by udev."

So I'm a bit stumped. Can anyone recommend a course of action?

Comment 6 Alasdair Kergon 2009-11-12 14:28:22 UTC
Looking at the date on this, assuming you have a support agreement, I suggest opening a ticket here (and mention this bugzilla number): https://www.redhat.com/apps/support/

Comment 8 Ben Marzinski 2010-05-12 23:07:41 UTC
Fixing the original bug should be no problem, but I am not able to reproduce the newer instance from Comment #5, and it's pretty unlikely that they share the same cause.  In RHEL5 multipath still does create /dev/dm-* files in udev, but this shouldn't happen unless the /dev/mapper/* file is also created, and that device node should be created before multipath returns, so It should definitely be there when the filesystem is mounted.

Are you still able to see this problem?  If so, can I see your /etc/multipath.conf file?

Comment 9 Mark Wu 2010-07-30 06:42:14 UTC
 I think /dev is a better location to create temporarily device node for mpath_prio_hds_modular, since it can also work when /tmp is mounted read-only during boot.

Comment 10 Mark Wu 2010-07-30 06:57:54 UTC
This issue happens on DF600 array which use priority callout mpath_prio_hds_modular

Comment 11 Mark Wu 2010-07-30 07:03:02 UTC
Currently, the customer add "_netdev" in the fstab entry which uses the multipath device, and it works fine with this around.

Comment 12 Mark Wu 2010-07-30 07:06:14 UTC
On server reboot the following messages are displayed on the console:
 
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:0
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:16
  /sbin/mpath_prio_hds_moduluar exitted with 1
  error calling out /sbin/mpath_prio_hds_modular 8:32

Comment 13 Ben Marzinski 2010-07-30 16:22:46 UTC
Actually, it shouldn't need to create a temporary file at all.  The device file for that scsi device must exist, if you already got this far.  The problem is that mpath_prio_hds_modular doesn't use it.  Most of the prio callouts
take a device name and simply open that device.  mpath_prio_hds_modular takes
a major:minor number, and creates a temporary device to use. The solution is to make it accept device names like /dev/sda, etc.  And then change the default configuration to call it with that.

Comment 14 Ben Marzinski 2010-09-27 16:05:44 UTC

*** This bug has been marked as a duplicate of bug 559852 ***