Bug 445734 - Documentation needed on booting from a multipath device
Documentation needed on booting from a multipath device
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: DM-Multipath_Guide (Show other bugs)
5.3
All Linux
high Severity low
: rc
: ---
Assigned To: Steven J. Levine
ecs-bugs
: Documentation
Depends On:
Blocks: 668957 746775
  Show dependency treegraph
 
Reported: 2008-05-08 16:06 EDT by Steven J. Levine
Modified: 2013-05-20 16:17 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 746775 (view as bug list)
Environment:
Last Closed: 2013-05-20 16:17:02 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Screen shot of potential multipath volume mount race condition (2.29 MB, image/jpeg)
2009-08-25 13:04 EDT, John Lange
no flags Details
Screen shot of potential multipath volume mount race condition (99.57 KB, image/jpeg)
2009-08-25 15:22 EDT, John Lange
no flags Details

  None (edit)
Description Steven J. Levine 2008-05-08 16:06:28 EDT
Description of problem:
There have been requests from the field to provide documentation for configuring
and administering a multipathed device that can be used as a boot device. 

What follows are four pieces of correspondence related to this information,
which is all that I currently know.

1. My correspondence with Ben Marzinski, as part of an exchange we had while I
was putting together the initial manual.

2. A followup note to Ben.

3. A second followup note to Ben.

4. A note to Tom Coughlin regarding this issue.


1. (Note from Ben in response to my question)


Here's the initial info you sent me...

> > - BOOTING FROM A MULTIPATH DEVICE:  We had an informal talk about this,
> > from which I took away the information that there are a bunch of issues
> > here. Can we put together at least a bit of advice about the gotchas in
> > using a multipathed device for booting?

When you are booting the system up, you don't have access to your normal
/etc/multipath.conf. The initrd has its own copy.  To update this copy,
you need to remake the initrd.

You do this by running

mkinitrd -f <initrd_file> <kernel_version>

The initrd file is the one that is pointed to in you current /etc/grub.conf
entry. For instance in my current grub entry looks like

title Red Hat Enterprise Linux Server (2.6.18-53.el5)
       root (hd0,0)
       kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 
console=t
tyS0,115200
       initrd /initrd-2.6.18-53.el5.img

the initrd is specified by the initrd line.  The filenames in grub.conf are
relative to /boot, so my initrd_file would be /boot/initrd-2.6.18-53.el5.img

You can get the kernel_version of your currently running kernel by running

# uname -r
2.6.18-53.el5

So, to remake my existing initrd, I would run


# mkinitrd -f /boot/initrd-2.6.18-53.el5.img 2.6.18-53.el5

Note: This overwrites your existing initrd file. To avoid that, you need to
pick a different initrd_file name, and then make a new entry in grub.conf
that uses that initrd file.

Now for the gotchas.

There are two issues here, multipathed root "/" and multipathed boot 
"/boot".
multipathed root is easy. All you have to do is to make sure to set up your
multipath devices as LVM physical volumes, and then make sure to make your
root filesystem on a logical volume from a volume group that ONLY includes
multipathed devices.  WARNING: Anaconda will not do this for you. You need
to manually set up your devices to ensure this.  If you do not do this, then
only part of your root filesystem will be multipathed, which is pretty
worthless. Once that is done, you can edit you multipath.conf however 
you like
(well, aside from blacklisting your root filesystem) and remake your initrd
and reboot, and you should see the changes take effect.

EXCEPT.. except if you are doing multipathed /boot. Just like with
non-multipathed boot, the GRUB bootloader is not able to read LVM 
devices, so
you must make the /boot filesystem directly on your mutipathed device. 
Obviously
this assumes that both your device driver and system BIOS support 
booting off
this device (this is the iSCSI and FC boot issue). having boot directly on
your multipath.conf device creates one important issue. If you cannot change
turn off user_friendly_names without also editting your /etc/fstab (see 
bugzill
#280761 for more details)

Just as a clarification, there is nothing to worry about with device drivers
or BIOS settings to do multipathed root. It will just work. These special
step are only necessary to set up multipathed /boot

2. A response I wrote up for Ben, after some research:

A few weeks ago you provided me with some basic info about
what you need to take into account when you boot from a multipath
device, or when you use a multipath device as root. I enclose that
information at the end of this note.

What I had asked for was a list of issues and gotchas -- but looking
at your response what I really think I need is something more
procedural (which you already provide in part). Actually, I
need two procedures:

- Configuring a multipath device to be used as a boot device
- Configuring a multipath device to be used as a root device

Those are two separate things, right? What I think we
need is a beginning-to-end procedure that
can be tested before putting into the manual.
I think that will be most useful here.

So what I want to clarify is: What are the steps in these procedures?
Here's what I've deciphered so far from your notes. But I think I'm
missing a few steps (general stuff about setting up root and
boot devices). So -- what's missing?  Is this correct in general?

For root device:
- Edit multipath.conf file if necessary for multipathed device to use
- modprobe, start multipathd, multipath
- Create logical volume using multipath devices as physical volumes
 * make sure volume group includes only multipathed devices
 * must do it manually -- cannot use Anaconda
- Remake initrd
 * mkinitrd, using different initrd_file name than existing file name
 * make a new entry in grub.conf using that file
- Reboot

For boot device:
- Edit multipath.conf file if necessary for multipathed device to use
- modprobe, start multipathd, multipath
- make the /boot filesystem directly on your multipath device
 (rather than through LVM?)
- Remake initrd
 * mkinitrd, using different initrd_file name than existing file name
 * make a new entry in grub.conf using that file
- As per bugzilla 280761: edit /etc/fstab and make sure the root device
 uses the non-user_friendly_name for your device.  You also need to 
make sure
 that the os root in /etc/grub.conf uses the non-user_friendly_name as 
well.
 The non-userfriendly-name will be /dev/mapper/WWID[p<partition-number>].
 If any one of these files (the initrd, /etc/fstab, or /etc/grub.conf)
 doesn't match, you won't be able to boot correctly.
- Reboot

Is that correct as far as it goes? Are there other steps involved?

Once we get this down I may write to the cluster/mpath list (as previously
mentioned) to see if anybody can help me come up with actual examples.

3. After further research, a note from me to Ben:

This note is a followup to this note of the other week, in which 
I'm trying to find out a beginning to end procedure for configuring a 
multipath swap and a multipath root device.

In the meantime, Chandra Seetharamam at IBM has put together a multipath 
document that includes this procedure for installing and booting from a 
multipath device in RHEL 5. I'm not sure if that corresponds to what I 
have below. This is the procedure he gives:
--------------
/4.2. Installation instructions for RHEL5/

Note: This is tested on RHEL5 U1. If you have any other version, your 
mileage may vary.

   1. Start the installation with the kernel command line "linux mpath"
          * You will see multipathed devices (/dev/mapper/mpath*) as
            installation devices.
   2. Finish the installation.
   3. Reboot.
          * If your boot device does not need multipath.conf and does
            not have a special hardware handler, then you are done.
            If you have either of these, follow the steps below.
   4. Once booted, update multipath.conf file, if needed.
   5. Run mkinitrd, if you need a hardware handler, add it to initrd
      with --with option.
          * # mkinitrd /boot/initrd.final.img --with=dm-rdac/ /
   6. Replace the initrd in your grub.conf/lilo.conf/yaboot.conf with
      the newly built initrd.
   7. Reboot.

The system will come up with the root disk on a multipathed device.

Note: You can switch off multipathing to the root device by adding 
multipath=off to the kernel command line.
////
/ Note: By default, RedHat <http://sources.redhat.com/lvm2/wiki/RedHat> 
disables dm-multipath by blacklisting all devices in 
/etc/multipath.conf. It just excludes your root device. If you do not 
see your other multipathdevices through "multipath -ll", then check and 
fix the blacklist in /etc/multipath.conf/

Should I use that as a start?

4. My note to Tom Coughlin

Tom:

Paul Kennedy came to me after this morning's field meeting to follow up on the
issue of documenting how to boot from a multipath device, or at least that's my
understanding. Please correct me if I have that wrong. He also noted that you
might be able to help me with this.

Where this issue stands is:  When I put together the first multipath manual, I
got some last-minute info about booting from a multipathed device, but it wasn't
enough to go into the manual, so I did not include it and tried to find out
more. My only contact here has been Ben Marzinski.

I happen to have the last note I sent Ben on this, which summarized what I know
and what I think I need. I will enclose that note as an attachment -- it's sort
of in reverse chronological order, but I think you can make that out. Here's
what it includes (again, in reverse order from this list):

- The original information Ben gave me in response to some questions I asked
- My followup, in which I tried to summarize a procedure.
- A further followup, after I learned of some other existing documentation that
differed a little from what I knew.

I did not back from Ben, who was buried in other work at the time. I should have
followed up myself a couple of weeks later.

Let me know where you think I should take this. Specifically: After you look
over what I sent Ben, could you let me know if this is the issue at hand and
whether I'm on the right track? This is a procedure that I really do need some
input for.

Thanks. 







Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Steven J. Levine 2008-08-10 17:02:36 EDT
On August 10 I sent a followup ping to Tom Coughlin and Ben Marzinski:

Subject: Followup Ping on Multipath Boot Procedure (Bug 445734)

I'm currently looking over all my open documentation issues in the hope of addressing them in time for the RHEL 5.3 release, so this is a re-ping on my looking for information on a procedure for booting from a multipath device. The history of my correspondence on this is all documented in bug 445734.

Instead of copying the whole history here in this email, I'll just provide the URL:

https://bugzilla.redhat.com/show_bug.cgi?id=445734

- Steven Levine
Comment 2 Steven J. Levine 2009-08-11 16:38:12 EDT
It looks as though I will not have this information in time for the RHEL 5.4 release. I've checked the Need additional info box.
Comment 3 John Lange 2009-08-25 12:57:53 EDT
This task is long standing and not much activity but never the less I'd like to offer some help. The environment involves a rack of HP DL360s connected via multipath fiber to an HP EVA4400.

For ease of management, the goal is to have all the disks (including boot and root) mounted by their volume names.

This works great in non-multipath environments but there seems to be issues with multipath. Specifically, it seems there may be a race condition where the mounting phase begins before multipath has fully settled.

Other comments above mention LVM, but it isn't clear if that is a requirement. LVM is obsolete in virtual disk environments so this point should be clarified. If anyone attached to this bug would like some help testing and/or documenting please let me know.
Comment 4 John Lange 2009-08-25 13:04:29 EDT
Created attachment 358602 [details]
Screen shot of potential multipath volume mount race condition
Comment 5 Bryn M. Reeves 2009-08-25 13:11:22 EDT
The attachment in comment #4 is a 2.3M uncompressed Windows bitmap (bmp) file not a jpeg as the attachment name suggests.
Comment 6 John Lange 2009-08-25 15:22:53 EDT
Created attachment 358622 [details]
Screen shot of potential multipath volume mount race condition

Sorry about the bmp. I was stuck using Windows at the time which apparently can not save anything except bmp. This new upload is actually jpg.
Comment 11 RHEL Product and Program Management 2010-08-09 14:35:38 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 12 RHEL Product and Program Management 2011-01-11 16:00:39 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 13 RHEL Product and Program Management 2011-01-11 17:29:28 EST
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 14 RHEL Product and Program Management 2011-05-31 09:37:56 EDT
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 16 Steven J. Levine 2011-10-17 11:50:11 EDT
This bug has been around for a long time, and it should be in both RHEL 5 and RHEL 6, but it seems to be a question of resources in terms of providing the requested procedure as well as evaluating the information provided in Comments 3-6. The bug is in NEEDINFO from Ben Marzinski.

I've set the dev_ack flag for 5.8, but I can't close this bug without the information that it requests.
Comment 17 Ben Marzinski 2011-10-19 16:55:35 EDT
Um.. I'm not
Comment 18 Ben Marzinski 2011-10-19 17:02:46 EDT
Oops.  accidentally clicked on save changes early.

I'm not sure that we actually support this in RHEL5.  I've tried to switch to a multipathed root, from a regular root on one of my own machines, and I can't do it without some serious editting of /sbin/mkinitrd itself.

CC'ing Brian Lane in case I'm missing something here, but it looks like mkinitrd
won't setup a multipathed root initrd, unless you already have a multipathed root
system. And getting a multipathed root system without a multipathed root initrd,
is not an easy feat.
Comment 19 Steven J. Levine 2012-01-10 10:33:39 EST
Still waiting to hear final confirmation regarding Comment#18 that this isn't really supported, so for now I'm moving this to 5.9 (and will likely close it eventually).
Comment 20 Steven J. Levine 2012-08-08 15:47:36 EDT
Moving to 5.10, so as not to lose this completely, but it still looks as if this isn't supported.
Comment 21 Steven J. Levine 2013-05-20 16:17:02 EDT
Based on Comment 18, and the lack of dispute over Ben's assessment, I'm closing this as WONTFIX since it appears we don't support the procedure.

Note You need to log in before you can comment on or make changes to this bug.