Bug 445734

Description of problem:
There have been requests from the field to provide documentation for configuring
and administering a multipathed device that can be used as a boot device. 

What follows are four pieces of correspondence related to this information,
which is all that I currently know.

1. My correspondence with Ben Marzinski, as part of an exchange we had while I
was putting together the initial manual.

2. A followup note to Ben.

3. A second followup note to Ben.

4. A note to Tom Coughlin regarding this issue.


1. (Note from Ben in response to my question)


Here's the initial info you sent me...

> > - BOOTING FROM A MULTIPATH DEVICE:  We had an informal talk about this,
> > from which I took away the information that there are a bunch of issues
> > here. Can we put together at least a bit of advice about the gotchas in
> > using a multipathed device for booting?

When you are booting the system up, you don't have access to your normal
/etc/multipath.conf. The initrd has its own copy.  To update this copy,
you need to remake the initrd.

You do this by running

mkinitrd -f <initrd_file> <kernel_version>

The initrd file is the one that is pointed to in you current /etc/grub.conf
entry. For instance in my current grub entry looks like

title Red Hat Enterprise Linux Server (2.6.18-53.el5)
       root (hd0,0)
       kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 
console=t
tyS0,115200
       initrd /initrd-2.6.18-53.el5.img

the initrd is specified by the initrd line.  The filenames in grub.conf are
relative to /boot, so my initrd_file would be /boot/initrd-2.6.18-53.el5.img

You can get the kernel_version of your currently running kernel by running

# uname -r
2.6.18-53.el5

So, to remake my existing initrd, I would run


# mkinitrd -f /boot/initrd-2.6.18-53.el5.img 2.6.18-53.el5

Note: This overwrites your existing initrd file. To avoid that, you need to
pick a different initrd_file name, and then make a new entry in grub.conf
that uses that initrd file.

Now for the gotchas.

There are two issues here, multipathed root "/" and multipathed boot 
"/boot".
multipathed root is easy. All you have to do is to make sure to set up your
multipath devices as LVM physical volumes, and then make sure to make your
root filesystem on a logical volume from a volume group that ONLY includes
multipathed devices.  WARNING: Anaconda will not do this for you. You need
to manually set up your devices to ensure this.  If you do not do this, then
only part of your root filesystem will be multipathed, which is pretty
worthless. Once that is done, you can edit you multipath.conf however 
you like
(well, aside from blacklisting your root filesystem) and remake your initrd
and reboot, and you should see the changes take effect.

EXCEPT.. except if you are doing multipathed /boot. Just like with
non-multipathed boot, the GRUB bootloader is not able to read LVM 
devices, so
you must make the /boot filesystem directly on your mutipathed device. 
Obviously
this assumes that both your device driver and system BIOS support 
booting off
this device (this is the iSCSI and FC boot issue). having boot directly on
your multipath.conf device creates one important issue. If you cannot change
turn off user_friendly_names without also editting your /etc/fstab (see 
bugzill
#280761 for more details)

Just as a clarification, there is nothing to worry about with device drivers
or BIOS settings to do multipathed root. It will just work. These special
step are only necessary to set up multipathed /boot

2. A response I wrote up for Ben, after some research:

A few weeks ago you provided me with some basic info about
what you need to take into account when you boot from a multipath
device, or when you use a multipath device as root. I enclose that
information at the end of this note.

What I had asked for was a list of issues and gotchas -- but looking
at your response what I really think I need is something more
procedural (which you already provide in part). Actually, I
need two procedures:

- Configuring a multipath device to be used as a boot device
- Configuring a multipath device to be used as a root device

Those are two separate things, right? What I think we
need is a beginning-to-end procedure that
can be tested before putting into the manual.
I think that will be most useful here.

So what I want to clarify is: What are the steps in these procedures?
Here's what I've deciphered so far from your notes. But I think I'm
missing a few steps (general stuff about setting up root and
boot devices). So -- what's missing?  Is this correct in general?

For root device:
- Edit multipath.conf file if necessary for multipathed device to use
- modprobe, start multipathd, multipath
- Create logical volume using multipath devices as physical volumes
 * make sure volume group includes only multipathed devices
 * must do it manually -- cannot use Anaconda
- Remake initrd
 * mkinitrd, using different initrd_file name than existing file name
 * make a new entry in grub.conf using that file
- Reboot

For boot device:
- Edit multipath.conf file if necessary for multipathed device to use
- modprobe, start multipathd, multipath
- make the /boot filesystem directly on your multipath device
 (rather than through LVM?)
- Remake initrd
 * mkinitrd, using different initrd_file name than existing file name
 * make a new entry in grub.conf using that file
- As per bugzilla 280761: edit /etc/fstab and make sure the root device
 uses the non-user_friendly_name for your device.  You also need to 
make sure
 that the os root in /etc/grub.conf uses the non-user_friendly_name as 
well.
 The non-userfriendly-name will be /dev/mapper/WWID[p<partition-number>].
 If any one of these files (the initrd, /etc/fstab, or /etc/grub.conf)
 doesn't match, you won't be able to boot correctly.
- Reboot

Is that correct as far as it goes? Are there other steps involved?

Once we get this down I may write to the cluster/mpath list (as previously
mentioned) to see if anybody can help me come up with actual examples.

3. After further research, a note from me to Ben:

This note is a followup to this note of the other week, in which 
I'm trying to find out a beginning to end procedure for configuring a 
multipath swap and a multipath root device.

In the meantime, Chandra Seetharamam at IBM has put together a multipath 
document that includes this procedure for installing and booting from a 
multipath device in RHEL 5. I'm not sure if that corresponds to what I 
have below. This is the procedure he gives:
--------------
/4.2. Installation instructions for RHEL5/

Note: This is tested on RHEL5 U1. If you have any other version, your 
mileage may vary.

   1. Start the installation with the kernel command line "linux mpath"
          * You will see multipathed devices (/dev/mapper/mpath*) as
            installation devices.
   2. Finish the installation.
   3. Reboot.
          * If your boot device does not need multipath.conf and does
            not have a special hardware handler, then you are done.
            If you have either of these, follow the steps below.
   4. Once booted, update multipath.conf file, if needed.
   5. Run mkinitrd, if you need a hardware handler, add it to initrd
      with --with option.
          * # mkinitrd /boot/initrd.final.img --with=dm-rdac/ /
   6. Replace the initrd in your grub.conf/lilo.conf/yaboot.conf with
      the newly built initrd.
   7. Reboot.

The system will come up with the root disk on a multipathed device.

Note: You can switch off multipathing to the root device by adding 
multipath=off to the kernel command line.
////
/ Note: By default, RedHat <http://sources.redhat.com/lvm2/wiki/RedHat> 
disables dm-multipath by blacklisting all devices in 
/etc/multipath.conf. It just excludes your root device. If you do not 
see your other multipathdevices through "multipath -ll", then check and 
fix the blacklist in /etc/multipath.conf/

Should I use that as a start?

4. My note to Tom Coughlin

Tom:

Paul Kennedy came to me after this morning's field meeting to follow up on the
issue of documenting how to boot from a multipath device, or at least that's my
understanding. Please correct me if I have that wrong. He also noted that you
might be able to help me with this.

Where this issue stands is:  When I put together the first multipath manual, I
got some last-minute info about booting from a multipathed device, but it wasn't
enough to go into the manual, so I did not include it and tried to find out
more. My only contact here has been Ben Marzinski.

I happen to have the last note I sent Ben on this, which summarized what I know
and what I think I need. I will enclose that note as an attachment -- it's sort
of in reverse chronological order, but I think you can make that out. Here's
what it includes (again, in reverse order from this list):

- The original information Ben gave me in response to some questions I asked
- My followup, in which I tried to summarize a procedure.
- A further followup, after I learned of some other existing documentation that
differed a little from what I knew.

I did not back from Ben, who was buried in other work at the time. I should have
followed up myself a couple of weeks later.

Let me know where you think I should take this. Specifically: After you look
over what I sent Ben, could you let me know if this is the issue at hand and
whether I'm on the right track? This is a procedure that I really do need some
input for.

Thanks. 







Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info: