Description of problem: There have been requests from the field to provide documentation for configuring and administering a multipathed device that can be used as a boot device. What follows are four pieces of correspondence related to this information, which is all that I currently know. 1. My correspondence with Ben Marzinski, as part of an exchange we had while I was putting together the initial manual. 2. A followup note to Ben. 3. A second followup note to Ben. 4. A note to Tom Coughlin regarding this issue. 1. (Note from Ben in response to my question) Here's the initial info you sent me... > > - BOOTING FROM A MULTIPATH DEVICE: We had an informal talk about this, > > from which I took away the information that there are a bunch of issues > > here. Can we put together at least a bit of advice about the gotchas in > > using a multipathed device for booting? When you are booting the system up, you don't have access to your normal /etc/multipath.conf. The initrd has its own copy. To update this copy, you need to remake the initrd. You do this by running mkinitrd -f <initrd_file> <kernel_version> The initrd file is the one that is pointed to in you current /etc/grub.conf entry. For instance in my current grub entry looks like title Red Hat Enterprise Linux Server (2.6.18-53.el5) root (hd0,0) kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 console=t tyS0,115200 initrd /initrd-2.6.18-53.el5.img the initrd is specified by the initrd line. The filenames in grub.conf are relative to /boot, so my initrd_file would be /boot/initrd-2.6.18-53.el5.img You can get the kernel_version of your currently running kernel by running # uname -r 2.6.18-53.el5 So, to remake my existing initrd, I would run # mkinitrd -f /boot/initrd-2.6.18-53.el5.img 2.6.18-53.el5 Note: This overwrites your existing initrd file. To avoid that, you need to pick a different initrd_file name, and then make a new entry in grub.conf that uses that initrd file. Now for the gotchas. There are two issues here, multipathed root "/" and multipathed boot "/boot". multipathed root is easy. All you have to do is to make sure to set up your multipath devices as LVM physical volumes, and then make sure to make your root filesystem on a logical volume from a volume group that ONLY includes multipathed devices. WARNING: Anaconda will not do this for you. You need to manually set up your devices to ensure this. If you do not do this, then only part of your root filesystem will be multipathed, which is pretty worthless. Once that is done, you can edit you multipath.conf however you like (well, aside from blacklisting your root filesystem) and remake your initrd and reboot, and you should see the changes take effect. EXCEPT.. except if you are doing multipathed /boot. Just like with non-multipathed boot, the GRUB bootloader is not able to read LVM devices, so you must make the /boot filesystem directly on your mutipathed device. Obviously this assumes that both your device driver and system BIOS support booting off this device (this is the iSCSI and FC boot issue). having boot directly on your multipath.conf device creates one important issue. If you cannot change turn off user_friendly_names without also editting your /etc/fstab (see bugzill #280761 for more details) Just as a clarification, there is nothing to worry about with device drivers or BIOS settings to do multipathed root. It will just work. These special step are only necessary to set up multipathed /boot 2. A response I wrote up for Ben, after some research: A few weeks ago you provided me with some basic info about what you need to take into account when you boot from a multipath device, or when you use a multipath device as root. I enclose that information at the end of this note. What I had asked for was a list of issues and gotchas -- but looking at your response what I really think I need is something more procedural (which you already provide in part). Actually, I need two procedures: - Configuring a multipath device to be used as a boot device - Configuring a multipath device to be used as a root device Those are two separate things, right? What I think we need is a beginning-to-end procedure that can be tested before putting into the manual. I think that will be most useful here. So what I want to clarify is: What are the steps in these procedures? Here's what I've deciphered so far from your notes. But I think I'm missing a few steps (general stuff about setting up root and boot devices). So -- what's missing? Is this correct in general? For root device: - Edit multipath.conf file if necessary for multipathed device to use - modprobe, start multipathd, multipath - Create logical volume using multipath devices as physical volumes * make sure volume group includes only multipathed devices * must do it manually -- cannot use Anaconda - Remake initrd * mkinitrd, using different initrd_file name than existing file name * make a new entry in grub.conf using that file - Reboot For boot device: - Edit multipath.conf file if necessary for multipathed device to use - modprobe, start multipathd, multipath - make the /boot filesystem directly on your multipath device (rather than through LVM?) - Remake initrd * mkinitrd, using different initrd_file name than existing file name * make a new entry in grub.conf using that file - As per bugzilla 280761: edit /etc/fstab and make sure the root device uses the non-user_friendly_name for your device. You also need to make sure that the os root in /etc/grub.conf uses the non-user_friendly_name as well. The non-userfriendly-name will be /dev/mapper/WWID[p<partition-number>]. If any one of these files (the initrd, /etc/fstab, or /etc/grub.conf) doesn't match, you won't be able to boot correctly. - Reboot Is that correct as far as it goes? Are there other steps involved? Once we get this down I may write to the cluster/mpath list (as previously mentioned) to see if anybody can help me come up with actual examples. 3. After further research, a note from me to Ben: This note is a followup to this note of the other week, in which I'm trying to find out a beginning to end procedure for configuring a multipath swap and a multipath root device. In the meantime, Chandra Seetharamam at IBM has put together a multipath document that includes this procedure for installing and booting from a multipath device in RHEL 5. I'm not sure if that corresponds to what I have below. This is the procedure he gives: -------------- /4.2. Installation instructions for RHEL5/ Note: This is tested on RHEL5 U1. If you have any other version, your mileage may vary. 1. Start the installation with the kernel command line "linux mpath" * You will see multipathed devices (/dev/mapper/mpath*) as installation devices. 2. Finish the installation. 3. Reboot. * If your boot device does not need multipath.conf and does not have a special hardware handler, then you are done. If you have either of these, follow the steps below. 4. Once booted, update multipath.conf file, if needed. 5. Run mkinitrd, if you need a hardware handler, add it to initrd with --with option. * # mkinitrd /boot/initrd.final.img --with=dm-rdac/ / 6. Replace the initrd in your grub.conf/lilo.conf/yaboot.conf with the newly built initrd. 7. Reboot. The system will come up with the root disk on a multipathed device. Note: You can switch off multipathing to the root device by adding multipath=off to the kernel command line. //// / Note: By default, RedHat <http://sources.redhat.com/lvm2/wiki/RedHat> disables dm-multipath by blacklisting all devices in /etc/multipath.conf. It just excludes your root device. If you do not see your other multipathdevices through "multipath -ll", then check and fix the blacklist in /etc/multipath.conf/ Should I use that as a start? 4. My note to Tom Coughlin Tom: Paul Kennedy came to me after this morning's field meeting to follow up on the issue of documenting how to boot from a multipath device, or at least that's my understanding. Please correct me if I have that wrong. He also noted that you might be able to help me with this. Where this issue stands is: When I put together the first multipath manual, I got some last-minute info about booting from a multipathed device, but it wasn't enough to go into the manual, so I did not include it and tried to find out more. My only contact here has been Ben Marzinski. I happen to have the last note I sent Ben on this, which summarized what I know and what I think I need. I will enclose that note as an attachment -- it's sort of in reverse chronological order, but I think you can make that out. Here's what it includes (again, in reverse order from this list): - The original information Ben gave me in response to some questions I asked - My followup, in which I tried to summarize a procedure. - A further followup, after I learned of some other existing documentation that differed a little from what I knew. I did not back from Ben, who was buried in other work at the time. I should have followed up myself a couple of weeks later. Let me know where you think I should take this. Specifically: After you look over what I sent Ben, could you let me know if this is the issue at hand and whether I'm on the right track? This is a procedure that I really do need some input for. Thanks. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
On August 10 I sent a followup ping to Tom Coughlin and Ben Marzinski: Subject: Followup Ping on Multipath Boot Procedure (Bug 445734) I'm currently looking over all my open documentation issues in the hope of addressing them in time for the RHEL 5.3 release, so this is a re-ping on my looking for information on a procedure for booting from a multipath device. The history of my correspondence on this is all documented in bug 445734. Instead of copying the whole history here in this email, I'll just provide the URL: https://bugzilla.redhat.com/show_bug.cgi?id=445734 - Steven Levine
It looks as though I will not have this information in time for the RHEL 5.4 release. I've checked the Need additional info box.
This task is long standing and not much activity but never the less I'd like to offer some help. The environment involves a rack of HP DL360s connected via multipath fiber to an HP EVA4400. For ease of management, the goal is to have all the disks (including boot and root) mounted by their volume names. This works great in non-multipath environments but there seems to be issues with multipath. Specifically, it seems there may be a race condition where the mounting phase begins before multipath has fully settled. Other comments above mention LVM, but it isn't clear if that is a requirement. LVM is obsolete in virtual disk environments so this point should be clarified. If anyone attached to this bug would like some help testing and/or documenting please let me know.
Created attachment 358602 [details] Screen shot of potential multipath volume mount race condition
The attachment in comment #4 is a 2.3M uncompressed Windows bitmap (bmp) file not a jpeg as the attachment name suggests.
Created attachment 358622 [details] Screen shot of potential multipath volume mount race condition Sorry about the bmp. I was stuck using Windows at the time which apparently can not save anything except bmp. This new upload is actually jpg.
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
This request was erroneously denied for the current release of Red Hat Enterprise Linux. The error has been fixed and this request has been re-proposed for the current release.
This bug has been around for a long time, and it should be in both RHEL 5 and RHEL 6, but it seems to be a question of resources in terms of providing the requested procedure as well as evaluating the information provided in Comments 3-6. The bug is in NEEDINFO from Ben Marzinski. I've set the dev_ack flag for 5.8, but I can't close this bug without the information that it requests.
Um.. I'm not
Oops. accidentally clicked on save changes early. I'm not sure that we actually support this in RHEL5. I've tried to switch to a multipathed root, from a regular root on one of my own machines, and I can't do it without some serious editting of /sbin/mkinitrd itself. CC'ing Brian Lane in case I'm missing something here, but it looks like mkinitrd won't setup a multipathed root initrd, unless you already have a multipathed root system. And getting a multipathed root system without a multipathed root initrd, is not an easy feat.
Still waiting to hear final confirmation regarding Comment#18 that this isn't really supported, so for now I'm moving this to 5.9 (and will likely close it eventually).
Moving to 5.10, so as not to lose this completely, but it still looks as if this isn't supported.
Based on Comment 18, and the lack of dispute over Ben's assessment, I'm closing this as WONTFIX since it appears we don't support the procedure.