Red Hat Bugzilla – Bug 626964
snapshot11 boot from external sas array but root device is not in multipath listing
Last modified: 2015-09-16 19:46:26 EDT
Created attachment 440728 [details] several system files and commands output Description of problem: root device is on external sas array with multiple paths available. OS was installed and boot up successfully. But multipath -ll is not showing the root device being under multipathd manage. Version-Release number of selected component (if applicable): Snapshot 11 How reproducible: Always Steps to Reproduce: 1. Present one lun from external sas array. During install, select the lun from Multipath Device tab. 2. After installation completed and server boot up, multipath -ll will show nothing. 3. Actual results: Expected results: Additional info: /lib/udev/rules.d/40-multipath is still missing, OPTIONS+="link_priority=5" However, modified 40-multipath still does not help the problem.
The diskby.txt seems to show that sda was configured, but sdb (the second path) was not configured. Please post /var/log/message showing the start-up messages. If there is no sdb, you could try something like: echo "- - -" > /sys/class/scsi_host/host<h>/scan to see if it gets configured on a second try, after the system is up. Post any associated messages for that as well. Ben, please take a look at the config files and see if they are correct.
sdb is configured on the host (see new attachment lsscsi output). I think /dev/disk sub-directories only shows one instance of the device/partitions and is soft-link to just one path of the device. But diskby.txt also shows nothing with dm- name. In contrast, on a working host, one would see something like this lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpatha -> ../../dm-0 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap1 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap2 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap3 -> ../../dm-3 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-mpath-36001438002a56fd60000800000180000 -> ../../dm-0 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part1-mpath-36001438002a56fd60000800000180000 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part2-mpath-36001438002a56fd60000800000180000 -> ../../dm-2 lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part3-mpath-36001438002a56fd60000800000180000 -> ../../dm-3 lrwxrwxrwx 1 root root 9 Aug 19 11:17 scsi-36001438002a56fd60000800000180000 -> ../../sdb lrwxrwxrwx 1 root root 9 Aug 19 11:17 wwn-0x6001438002a56fd60000800000180000 -> ../../sdb I am also attaching the message file.
Created attachment 441542 [details] host messages file
Created attachment 441543 [details] lsscsi output
Looking at the attached multipath.conf, you are blacklisting every device except the ones with the wwid of 3600c0ff000f0043958fbf64901000000 (sda and sdb). So those are the only devices that could be multipathed. Looking at the multipath output, it appears that multipath does think that these devices are valid. however it can't create the multipath device because sda is already in use (as the root device). Whatever problem you are seeing, it's happening in the initramfs, so what is in /etc/multipath.conf is irrelevant. multipath needs to create the multipath device before the root filesystem is mounted. Could you attach your initramfs-2.6.32-63.el6.x86_64.img?
Created attachment 441616 [details] original created during OS installation Argh, attached these earlier but to the wrong bug!! Here they're
Created attachment 441617 [details] new one created afterward For experiment, this new one was created after multipath.conf multipath{} part was commented out.
Well, I played with another server with similar setup and that's working fine. Installed, boot-up, and root device under multipath manage as expected. Let me know if you find anything in the initramfs obvious wrong. Otherwise, I can try install again on the original server.
Hmm somehow there has been no response to needinfo?? I went ahead and tried snapshot13. I have 2 servers and I am installing them to same SAS external array. Each server has just 1 lun presented to it with 2 available paths. I am pretty sure I went the install same way for both servers. At the end 1 server (snowy) came up as expected, multipathd running, boot lun is showing in multipath -ll listing, root directory mounted from /dev/mapper/mpathbp1. But the other server (jersey) is still not coming up right. As before boot lun does not have multipath mapping. The only difference I can see is that jersey is using 3Gb LSI sas adapter (driver mptsas) and snowy is using 6Gb LSI sas adapter (driver mpt2sas). Let me know what info I can provide.
Please respond, product launch depends on resolution to this.
The best guess I have is that multipath might not be building the device because something else has the file open already, either temporarily or permanently. The only other possible issue I see is that multipath.conf sets the alias to be mpathb. The mpathX names are reserved for user_friendly_names. If that alias is already taken by a device with a default user_friendly_name by the time that multipath tries to build the device with the hardcoded alias, it will fail.
Created attachment 448721 [details] new files from snapshot13 Ben, attachment you looked at were from snapshot11. Here are the new ones and multipath.conf is the original one (no change at all), right after snapshot13 install. There is only 1 device from the array presented to the system, the boot lun. So no device should be using mpathb name.
I see the same thing here. multipath is all set up to make the device. However it can't because the device is apparenly in use. If you give me the output of /var/log/messages, from when you run this command, I verify that. But since the root device is obviously mounted, it's pretty clear that the device is already in use. Looking at the initramfs stuff you sent me, multipath should be getting called before the code to mount the root filesystem. If there aren't other devices being mounted, then my best guess is that something is temporarily keeping the device busy. My best idea for debugging this would be to edit the initramfs to print out debug messages.
Created attachment 448981 [details] patch to make multipath try to assemble the maps with debugging on in the initramfs This patches pre-trigger/02multipathd.sh in the initramfs to make multipath run with the verbosity turned up before it starts multipathd. If you manually remake your initramfs with this patch added, you should get multipath to print out what's going on to the console during boot. Can you please reboot with the patch and send me the console messages from during bootup. Hopefully, this will tell me why multipath isn't building the path in the initramfs.
If you have problems remaking the initrd, let me know, and I can either give you some instructions, or if you post the initrd you want patched, I can do it for you.
Ben, I should use the patch to change /usr/share/dracut/modules.d/90multipath/multipathd.sh then make a new initramfs, correct?
Yeah, That should work.
Anthony, Can you try this in RC? thanks, Beth
Is there a reason that this bug can't just be closed as a duplicate of 636668? Was it supposed to be used for something else?
Dup'ing works for me. As Ben recommended, dup'ing to 636668. thanks, Beth *** This bug has been marked as a duplicate of bug 636668 ***
This bug was where 636668 started. Later on Beth dupli to 636668 with limited group/viewing.