RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 626964 - snapshot11 boot from external sas array but root device is not in multipath listing
Summary: snapshot11 boot from external sas array but root device is not in multipath l...
Keywords:
Status: CLOSED DUPLICATE of bug 636668
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: device-mapper-multipath
Version: 6.0
Hardware: All
OS: Linux
low
high
Target Milestone: rc
: 6.1
Assignee: Ben Marzinski
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 563334 564512 580566
TreeView+ depends on / blocked
 
Reported: 2010-08-24 18:45 UTC by Anthony Cheung
Modified: 2015-09-16 23:46 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-11-15 04:48:25 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
several system files and commands output (20.00 KB, application/x-tar)
2010-08-24 18:45 UTC, Anthony Cheung 		no flags Details
host messages file (85.84 KB, text/plain)
2010-08-27 15:22 UTC, Anthony Cheung 		no flags Details
lsscsi output (313 bytes, text/plain)
2010-08-27 15:22 UTC, Anthony Cheung 		no flags Details
original created during OS installation (13.37 MB, application/octet-stream)
2010-08-27 21:33 UTC, Anthony Cheung 		no flags Details
new one created afterward (13.21 MB, application/octet-stream)
2010-08-27 21:35 UTC, Anthony Cheung 		no flags Details
new files from snapshot13 (20.00 KB, application/x-tar)
2010-09-21 15:12 UTC, Anthony Cheung 		no flags Details
patch to make multipath try to assemble the maps with debugging on in the initramfs (430 bytes, patch)
2010-09-22 16:36 UTC, Ben Marzinski 		no flags Details | Diff
View All

Description Anthony Cheung 2010-08-24 18:45:28 UTC 
Created attachment 440728 [details]
several system files and commands output

Description of problem:
root device is on external sas array with multiple paths available.  OS was installed and boot up successfully.  But multipath -ll is not showing the root device being under multipathd manage.


Version-Release number of selected component (if applicable):
Snapshot 11

How reproducible:
Always

Steps to Reproduce:
1. Present one lun from external sas array. During install, select the lun from Multipath Device tab.
2. After installation completed and server boot up, multipath -ll will show nothing.
3.
  
Actual results:


Expected results:


Additional info:
/lib/udev/rules.d/40-multipath is still missing, OPTIONS+="link_priority=5"

However, modified 40-multipath still does not help the problem.
Comment 2 Tom Coughlan 2010-08-27 13:02:56 UTC 
The diskby.txt seems to show that sda was configured, but sdb (the second path) was not configured. 

Please post /var/log/message showing the start-up messages. 

If there is no sdb, you could try something like:

echo "- - -" > /sys/class/scsi_host/host<h>/scan

to see if it gets configured on a second try, after the system is up. Post any associated messages for that as well. 

Ben, please take a look at the config files and see if they are correct.
Comment 3 Anthony Cheung 2010-08-27 15:21:35 UTC 
sdb is configured on the host (see new attachment lsscsi output).  I think /dev/disk sub-directories only shows one instance of the device/partitions and is soft-link to just one path of the device.  But diskby.txt also shows nothing with dm- name.  In contrast, on a working host, one would see something like this
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpatha -> ../../dm-0
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap1 -> ../../dm-1
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap2 -> ../../dm-2
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-name-mpathap3 -> ../../dm-3
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-mpath-36001438002a56fd60000800000180000 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part1-mpath-36001438002a56fd60000800000180000 -> ../../dm-1
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part2-mpath-36001438002a56fd60000800000180000 -> ../../dm-2
lrwxrwxrwx 1 root root 10 Aug 19 11:17 dm-uuid-part3-mpath-36001438002a56fd60000800000180000 -> ../../dm-3
lrwxrwxrwx 1 root root  9 Aug 19 11:17 scsi-36001438002a56fd60000800000180000 -> ../../sdb
lrwxrwxrwx 1 root root  9 Aug 19 11:17 wwn-0x6001438002a56fd60000800000180000 -> ../../sdb

I am also attaching the message file.
Comment 4 Anthony Cheung 2010-08-27 15:22:04 UTC 
Created attachment 441542 [details]
host messages file
Comment 5 Anthony Cheung 2010-08-27 15:22:33 UTC 
Created attachment 441543 [details]
lsscsi output
Comment 6 Ben Marzinski 2010-08-27 16:49:29 UTC 
Looking at the attached multipath.conf, you are blacklisting every device except the ones with the wwid of 3600c0ff000f0043958fbf64901000000 (sda and sdb).  So those are the only devices that could be multipathed.  Looking at the multipath output, it appears that multipath does think that these devices are valid. however it can't create the multipath device because sda is already in use (as the root device).  Whatever problem you are seeing, it's happening in the initramfs, so what is in /etc/multipath.conf is irrelevant.  multipath needs to create the multipath device before the root filesystem is mounted. Could you attach your initramfs-2.6.32-63.el6.x86_64.img?
Comment 7 Anthony Cheung 2010-08-27 21:33:36 UTC 
Created attachment 441616 [details]
original created during OS installation

Argh, attached these earlier but to the wrong bug!!  Here they're
Comment 8 Anthony Cheung 2010-08-27 21:35:29 UTC 
Created attachment 441617 [details]
new one created afterward

For experiment, this new one was created after multipath.conf multipath{} part was commented out.
Comment 9 Anthony Cheung 2010-08-27 23:58:46 UTC 
Well, I played with another server with similar setup and that's working fine.  Installed, boot-up, and root device under multipath manage as expected.

Let me know if you find anything in the initramfs obvious wrong.  Otherwise, I can try install again on the original server.
Comment 10 Anthony Cheung 2010-09-10 17:46:08 UTC 
Hmm somehow there has been no response to needinfo??

I went ahead and tried snapshot13.  I have 2 servers and I am installing them to same SAS external array.  Each server has just 1 lun presented to it with 2 available paths.  I am pretty sure I went the install same way for both servers.  At the end 1 server (snowy) came up as expected, multipathd running, boot lun is showing in multipath -ll listing, root directory mounted from /dev/mapper/mpathbp1.  But the other server (jersey) is still not coming up right.  As before boot lun does not have multipath mapping.  The only difference I can see is that jersey is using 3Gb LSI sas adapter (driver mptsas) and snowy is using 6Gb LSI sas adapter (driver mpt2sas).

Let me know what info I can provide.
Comment 11 Paul Hinchman 2010-09-17 21:09:16 UTC 
Please respond, product launch depends on resolution to this.
Comment 12 Ben Marzinski 2010-09-20 17:43:49 UTC 
The best guess I have is that multipath might not be building the device because something else has the file open already, either temporarily or permanently.

The only other possible issue I see is that multipath.conf sets the alias to be mpathb.  The mpathX names are reserved for user_friendly_names.  If that alias is already taken by a device with a default user_friendly_name by the time that multipath tries to build the device with the hardcoded alias, it will fail.
Comment 13 Anthony Cheung 2010-09-21 15:12:46 UTC 
Created attachment 448721 [details]
new files from snapshot13

Ben, attachment you looked at were from snapshot11.  Here are the new ones and multipath.conf is the original one (no change at all), right after snapshot13 install.  There is only 1 device from the array presented to the system, the boot lun.  So no device should be using mpathb name.
Comment 14 Ben Marzinski 2010-09-22 16:29:45 UTC 
I see the same thing here. multipath is all set up to make the device. However it can't because the device is apparenly in use.  If you give me the output of /var/log/messages, from when you run this command, I verify that. But since the root device is obviously mounted, it's pretty clear that the device is already in use.  Looking at the initramfs stuff you sent me, multipath should be getting called before the code to mount the root filesystem.  If there aren't other devices being mounted, then my best guess is that something is temporarily keeping the device busy.

My best idea for debugging this would be to edit the initramfs to print out debug messages.
Comment 15 Ben Marzinski 2010-09-22 16:36:10 UTC 
Created attachment 448981 [details]
patch to make multipath try to assemble the maps with debugging on in the initramfs

This patches pre-trigger/02multipathd.sh in the initramfs to make multipath run with the verbosity turned up before it starts multipathd.  If you manually remake your initramfs with this patch added, you should get multipath to print out what's going on to the console during boot.

Can you please reboot with the patch and send me the console messages from during bootup. Hopefully, this will tell me why multipath isn't building the path in the initramfs.
Comment 16 Ben Marzinski 2010-09-22 17:00:41 UTC 
If you have problems remaking the initrd, let me know, and I can either give you some instructions, or if you post the initrd you want patched, I can do it for you.
Comment 17 Anthony Cheung 2010-09-22 17:55:03 UTC 
Ben, I should use the patch to change /usr/share/dracut/modules.d/90multipath/multipathd.sh then make a new initramfs, correct?
Comment 18 Ben Marzinski 2010-09-22 19:50:16 UTC 
Yeah, That should work.
Comment 19 Beth Zeranski 2010-09-22 20:08:35 UTC 
Anthony,
Can you try this in RC?

thanks,
 Beth
Comment 20 Ben Marzinski 2010-11-15 04:04:53 UTC 
Is there a reason that this bug can't just be closed as a duplicate of 636668? Was it supposed to be used for something else?
Comment 21 Beth Zeranski 2010-11-15 04:48:25 UTC 
Dup'ing works for me. As Ben recommended, dup'ing to 636668.

thanks,
 Beth

*** This bug has been marked as a duplicate of bug 636668 ***
Comment 22 Anthony Cheung 2010-11-15 16:07:28 UTC 
This bug was where 636668 started.  Later on Beth dupli to 636668 with limited group/viewing.
Note You need to log in before you can comment on or make changes to this bug.