Created attachment 454226 [details]
console capture of call traces

Created attachment 454228 [details]
multipath.conf used

(In reply to comment #0)
> Description of problem:
> Server is running RHEL6 RC1 2.6.32-71.el6.x86_64.  It is booting from external
> SAS array after overcoming bug #636668.

Looking at bug #636668 it doesn't appear you've completely overcome the issues.
 
> More luns were presented from the array to the server but after reboot the
> server crashed in middle of boot.

There isn't a crash; I'm seeing a jbd2 hang (presumably of the root fs).

Do you consistently see this jbd2 hang after successfully booting, then adding more luns and then rebooting?

Given the multipath.conf you've provided you're using queue_if_no_path after 18 retries (via 'no_path_retry 18').  That could be one explanation for the jbd2 hangs on the mpath device (provided all paths were lost).  But I'm not seeing any messages about paths failing...

There are certainly some concerning things in the kernel log (from comment#2), e.g. the various "dm-X too small for target".  Could just be a whole lot of noise but in general we're not seeing a clean bring up here.

I verified there is no path issues before I filed the bug.

I have 2 sizes of LUNs on the array, actually 3 the root lun is 14Gb by itself.  I presented a bunch of 8.5Gb LUNs, all have 2 paths, and the server was fine.  After presented some 1Gb LUNs, the server starts to hang.  Unpresent these 1Gb LUNs, the server was good.

(In reply to comment #5)

> After presented some 1Gb LUNs, the server starts to hang.  Unpresent these 1Gb
> LUNs, the server was good.

I'm missing some detail here: what is the nature of the hang?  Is the root FS having problems like was originally reported with the jbd2 hang?

The system was not rebooted after presenting the 1Gb LUNs?

Are the 8.5GB LUNs and the 1GB LUNs being provisioned from the same storage backend?

By unpresenting the LUNs in the storage backend the hang just went away (again without reboot)?  We need to isolate exactly what is hanging (multipathd?, the root FS?, etc.)

Could you provide logs from the time where you:
1) expose the 1GB luns
2) see the hang
3) unpresent the 1GB luns

It would also be useful to get traces for all processes while you're seeing the hang in step 2 (via: echo t > /proc/sysrq-trigger)

Yes, both the 8.5GB and 1GB LUNs are presented from same array.

Reported originally, 1GB LUNs were presented to the server.  No online scanning or discovery was executed from the server.  Just reboot the server.  During boot up, hang on jbd2 happened and the server was never able to boot up completely.  Unpresent 1GB LUNs, power-cycle the server and it booted up fine.

That was happened month ago.  Now the server was re-installed and running RC2/GA bit 2.6.32-71.el6.x86_64.  It doesn't fail the same way as I just described.  

Now the server doesn't hang during the boot up process when 1GB LUN is presented.  It can boot up completely but it will hang on some commands like vgdisplay, partprobe.  Again, if the 1GB LUNs were unpresented and reboot the server (because there is command hanging), commands like vgdisplay or partprobe will succeed.

/var/log/messages to be attached with Call Traces and sysrq as you requested

Created attachment 464080 [details]
/var/log/messages with Call Traces and SysRq

I think I may have figured out the problem.  Again, it's related to multipath bindings file in initramfs (since this server is booting from SAN) as discussed in bug #636668.

The server(jersey) initramfs bindings file had only 1 lun mapping (mpatha, the boot/root lun).  As more luns are presented bindings file in /etc/multipath directory was updated accordingly but obviously not the one in initramfs image.  Because of this, every time the server is rebooted, with exception of the boot lun mapping, other luns mapping may come up different from last time.  Even worse is, mappings actually come up mingled and hence dmesg showing messages "dm-X too small for target" you mentioned in comment #4 above.  Whenever "too small" messages occurred, commands like vgdisplay, partprobe will hang.

As can be seen below, len or start does not even make sense for dev_size because it got length of partition from different device.
device-mapper: table: 253:11: dm-10 too small for target: start=1, len=17498916, dev_size=1953120
device-mapper: table: 253:12: dm-10 too small for target: start=17498917, len=79179, dev_size=1953120

I will attached few files which captured some data and the file confuse.txt gives some of annotation.

Created attachment 467319 [details]
various files capture to show how mapping mangled and partition size wrong

Mike,
Will the fix from 636668 fix this or will this bz require a different patch?

thanks,
 Beth

(In reply to comment #12)
> Mike,
> Will the fix from 636668 fix this or will this bz require a different patch?

No, seems we still need a different patch given this comment from Anthony:
https://bugzilla.redhat.com/show_bug.cgi?id=636668#c51

Anthony,
Once you present these additional LUNs (and the bindings file between the initramfs and the system become out of sync) what is being done to these LUNs?  I assume that when you add these LUNs you're discovering the new LUNs rather than rebooting (and multipath is automatically creating mpath devices as soon as multipathd sees these devices)?

Are you creating LVM volumes on the associated mpath devices?  Anything else of note about the steps to reproduce?

And are you always seeing "dm-X too small for target" messages -- you noted in a previous comment that this was the tell tale sign that lvm commands would hang (if attempted).

If you'd be willing to offer a concise sequence of steps to reproduce that would be quite helpful (sorry to possibly make you repeat yourself but I'm wondering if you have new clarity on the sequence given how many different BZs have been a product of your use-case).  Seems this is your only remaining bug though.

Mike,

1). Server has 2 paths to array.  1 LUN is presented to the server for boot lun purpose.  Complete installation of RHEL6 to the single multipath LUN and let it boot.

2). Present additional LUNs from the array.  Doesn't matter they are discovered by command or reboot.  Either way, mpath devices were created and bindings in the system was updated.  And obviously at this point bindings files became out-of-sync.

3). Reboot server number of times after. Sometimes, it will boot to the situation of the problem.

I don't have any LVM volumes on LUNs. I was just trying to see hang happens with different commands.  LUNs do have 2 partitions and they are different sizes.  That's how it caught our attention of "too small target" messages otherwise, it may not be obvious mappings were messed up.

I still have couple open bugs that are happening during I/O load test.

(In reply to comment #14)
> Mike,
> 
> 1). Server has 2 paths to array.  1 LUN is presented to the server for boot lun
> purpose.  Complete installation of RHEL6 to the single multipath LUN and let it
> boot.
> 
> 2). Present additional LUNs from the array.  Doesn't matter they are discovered
> by command or reboot.  Either way, mpath devices were created and bindings in
> the system was updated.  And obviously at this point bindings files became
> out-of-sync.

I've tried this and I've found a couple issues with 6.0:
. anaconda doesn't copy the bindings file to /etc/multipath/
  - matches https://bugzilla.redhat.com/show_bug.cgi?id=636668#c41
. when I add an additional mpath LUN multipath creates the bindings file
  - but the new LUN takes the device name that was already in use by the mpath root device

# multipath -ll
mpatha (20090ef1270000000) dm-0 IQSTOR,iQ2880
...

# multipath -v2
Jan 20 14:34:54 | mpatha: remove (wwid changed)
Jan 20 14:34:54 | mpatha: map in use

# cat /etc/multipath/bindings 
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpatha 20090ef1270000011

Could be that this is a device-mapper-multipath bug that has been resolved in a newer release?

But I'm left with a system that:
1) doesn't have a bindings in the initramfs
   - due to dracut not having saved it during anaconda's install, AFAIK, bug#630911
2) has a bindings on the system disk that collides with the root mpatha device

> 3). Reboot server number of times after. Sometimes, it will boot to the
> situation of the problem.
> 
> I don't have any LVM volumes on LUNs. I was just trying to see hang happens
> with different commands.  LUNs do have 2 partitions and they are different
> sizes.  That's how it caught our attention of "too small target" messages
> otherwise, it may not be obvious mappings were messed up.
> 
> I still have couple open bugs that are happening during I/O load test.

After reboot I didn't have any problems with "too small target" messages.

But mpathb was not initailized because of the system bindings having an entry for the new LUN that collides with the root mpatha device.

I manually added mpatha and mpathb with appropriate bindings entries and rebuilt the initramfs with:
# dracut -f -a multipath --install /etc/multipath/bindings

same as was advised here:
https://bugzilla.redhat.com/show_bug.cgi?id=636668#c38

After reboot dracut could _not_ find the root device.  I then rebooted and used the saved original initramfs and it booted successfully.

I retried the new initramfs and dracut was able to find the root device that time.

All this is with dracut-004-35.el6.noarch

When the system fails to boot (which is almost always the case) I see:

dracut: Couldn't find device with uuid M5gmaI-NHuW-I4to-jEY2-BUCD-oJyW-8eN2AS.
dracut: Couldn't find device with uuid bY5mw8-qbzs-zPN2-rUnq-WbYN-cbXi-20TJED.
dracut: Couldn't find device with uuid cmVJMG-qwBF-5H54-F0Dd-0ijO-YSYW-PR0Na7.
dracut: Couldn't find device with uuid hXVfPc-x176-zIhH-kb20-QLSr-HFDx-wneEaH.
dracut: Couldn't find device with uuid r7gsp4-Q7CC-9508-FFFO-EANe-LLkv-m9j6EC.
dracut: Couldn't find device with uuid YzdIcn-zT6W-3lDI-Et3B-N4f0-9kPP-MpeVUI.
dracut: Couldn't find device with uuid XE99Tg-BaMx-bv60-3Ou0-GAx5-Tu1p-TM7TgM.
dracut: Couldn't find device with uuid Dgq4HP-xDbv-Zevd-qFkS-cUVt-v4IJ-bRhjNm.
dracut: Couldn't find device with uuid SytZGe-crsx-Gh4b-NT8l-I3FV-jnKr-fJZY2u.
dracut: Couldn't find device with uuid PEkm0n-RFsV-rzxF-Z1A9-8Usf-9J9n-6et5hu.
dracut: Couldn't find device with uuid fYX4Mv-fqZp-0WxM-3nqD-8NCy-6QmX-MymiI6.
dracut: Couldn't find device with uuid x0GQ2v-1kCC-4w6y-mszg-uxIR-XS4e-ceBzx4.
dracut: Couldn't find device with uuid KCFtrF-c2xU-hC5p-50la-8sHs-9f2C-N1v87S.
dracut: Couldn't find device with uuid D7fI6l-JTpE-yHkj-kUZr-aXEI-y42X-And1e6.
dracut: inactive '/dev/vg_storageqe06/lv_root' [50.00 GiB] inherit
dracut: inactive '/dev/vg_storageqe06/lv_home' [80.41 GiB] inherit
dracut: inactive '/dev/vg_storageqe06/lv_swap' [3.91 GiB] inherit
dracut: Found duplicate PV dEFhN1N2qn6ON080XOdhVRd4hJ5CM6IE: using /dev/sde1 not /dev/sdb1
dracut: Found duplicate PV dEFhN1N2qn6ON080XOdhVRd4hJ5CM6IE: using /dev/sdh1 not /dev/sde1
dracut: Found duplicate PV dEFhN1N2qn6ON080XOdhVRd4hJ5CM6IE: using /dev/sdk1 not /dev/sdh1
dracut: Couldn't find device with uuid p1f4dL-Wy7C-VTbz-LRnP-uDt7-ic0r-PKj7Rc.
dracut: Couldn't find device with uuid buaBWX-Rsr4-5l50-kQOl-aun1-DmQg-iLc5aR.

dracut: Couldn't find device with uuid M5gmaI-NHuW-I4to-jEY2-BUCD-oJyW-8eN2AS.
dracut: Couldn't find device with uuid bY5mw8-qbzs-zPN2-rUnq-WbYN-cbXi-20TJED.
dracut: Couldn't find device with uuid cmVJMG-qwBF-5H54-F0Dd-0ijO-YSYW-PR0Na7.
dracut: Couldn't find device with uuid hXVfPc-x176-zIhH-kb20-QLSr-HFDx-wneEaH.
dracut: Couldn't find device with uuid r7gsp4-Q7CC-9508-FFFO-EANe-LLkv-m9j6EC.
dracut: Couldn't find device with uuid YzdIcn-zT6W-3lDI-Et3B-N4f0-9kPP-MpeVUI.
dracut: Couldn't find device with uuid XE99Tg-BaMx-bv60-3Ou0-GAx5-Tu1p-TM7TgM.
dracut: Couldn't find device with uuid Dgq4HP-xDbv-Zevd-qFkS-cUVt-v4IJ-bRhjNm.
dracut: Couldn't find device with uuid SytZGe-crsx-Gh4b-NT8l-I3FV-jnKr-fJZY2u.
dracut: Couldn't find device with uuid PEkm0n-RFsV-rzxF-Z1A9-8Usf-9J9n-6et5hu.
dracut: Couldn't find device with uuid fYX4Mv-fqZp-0WxM-3nqD-8NCy-6QmX-MymiI6.
dracut: Couldn't find device with uuid x0GQ2v-1kCC-4w6y-mszg-uxIR-XS4e-ceBzx4.
dracut: Couldn't find device with uuid KCFtrF-c2xU-hC5p-50la-8sHs-9f2C-N1v87S.
dracut: Couldn't find device with uuid D7fI6l-JTpE-yHkj-kUZr-aXEI-y42X-And1e6.
dracut: Refusing activation of partial LV lv_root. Use --partial to override.
dracut: Refusing activation of partial LV lv_swap. Use --partial to override.


No root device found
dracut Warning: LVM vg_storageqe06/lv_root not found
dracut Warning: LVM vg_storageqe06/lv_swap not found

Boot has failed, sleeping forever.


Now I can't reproduce the successful boot with the new initramfs after 5 boot attempts...

I was able to get this working with the updated bindings file by removeing the "uid", "gid", and "mode" lines from the multipaths section.  I'm not sure why this makes a difference yet.  It's possible that a device node was created to deal with a dmsetup call by a script. This device node wouldn't have the correct mode.  In rhel5, this issue caused a problem where multipath device nodes wouldn't get created with the proper modes.  However, what that has to do with user_friendly_names is a mystery.  I can also make the problem go away simply by removing the bindings file from the initramfs.

The problem with the bindings getting messed up when additional LUNs are presented should get fixed by the patch referred to in Bug #649419.

I've been able to get the hang that mike reproduced with entire multipaths section commented out. This is looking more like a race to me. Also There were two VGs with the name vg_storageqe06, one of them missing most of it's devices. Once I cleared the rest of the rest of the devices from this partial VG, I've been completely unable to see the issue from Comment 16 and 17.  I have been able to
reproduce the "too small target" errors and verify that they are caused by the
luns getting mixed up between the two bindings files.  This will be solved by Bug #649419. Since that bug doesn't have flags set for RHEL-6.1, I'm going to fix it in this one.

Would it be possible to open this bug up, so that I could dup #649419 to this one?

Oops. I just noticed I was referencing the wrong bugzilla. It should be Bug #669419

@Ben, you're welcome to open this bug up.  As a matter, if possible, I would like to see what is bug #669419 and the fix you mentioned.  Since this "bindings file out-of-sync" seem to be root-cause or partial cause to few bugs we filed.  And it's happening with RHEL5.6 also bug #668968

O.k. you can pick up packages at:

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/x86_64/

or

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/i686/

What these do is add a -B option to multipath and multipathd.  When you run with that option, the bindings file becomes read-only, and if a device doesn't have a binding in there, it defaults back to using the wwid for the name. To make this
work, also need to edit /usr/share/dracut/modules.d/90multipath/multipathd.sh
and replace

    multipathd

with

    multipathd -B
 
And remake your initramfs

# dracut -a multipath --include /etc/multipath /etc/multipath

Note, this won't save you if your multipath.conf file sets the root device's alias to mpath<x>.  You need to either remove the alias line, or change it to something else.

I'm thinking of reworking this to allow you to completely disable user_friendly_names with a command line option.  This would make all of your initial devices come up with wwid names, which would safely be renamed when
multipathd is started later in boot. Also, I'd like to make multipath ignore
any multipaths alias that's set to mpath<x>.  As long as nothing is set up to
rely on the device name, this will fix the problem without needing anything from anaconda.  In all the installs I've looked at, I don't see anything that does.
They all go off of the UUID or the PV label, but I'm not sure that there isn't a way to setup the machine on install that would require the device name to stay the same. If there is, then this change would need to be coordinated with anaconda, so that they don't keep assigning these names.

*** Bug 669419 has been marked as a duplicate of this bug. ***

The problem with ignoring the conflicting aliases is that it has the potential to cause regressions on working systems that rely on the device name.  In the end, I decided to leave the fix as it is in the test packages.

@Ben, I tried your packages and direction from comment #21 and for the "too small for target" problem original encountered that seems to be resolved.

However I am trying to understand what you're describing the fix.  Is it correct with this patch, it is not necessary to have the 2 bindings (initramfs copy and /etc/multipath copy) to be in-sync anymore?  As you know this bindings out-of-sync issue have popped up in quite a few different bugzilla.  And it's quite usability issue for our storage customers if they have to re-make a initramfs every time they make lun presentation change.

@RedHat with today released RHEL6.0 media, if users is not using multipath on their boot lun, it is fairly straight forward for them after RHEL6.0 install: create/edit their copy of multipath.conf, present and/or unpresent luns as wish and /etc/multipath/bindings will take care of itself.  I haven’t have a server today not using multipath on boot lun so I assume this is the case, correct assessment?

However, if users is using multipath on their boot lun with RHEL6.0 today, 
-	ananconda will by default create /etc/multipath.conf with alias for the boot lun
-	after fresh install /etc/multipath/bindings does not exist and copy of that does not exist in initramfs either
-	2 bindings files out-of-sync issue
-	Multipath patches
-	Dracut patches
You got the ideas.  It’s not straight forward to set thing up right.  Particularly when comparing to RHEL5 how easy it’s to setup multipath on boot lun.  I hope these cumbersome would be resolved in RHEL6.1.  But for RHEL6.0 it will be beneficial for RedHat to come up with short documentation to step user through how to properly finishing setting multipath on boot lun after installation.

For RHEL 6.1 this should be pain-free.  When you make update your initramfs, it will pull in the existing bindings, so at that time they should match.  After that, unless a user manually edits the bindings file, everything should be o.k.  If new devices are added to the system, they won't get user_friendly_names in early boot, and they get renamed later in the boot process.

For RHEL-6.0, since the fixes are not available in a zstream, users must rerun

# dracut -a multipath --include /etc/multipath /etc/multipath

whenever they add new storage, and in some cases, after the first bootup.

------- Comment From finnegan.com 2011-02-10 13:38 EDT-------
Do we have an update from Red Hat on this Bugzilla ?

------- Comment From finnegan.com 2011-02-14 12:23 EDT-------
In response to
>fixed in: device-mapper-multipath-0.4.9-36.el6
Which version of RHEL6 will device-mapper-multipath-0.4.9-36.el6 be released on

Raising the severity from normal to high so we have a patch available for current installations running RHEL6 GA levels.

Red Hat - Please update severity on your side.

Created attachment 487904 [details]
multipath -l and multipath -ll -v3

Created attachment 487905 [details]
sosreport attached


------- Comment (attachment only) From finnegan.com 2011-01-13 13:47 EDT-------

Hello,
This is currently accepted for 6.1.

Thank You
Joe Kachuck

~~ Partners and Customers ~~

This bug was included in RHEL 6.1 Beta. Please confirm the status of this request as soon as possible.

If you're having problems accessing 6.1 bits, are delayed in your test execution or find in testing that the request was not addressed adequately, please let us know.

Thanks!

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If the initramfs file system was not rebuilt when a new storage device was added to the system, the new device could have been assigned a user_friendly_names value that matched the user_friendly_names value already-assigned to another device. This device then stopped working correctly. The multipathd deamon now accepts a -B option, which makes the user_friendly_names bindings file read-only. When initramfs calls multipath with the -B option, devices without a binding to a user_friendly_names use their World Wide Identifier (WWID).

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0725.html