Bug 274851 - openib: openmpi crashes on ia64 platform.
openib: openmpi crashes on ia64 platform.
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: redhat-release-notes (Show other bugs)
ia64 Linux
medium Severity medium
: ---
: ---
Assigned To: Doug Ledford
: Documentation, Reopened
: 235946 (view as bug list)
Depends On:
Blocks: 222082 RHEL5u2_relnotes
  Show dependency treegraph
Reported: 2007-09-03 00:42 EDT by Gurhan Ozen
Modified: 2013-11-03 20:33 EST (History)
2 users (show)

See Also:
Fixed In Version: 5.2 errata
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-06-14 11:34:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Gurhan Ozen 2007-09-03 00:42:40 EDT
Description of problem:
  Openmpi programs are not runnable on ia64 platform due to random crashes:
# mpirun -np 2 --mca btl openib,self /usr/bin/mpitests-osu_bw 
# OSU MPI Bandwidth Test (Version 2.2)
# Size          Bandwidth (MB/s) 
[intel-s6e4533-01-mm:28875] *** Process received signal ***
[intel-s6e4533-01-mm:28875] Signal: Segmentation fault (11)
[intel-s6e4533-01-mm:28875] Signal code: Invalid permissions (2)
[intel-s6e4533-01-mm:28875] Failing at address: (nil)
[intel-s6e4533-01-mm:28875] [ 0] [0xa0000000000107e0]
[intel-s6e4533-01-mm:28875] [ 1]
/usr/lib/openmpi/1.2.3-gcc/openmpi/mca_btl_openib.so [0x2000000000c08e31]
[intel-s6e4533-01-mm:28875] [ 2]
[intel-s6e4533-01-mm:28875] [ 3]
[intel-s6e4533-01-mm:28875] [ 4]
[intel-s6e4533-01-mm:28875] [ 5]
/usr/lib/openmpi/1.2.3-gcc/libmpi.so.0(PMPI_Waitall-0x5f5d10) [0x2000000000138f10]
[intel-s6e4533-01-mm:28875] [ 6]
/usr/bin/mpitests-osu_bw(main+0x1fffffffff8d7b90) [0x40000000000019c0]
[intel-s6e4533-01-mm:28875] [ 7] /lib/libc.so.6.1(__libc_start_main-0x287540)
[intel-s6e4533-01-mm:28875] [ 8]
/usr/bin/mpitests-osu_bw(_start+0x1fffffffff8d23c0) [0x4000000000001000]
[intel-s6e4533-01-mm:28875] *** End of error message ***
mpirun noticed that job rank 0 with PID 28874 on node
intel-s6e4533-01-mm.rhts.boston.redhat.com exited on signal 15 (Terminated). 
1 additional process aborted (not shown)

Above was retrieved when running the program only on 1 node. When running ia64
as the master node execution crashed with the following error:

error posting send request errno says Invalid argument

error posting send request errno says Invalid argument

When running as ia64 remote node, the following happens:
[intel-s6e4533-01-mm:28971] *** Process received signal ***
[intel-s6e4533-01-mm:28971] Signal: Segmentation fault (11)
[intel-s6e4533-01-mm:28971] Signal code: Invalid permissions (2)
[intel-s6e4533-01-mm:28971] Failing at address: (nil)
[intel-s6e4533-01-mm:28971] [ 0] [0xa0000000000107e0]
[intel-s6e4533-01-mm:28971] [ 1]
[intel-s6e4533-01-mm:28971] [ 2]
/usr/lib/openmpi/1.2.3-gcc/openmpi/mca_btl_openib.so [0x2000000000c06c00]
[intel-s6e4533-01-mm:28971] [ 3]
/usr/lib/openmpi/1.2.3-gcc/openmpi/mca_btl_openib.so [0x2000000000c09180]
[intel-s6e4533-01-mm:28971] [ 4]
[intel-s6e4533-01-mm:28971] [ 5]
[intel-s6e4533-01-mm:28971] [ 6]
[intel-s6e4533-01-mm:28971] [ 7]
/usr/lib/openmpi/1.2.3-gcc/libmpi.so.0(PMPI_Waitall-0x5f5d10) [0x2000000000138f10]
[intel-s6e4533-01-mm:28971] [ 8]
/usr/bin/mpitests-osu_bw(main+0x1fffffffff8d7b90) [0x40000000000019c0]
[intel-s6e4533-01-mm:28971] [ 9] /lib/libc.so.6.1(__libc_start_main-0x287540)
[intel-s6e4533-01-mm:28971] [10]
/usr/bin/mpitests-osu_bw(_start+0x1fffffffff8d23c0) [0x4000000000001000]
[intel-s6e4533-01-mm:28971] *** End of error message ***
mpirun noticed that job rank 0 with PID 19184 on node
dell-pe1950-03.rhts.boston.redhat.com exited on signal 15 (Terminated).

Version-Release number of selected component (if applicable):
# uname -a
Linux intel-s6e4533-01-mm.rhts.boston.redhat.com 2.6.18-43.el5 #1 SMP Tue Aug 21
17:52:47 EDT 2007 ia64 ia64 ia64 GNU/Linux
# rpm -qa | egrep "openib|openmpi"

How reproducible:

Steps to Reproduce:
1. Try to run an openmpi program when 1 or more nodes is ia64
Actual results:

Expected results:

Additional info:
Comment 1 Doug Ledford 2007-09-19 03:08:47 EDT
The observed error scenarios are:

1) When running on ia64 as master, receive retry count exceeded error.  This is
not an openmpi error, but a physical link error caused either by a bad driver,
bad core IB code, or bad hardware.  We have tried two different cables and two
different IB cards that used two totally different drivers (ib_mthca and
ib_ipath) and two different ports on the switch and the hardware errors persist.
 This indicates that this specific failure mode may have something to do with
the core IB stack.  The less likely situation is that both cards or both ports
on the switch that we tried are suspect.  Of note is that after resetting the IB
switch to clear all the error counters, then restarting an RDMA transfer from
the ia64 machine to one of the x86_64 machines, the symbol error counter on the
switch port connected to the ia64 machine reached its limit of 65535 errors
within seconds (the default alarm threshold for this error counter is only 100).
 Of further note is that the error only occurs on ia64 (our only big endian
machine in the test fabric, as well as the arch with the highest level of
unordered writes in the test fabric).  All the other machines produce no errors.

2) When running on ia64 as master, segfault in the openmpi code related to
callback handling on completion.  This situation represents a "can't happen"
situation in the openmpi code.  Specifically, a request completes without the
appropriate callback handler pointer set in the request structure
(frag.base.des_cbfunc == NULL).

3) When running on something other than ia64 as master, error posting send
request errno says Invalid argument (the initial bug report had this attributed
to running on ia64 as master, which has not been seen and does not make sense
for ia64 as master).  This is caused when another node in the mpirun exits
abnormally, causing the connection to be closed while reads/writes are still
pending.  It is to be expected on any node if the ia64 remote node crashed. 
This is consistent with the observed behavior on ia64 and merely indicates that
likely one of the two preceding errors has occurred on the ia64 node.

Items of note:  Both problems #1 and #2 have been verified with both the RHEL5.1
and RHEL5.0 openmpi stacks.  This indicates that the problem is likely not in
openmpi, but instead somewhere lower down in the stack.  Next on the agenda is
to test the RHEL5.0 libibverbs user space drivers (the openmpi stack relies on
the libibvers stack, which interfaces directly with the kernel stack).  Software
has been installed to test the RHEL5.0 kernel in isolation as well.  However,
the testing of the libibverbs RHEL5.0 user space driver and the kernel will have
to wait until someone is in the Westford office that can reset the test machine
as it didn't successfully return from a reboot (in addition to needing a cable
that is currently disconnected connected).
Comment 3 Doug Ledford 2007-09-19 17:11:06 EDT
The crashes turn out to not be related to openmpi at all, it's an openib
(libibverbs) bug.  Switching component.
Comment 4 Doug Ledford 2007-09-19 17:49:52 EDT
OK, there are two distinctly different aspects to this issue:

1) When using ipath on ia64, openmpi breaks.  This isn't a libibverbs issue,
this is an ipath driver issue.  The ipath driver is not safe for use on ia64 and
needs to be called out as tech preview and unsupported in the kernel release
notes for 5.1.

2) There is a bug in libibverbs that can result in sporadic memory corruption,
and I believe this is the cause of the segfaults when running openmpi on mthca
hardware on ia64 and talking to ipath hardware on x86_64.  When talking to mthca
hardware on x86_64, openmpi doesn't have any problems.  The ipath driver handles
memory differently than the mthca, and therefore has a different pattern for
triggering the libibverbs bug than the mthca driver does.  It is this
difference, I think, that is causing the ipath/x86_64 to mthca/ia64 issue.  I'm
building a test version of libibverbs through brew right now to see if it does
indeed resolve the lingering issue.

It is worth noting that bug #2 in this is something that can hit any application
given the right memory usage pattern.  The fix for this has been tested
upstream, and the reproducer program they used to trigger the bug showed that
the bug hit i386 easily, while being very difficult to trigger on x86_64.  I'm
guessing that ia64 is easier to trigger than x86_64 as well and that's why we've
been seeing the crashes there instead of on all the machines.  The patch I'm
using was tested by the upstream author and resolved the issue, including that
the reproducer programs were no longer able to reproduce the problem.  Since
this bug is legitimate and fairly severe, I would request that the blocker
status be granted so I can get the fix checked into CVS and an official build made.
Comment 5 Doug Ledford 2007-09-19 17:54:28 EDT
*** Bug 235946 has been marked as a duplicate of this bug. ***
Comment 6 Don Domingo 2007-09-19 20:14:35 EDT
adding to release notes updates:

(ia64) The ipath driver is not safe for use in this architecture, as its use may
result in errors while using openmpi. As such, the ipath driver is currently
released for this architecture as Technology Preview.

please advise if any further revisions are required. thanks!
Comment 9 Don Domingo 2008-02-19 22:45:16 EST
adding same release note to "Known Issues" of RHEL5.2. please advise if openmpi
is now fully supported so we can document as such. thanks!
Comment 10 Doug Ledford 2008-04-01 17:40:39 EDT
rhel5.2 does not need any notes and is working properly
Comment 11 Don Domingo 2008-04-01 19:12:51 EDT
thanks. adding the following to "Resolved Issues" of RHEl5.2 release notes:

(ia64) Using the ipath driver in this architecture no longer results in openmpi

please advise if any further revisions are required. also, i assume that ipath
in ia64 is now fully supported (i.e. no longer tech preview)?

(In reply to comment #10)
> rhel5.2 does not need any notes and is working properly

Comment 12 Doug Ledford 2008-04-01 19:46:04 EDT
No, no...ipath is still not supported on ia64, but at this point we don't think
it *should* be supported on ia64.  This issue cropped up before upstream ipath
driver authors told us that they hadn't even tested ipath on ia64 and didn't
consider it a priority.  So, now that we know that, we know better than to use
ipath on ia64.
Comment 13 Don Domingo 2008-04-01 22:10:38 EDT
the RHEL5.2 release notes will be dropped to translation on April 15, 2008, at
which point no further additions or revisions will be entertained.

a mockup of the RHEL5.2 release notes can be viewed at the following link:

please use the aforementioned link to verify if your bugzilla is already in the
release notes (if it needs to be). each item in the release notes contains a
link to its original bug; as such, you can search through the release notes by
bug number.

Comment 14 Gurhan Ozen 2008-04-02 17:03:56 EDT
Hi Don,
Please update RHEL 5.2 release notes per dledford's comment on comment #12 .. We
do not support ipath module on ia64 at all.

Comment 15 Don Domingo 2008-04-02 23:47:41 EDT
thanks Gurhan. revising as follows (added back to "Known Issues"):

(ia64) The ipath module is not supported in this architecture, as its use may
result in errors while using openmpi.

please advise before April 15 if any further revisions are required. thanks!
Comment 16 Gurhan Ozen 2008-04-04 12:27:14 EDT
(In reply to comment #15)
> thanks Gurhan. revising as follows (added back to "Known Issues"):
> <quote>
> (ia64) The ipath module is not supported in this architecture, as its use may
> result in errors while using openmpi.
> </quote>

  I think that sentence is not concise enough. We should just say using ib_ipath
module does only work on x86_64 architecture. It doesn't work on i386 or ia64.

   Throwing openmpi into this is unnecessary, since openmpi can be run via other
means, thru TCP or thru IB over  Mellanox cards.

   Doug, please feel free to advise. 

> please advise before April 15 if any further revisions are required. thanks!

Comment 17 Don Domingo 2008-04-08 00:25:15 EDT

i removed the reference to openmpi at your suggestion, however i thought it best
to simply add the release note to both ia64 and x86 arches. for "Known Issues",
i feel it best to inform the affected parties directly what doesn't work as
expected (i assume that this bug was release-noted because ia64 users expect the
ipath module to be supported?).
Comment 18 Doug Ledford 2008-04-08 00:45:40 EDT
No, in actuality, the bug was release noted because *we* expected ipath to
support ia64.  We were later corrected on that issue by the ipath driver
maintainer.  Now, in regards to our customers, they buy their hardware from the
vendor and the vendors knew that ipath didn't support ia64 even though we
didn't.  So, the customers didn't have an expectation that ipath supported ia64,
only we did, and we were corrected.  Therefore I don't think any note at all is
necessary.  We don't need to point out that it isn't supported, nor that
anything was resolved since it wasn't.  We just don't need to be pointing out
something that the hardware vendors already know and deal with.
Comment 19 Don Domingo 2008-04-08 19:24:33 EDT
thanks for clarifying this, Doug. removing release note altogether, clearing
RHEl5.2 release note flag. 
Comment 20 Doug Ledford 2008-06-14 11:34:03 EDT
Closing this bug out entirely as there is no longer any need for a release note
(as of 5.2 I shipped an awk script that resolves the unbelievable slowness of
the ipath hardware on some machines and it was that slowness that was causing
openmpi to crash).

Note You need to log in before you can comment on or make changes to this bug.