Red Hat Bugzilla – Bug 274851
openib: openmpi crashes on ia64 platform.
Last modified: 2013-11-03 20:33:59 EST
Description of problem:
Openmpi programs are not runnable on ia64 platform due to random crashes:
# mpirun -np 2 --mca btl openib,self /usr/bin/mpitests-osu_bw
# OSU MPI Bandwidth Test (Version 2.2)
# Size Bandwidth (MB/s)
[intel-s6e4533-01-mm:28875] *** Process received signal ***
[intel-s6e4533-01-mm:28875] Signal: Segmentation fault (11)
[intel-s6e4533-01-mm:28875] Signal code: Invalid permissions (2)
[intel-s6e4533-01-mm:28875] Failing at address: (nil)
[intel-s6e4533-01-mm:28875] [ 0] [0xa0000000000107e0]
[intel-s6e4533-01-mm:28875] [ 1]
[intel-s6e4533-01-mm:28875] [ 2]
[intel-s6e4533-01-mm:28875] [ 3]
[intel-s6e4533-01-mm:28875] [ 4]
[intel-s6e4533-01-mm:28875] [ 5]
[intel-s6e4533-01-mm:28875] [ 6]
[intel-s6e4533-01-mm:28875] [ 7] /lib/libc.so.6.1(__libc_start_main-0x287540)
[intel-s6e4533-01-mm:28875] [ 8]
[intel-s6e4533-01-mm:28875] *** End of error message ***
mpirun noticed that job rank 0 with PID 28874 on node
intel-s6e4533-01-mm.rhts.boston.redhat.com exited on signal 15 (Terminated).
1 additional process aborted (not shown)
Above was retrieved when running the program only on 1 node. When running ia64
as the master node execution crashed with the following error:
error posting send request errno says Invalid argument
error posting send request errno says Invalid argument
When running as ia64 remote node, the following happens:
[intel-s6e4533-01-mm:28971] *** Process received signal ***
[intel-s6e4533-01-mm:28971] Signal: Segmentation fault (11)
[intel-s6e4533-01-mm:28971] Signal code: Invalid permissions (2)
[intel-s6e4533-01-mm:28971] Failing at address: (nil)
[intel-s6e4533-01-mm:28971] [ 0] [0xa0000000000107e0]
[intel-s6e4533-01-mm:28971] [ 1]
[intel-s6e4533-01-mm:28971] [ 2]
[intel-s6e4533-01-mm:28971] [ 3]
[intel-s6e4533-01-mm:28971] [ 4]
[intel-s6e4533-01-mm:28971] [ 5]
[intel-s6e4533-01-mm:28971] [ 6]
[intel-s6e4533-01-mm:28971] [ 7]
[intel-s6e4533-01-mm:28971] [ 8]
[intel-s6e4533-01-mm:28971] [ 9] /lib/libc.so.6.1(__libc_start_main-0x287540)
[intel-s6e4533-01-mm:28971] *** End of error message ***
mpirun noticed that job rank 0 with PID 19184 on node
dell-pe1950-03.rhts.boston.redhat.com exited on signal 15 (Terminated).
Version-Release number of selected component (if applicable):
# uname -a
Linux intel-s6e4533-01-mm.rhts.boston.redhat.com 2.6.18-43.el5 #1 SMP Tue Aug 21
17:52:47 EDT 2007 ia64 ia64 ia64 GNU/Linux
# rpm -qa | egrep "openib|openmpi"
Steps to Reproduce:
1. Try to run an openmpi program when 1 or more nodes is ia64
The observed error scenarios are:
1) When running on ia64 as master, receive retry count exceeded error. This is
not an openmpi error, but a physical link error caused either by a bad driver,
bad core IB code, or bad hardware. We have tried two different cables and two
different IB cards that used two totally different drivers (ib_mthca and
ib_ipath) and two different ports on the switch and the hardware errors persist.
This indicates that this specific failure mode may have something to do with
the core IB stack. The less likely situation is that both cards or both ports
on the switch that we tried are suspect. Of note is that after resetting the IB
switch to clear all the error counters, then restarting an RDMA transfer from
the ia64 machine to one of the x86_64 machines, the symbol error counter on the
switch port connected to the ia64 machine reached its limit of 65535 errors
within seconds (the default alarm threshold for this error counter is only 100).
Of further note is that the error only occurs on ia64 (our only big endian
machine in the test fabric, as well as the arch with the highest level of
unordered writes in the test fabric). All the other machines produce no errors.
2) When running on ia64 as master, segfault in the openmpi code related to
callback handling on completion. This situation represents a "can't happen"
situation in the openmpi code. Specifically, a request completes without the
appropriate callback handler pointer set in the request structure
(frag.base.des_cbfunc == NULL).
3) When running on something other than ia64 as master, error posting send
request errno says Invalid argument (the initial bug report had this attributed
to running on ia64 as master, which has not been seen and does not make sense
for ia64 as master). This is caused when another node in the mpirun exits
abnormally, causing the connection to be closed while reads/writes are still
pending. It is to be expected on any node if the ia64 remote node crashed.
This is consistent with the observed behavior on ia64 and merely indicates that
likely one of the two preceding errors has occurred on the ia64 node.
Items of note: Both problems #1 and #2 have been verified with both the RHEL5.1
and RHEL5.0 openmpi stacks. This indicates that the problem is likely not in
openmpi, but instead somewhere lower down in the stack. Next on the agenda is
to test the RHEL5.0 libibverbs user space drivers (the openmpi stack relies on
the libibvers stack, which interfaces directly with the kernel stack). Software
has been installed to test the RHEL5.0 kernel in isolation as well. However,
the testing of the libibverbs RHEL5.0 user space driver and the kernel will have
to wait until someone is in the Westford office that can reset the test machine
as it didn't successfully return from a reboot (in addition to needing a cable
that is currently disconnected connected).
The crashes turn out to not be related to openmpi at all, it's an openib
(libibverbs) bug. Switching component.
OK, there are two distinctly different aspects to this issue:
1) When using ipath on ia64, openmpi breaks. This isn't a libibverbs issue,
this is an ipath driver issue. The ipath driver is not safe for use on ia64 and
needs to be called out as tech preview and unsupported in the kernel release
notes for 5.1.
2) There is a bug in libibverbs that can result in sporadic memory corruption,
and I believe this is the cause of the segfaults when running openmpi on mthca
hardware on ia64 and talking to ipath hardware on x86_64. When talking to mthca
hardware on x86_64, openmpi doesn't have any problems. The ipath driver handles
memory differently than the mthca, and therefore has a different pattern for
triggering the libibverbs bug than the mthca driver does. It is this
difference, I think, that is causing the ipath/x86_64 to mthca/ia64 issue. I'm
building a test version of libibverbs through brew right now to see if it does
indeed resolve the lingering issue.
It is worth noting that bug #2 in this is something that can hit any application
given the right memory usage pattern. The fix for this has been tested
upstream, and the reproducer program they used to trigger the bug showed that
the bug hit i386 easily, while being very difficult to trigger on x86_64. I'm
guessing that ia64 is easier to trigger than x86_64 as well and that's why we've
been seeing the crashes there instead of on all the machines. The patch I'm
using was tested by the upstream author and resolved the issue, including that
the reproducer programs were no longer able to reproduce the problem. Since
this bug is legitimate and fairly severe, I would request that the blocker
status be granted so I can get the fix checked into CVS and an official build made.
*** Bug 235946 has been marked as a duplicate of this bug. ***
adding to release notes updates:
(ia64) The ipath driver is not safe for use in this architecture, as its use may
result in errors while using openmpi. As such, the ipath driver is currently
released for this architecture as Technology Preview.
please advise if any further revisions are required. thanks!
adding same release note to "Known Issues" of RHEL5.2. please advise if openmpi
is now fully supported so we can document as such. thanks!
rhel5.2 does not need any notes and is working properly
thanks. adding the following to "Resolved Issues" of RHEl5.2 release notes:
(ia64) Using the ipath driver in this architecture no longer results in openmpi
please advise if any further revisions are required. also, i assume that ipath
in ia64 is now fully supported (i.e. no longer tech preview)?
(In reply to comment #10)
> rhel5.2 does not need any notes and is working properly
No, no...ipath is still not supported on ia64, but at this point we don't think
it *should* be supported on ia64. This issue cropped up before upstream ipath
driver authors told us that they hadn't even tested ipath on ia64 and didn't
consider it a priority. So, now that we know that, we know better than to use
ipath on ia64.
the RHEL5.2 release notes will be dropped to translation on April 15, 2008, at
which point no further additions or revisions will be entertained.
a mockup of the RHEL5.2 release notes can be viewed at the following link:
please use the aforementioned link to verify if your bugzilla is already in the
release notes (if it needs to be). each item in the release notes contains a
link to its original bug; as such, you can search through the release notes by
Please update RHEL 5.2 release notes per dledford's comment on comment #12 .. We
do not support ipath module on ia64 at all.
thanks Gurhan. revising as follows (added back to "Known Issues"):
(ia64) The ipath module is not supported in this architecture, as its use may
result in errors while using openmpi.
please advise before April 15 if any further revisions are required. thanks!
(In reply to comment #15)
> thanks Gurhan. revising as follows (added back to "Known Issues"):
> (ia64) The ipath module is not supported in this architecture, as its use may
> result in errors while using openmpi.
I think that sentence is not concise enough. We should just say using ib_ipath
module does only work on x86_64 architecture. It doesn't work on i386 or ia64.
Throwing openmpi into this is unnecessary, since openmpi can be run via other
means, thru TCP or thru IB over Mellanox cards.
Doug, please feel free to advise.
> please advise before April 15 if any further revisions are required. thanks!
i removed the reference to openmpi at your suggestion, however i thought it best
to simply add the release note to both ia64 and x86 arches. for "Known Issues",
i feel it best to inform the affected parties directly what doesn't work as
expected (i assume that this bug was release-noted because ia64 users expect the
ipath module to be supported?).
No, in actuality, the bug was release noted because *we* expected ipath to
support ia64. We were later corrected on that issue by the ipath driver
maintainer. Now, in regards to our customers, they buy their hardware from the
vendor and the vendors knew that ipath didn't support ia64 even though we
didn't. So, the customers didn't have an expectation that ipath supported ia64,
only we did, and we were corrected. Therefore I don't think any note at all is
necessary. We don't need to point out that it isn't supported, nor that
anything was resolved since it wasn't. We just don't need to be pointing out
something that the hardware vendors already know and deal with.
thanks for clarifying this, Doug. removing release note altogether, clearing
RHEl5.2 release note flag.
Closing this bug out entirely as there is no longer any need for a release note
(as of 5.2 I shipped an awk script that resolves the unbelievable slowness of
the ipath hardware on some machines and it was that slowness that was causing
openmpi to crash).