Created attachment 369962 [details] Test case using Allreduce Description of problem: Regression using MPI::Allreduce with more than 4 processes. I'm working an 8 core workstation, and when I run the attached code with more that 4 processes, it hangs. Generally around iteration 100, but always less than 200. I have detected no pattern to the number of iterations. Version-Release number of selected component (if applicable): openmpi-1.3.3-6.fc11.x86_64 FAILS! openmpi-1.3.1-1.fc11.x86_64 WORKS! How reproducible: Every time. Steps to Reproduce: 1. Update to openmpi-1.3.3-6.fc11.x86_64 2. make allreduce CXX=mpic++ CXXFLAGS="-O0 -g" 3. mpiexec -n 8 ./allreduce Actual results: Stops before reaching 99999. Expected results: All numbers from 0 to 99999 printed. Additional info: I haven't tried other collective functions, so others may fail as well. I also haven't tried other architectures or running on a true cluster.
Sounds like what I've been seeing on Fedora 11 and 12 (and RHEL 5.4) with openmpi >= 1.3.2. But it is not just with allreduce. These random hangs are easy to reproduce with the Intel MPI benchmarks. Corresponding openmpi ticket is probably this one https://svn.open-mpi.org/trac/ompi/ticket/2043
The r22324 patch mentioned in the openmpi ticket seems to fix the problem for me. Looks like a bug in the use of gcc inline assembly.
I added the r22324 patch to the openmpi-1.4-1 build.
openmpi-1.4-1.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/openmpi-1.4-1.fc11
openmpi-1.4-1.fc12 has been submitted as an update for Fedora 12. http://admin.fedoraproject.org/updates/openmpi-1.4-1.fc12
openmpi-1.4-1.fc11 has been pushed to the Fedora 11 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update openmpi'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2010-0544
openmpi-1.4-1.fc12 has been pushed to the Fedora 12 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update openmpi'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F12/FEDORA-2010-0552
This message is a reminder that Fedora 11 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 11. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '11'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 11's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 11 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
The current version (1.4.1-2.fc12) no longer has this bug. Thanks for updating.
Thank you for confirming this is fixed.