Bug 794880

Summary: [abrt] openmpi-1.5.4-3.fc16: Process /usr/lib/openmpi/bin/orted was killed by signal 11 (SIGSEGV)
Product: [Fedora] Fedora Reporter: Sergei LITVINENKO <sergei.litvinenko>
Component: openmpiAssignee: Jay Fenlason <fenlason>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16CC: dledford, fenlason, jfeeney
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Unspecified   
Whiteboard: abrt_hash:864d4a3b6d29cfc0f491b826b49f4f297822cd41
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-13 14:25:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
File: maps
none
File: smolt_data
none
File: backtrace none

Description Sergei LITVINENKO 2012-02-17 20:29:18 UTC
libreport version: 2.0.8
abrt_version:   2.0.7
backtrace_rating: 4
cmdline:        orted --debug -mca ess env -mca orte_ess_jobid 2100559872 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 2 --hnp-uri 2100559872.0;tcp://10.119.100.6:56141 -mca plm rsh
crash_function: orte_odls_base_default_construct_child_list
executable:     /usr/lib/openmpi/bin/orted
kernel:         3.2.6-3.fc16.i686.PAE
pid:            32296
pwd:            /home/sergeil
reason:         Process /usr/lib/openmpi/bin/orted was killed by signal 11 (SIGSEGV)
time:           Пт. 17 февр. 2012 22:14:07
uid:            1000
username:       sergeil

backtrace:      Text file, 67192 bytes
maps:           Text file, 6418 bytes
smolt_data:     Binary file, 4273 bytes

build_ids:
:85b11c220dc92b70a4405c3afffc410608ab29f6
:e8e1e23704acbfe4c07b40b4277e792b663cd053
:d7ca3751445b6644328a81f136899751121a2e32
:96b666a7f6d7a80ea6f9aef54f0cdd0f6190c058
:5b853064c3d832d471e71bb67b714a7f69ff11d3
:3aed71a6e79a28101ca36e6e9b3e8e053b73421a
:e08267f8f9d16ae8c82160abe37a16dfcd3504da
:6e7f601b12a9cbb61564a6f4b3f90d5629b073e2
:62a06c0714dc286b30a2ec185f53e1cedbf800ca
:1273fae6d85d3347f08bae98028dafbef65639c3
:d1bd2374eb12ee6116690a80ce23d695b3d585e8
:83f4eb739af4440c5c0c8f8c9c35cee4f2862951
:f8e2e3adf420486889169759ccf806dc393ca12f
:8ecf640dfdeffb08819e97ec1bcf9667e2f85e4c
:4cd55f94e68f421480f0c88fc5ca3d642fe74aec
:f9e2b98a3ed0fc9ff7657ad8f48bfc74a78c0f6d
:879571cf1e8dd31736d0f24b55e0daffc1591278
:c28ba54f3bf4e4dcceb579daa3654394d1610fdb
:12da4d1dc9764d3990ab6a2def5a88c4687d4550
:a4b20d2b24b6e526c7ec83376fea135ce1586ef9
:6d37684511b1a3694e5acde75d73cc196e629977
:89d04bfbe4b627a756c1c4d82fb47664701f7a86
:d1fa9054db61eabe5e8646f93269d6300bbdba48
:82674b49345364df4c158e8258b93fd4d8fdbae0
:d7cd016536dcce05ea28a1219c92eb9507121fa3
:d94f52f020f5a21d89ce70745a9bfb78cee38ba4
:8163f121b94883addbeb3dc331bfb34338aca566
:b1e761e9a1863a759201e4a2c9211aee03042b31
:10cec21724c13b95ec03f00985b0405412817a49
:b5f249300c57e7c5beb42b37ae04516154dac0ba
:9576c7b9f86ee7390a33137a1b745a7d4adc67d8
:e742363bfe59e97f0baaf679ba58b9175b7228dd

comment:
:Run simple openmpi test:
:
:==> from centos-6.x 
:
:[sergeil@centos6 ~]$ set | grep openmpi
:LD_LIBRARY_PATH=/usr/lib/openmpi/lib:/usr/lib/openmpi/lib/openmpi:/usr/lib/openmpi/bin:
:PATH=/usr/lib/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/lib/openmpi/bin:/home/sergeil/bin
:
:[sergeil@centos6 ~]$ rpm -qa | grep openmpi
:openmpi-1.5.3-3.el6.i686
:openmpi-devel-1.5.3-3.el6.i686
:
:
:==> to fedora-16
:
:[sergeil@homedesk ~]$ set | grep openmpi
:LD_LIBRARY_PATH=/usr/lib/openmpi/lib:/usr/lib/openmpi/lib/openmpi:/usr/lib/openmpi/lib:
:PATH=/usr/lib/ccache:/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/local/sbin:/usr/sbin:/sbin:/usr/lib/openmpi/bin:/home/sergeil/bin:/usr/lib/openmpi/bin
:
:[sergeil@homedesk ~]$ rpm -qa | grep openmpi
:openmpi-1.5.4-3.fc16.i686
:openmpi-devel-1.5.4-3.fc16.i686
:
:[sergeil@centos6 ~]$ mpirun -d -np 2 --host homedesk /bin/hostname
:[centos6:02504] procdir: /tmp/openmpi-sessions-sergeil@centos6_0/32052/0/0
:[centos6:02504] jobdir: /tmp/openmpi-sessions-sergeil@centos6_0/32052/0
:[centos6:02504] top: openmpi-sessions-sergeil@centos6_0
:[centos6:02504] tmp: /tmp
:[homedesk:32296] procdir: /tmp/openmpi-sessions-sergeil@homedesk_0/32052/0/1
:[homedesk:32296] jobdir: /tmp/openmpi-sessions-sergeil@homedesk_0/32052/0
:[homedesk:32296] top: openmpi-sessions-sergeil@homedesk_0
:[homedesk:32296] tmp: /tmp
:[homedesk:32296] [[32052,0],1] node[0].name centos6 daemon 0
:[homedesk:32296] [[32052,0],1] node[1].name homedesk daemon 1
:[homedesk:32296] *** Process received signal ***
:[homedesk:32296] Signal: Segmentation fault (11)
:[homedesk:32296] Signal code: Address not mapped (1)
:[homedesk:32296] Failing at address: 0x9336380
:[homedesk:32296] [ 0] [0xb76e240c]
:[homedesk:32296] [ 1] /usr/lib/openmpi/lib/libopen-rte.so.3(orte_odls_base_default_construct_child_list+0x5ea) [0xb765e46a]
:[homedesk:32296] [ 2] /usr/lib/openmpi/lib/openmpi/mca_odls_default.so(+0x166b) [0xb73ab66b]
:[homedesk:32296] [ 3] /usr/lib/openmpi/lib/libopen-rte.so.3(+0x329c4) [0xb764e9c4]
:[homedesk:32296] [ 4] /usr/lib/openmpi/lib/libopen-rte.so.3(orte_daemon_cmd_processor+0x474) [0xb76516d4]
:[homedesk:32296] [ 5] /usr/lib/openmpi/lib/libopen-rte.so.3(+0x71fc4) [0xb768dfc4]
:[homedesk:32296] [ 6] /usr/lib/openmpi/lib/libopen-rte.so.3(opal_event_loop+0x26) [0xb768e1b6]
:[homedesk:32296] [ 7] /usr/lib/openmpi/lib/libopen-rte.so.3(opal_event_dispatch+0x1c) [0xb768e1dc]
:[homedesk:32296] [ 8] /usr/lib/openmpi/lib/libopen-rte.so.3(orte_daemon+0xf77) [0xb764d917]
:[homedesk:32296] [ 9] orted() [0x8048595]
:[homedesk:32296] [10] /lib/libc.so.6(__libc_start_main+0xf3) [0xb73eb6b3]
:[homedesk:32296] [11] orted() [0x80485c5]
:[homedesk:32296] *** End of error message ***
:--------------------------------------------------------------------------
:A daemon (pid 2505) died unexpectedly with status 255 while attempting
:to launch so we are aborting.
:
:There may be more information reported by the environment (see above).
:
:This may be because the daemon was unable to find all the needed shared
:libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
:location of the shared libraries on the remote nodes and this will
:automatically be forwarded to the remote nodes.
:--------------------------------------------------------------------------
:--------------------------------------------------------------------------
:mpirun noticed that the job aborted, but has no info as to the process
:that caused that situation.
:--------------------------------------------------------------------------
:[centos6:02504] sess_dir_finalize: job session dir not empty - leaving
:[centos6:02504] [[32052,0],0] Releasing job data for [32052,0]
:[centos6:02504] [[32052,0],0] Releasing job data for [32052,1]
:[centos6:02504] sess_dir_finalize: proc session dir not empty - leaving
:orterun: exiting with status 255

dso_list:
:/usr/lib/openmpi/lib/openmpi/mca_grpcomm_bad.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libpthread-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/openmpi/mca_iof_orted.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/libhwloc.so.4.0.4 hwloc-1.3-1.fc16.i686 (Fedora Project) 1328205851
:/lib/libutil-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/openmpi/mca_carto_auto_detect.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/openmpi/lib/openmpi/mca_sysinfo_linux.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/ld-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/libltdl.so.7.3.0 libtool-ltdl-2.4-8.fc16.i686 (Fedora Project) 1329419671
:/usr/lib/openmpi/lib/openmpi/mca_routed_binomial.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libnss_files-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/lib/libdl-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/openmpi/mca_pstat_linux.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libresolv-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/libopen-rte.so.3.0.0 openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libnsl-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/openmpi/mca_plm_rsh.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libz.so.1.2.5 zlib-1.2.5-6.fc16.i686 (Fedora Project) 1326909364
:/lib/libgcc_s-4.6.2-20111027.so.1 libgcc-4.6.2-1.fc16.i686 (Fedora Project) 1321097526
:/usr/lib/openmpi/lib/openmpi/mca_paffinity_hwloc.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libc-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/openmpi/lib/openmpi/mca_oob_tcp.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/openmpi/lib/openmpi/mca_ess_env.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/openmpi/lib/openmpi/mca_odls_default.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libpci.so.3.1.7 pciutils-libs-3.1.7-6.fc16.i686 (Fedora Project) 1321097624
:/usr/lib/libnuma.so.1 numactl-2.0.7-2.fc16.i686 (Fedora Project) 1321097711
:/usr/lib/openmpi/bin/orted openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/openmpi/lib/openmpi/mca_filem_rsh.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/usr/lib/openmpi/lib/openmpi/mca_rml_oob.so openmpi-1.5.4-3.fc16.i686 (Fedora Project) 1328205853
:/lib/libm-2.14.90.so glibc-2.14.90-24.fc16.4.i686 (Fedora Project) 1325330582
:/usr/lib/libxml2.so.2.7.8 libxml2-2.7.8-6.fc16.i686 (Fedora Project) 1321097569

environ:
:XDG_SESSION_ID=33
:SELINUX_ROLE_REQUESTED=
:SHELL=/bin/bash
:'SSH_CLIENT=10.119.100.6 47920 22'
:SELINUX_USE_CURRENT_RANGE=
:USER=sergeil
:LD_LIBRARY_PATH=/usr/lib/openmpi/lib:/usr/lib/openmpi/lib/openmpi:
:MAIL=/var/mail/sergeil
:PATH=/usr/lib/ccache:/usr/local/bin:/bin:/usr/bin:/usr/lib/openmpi/bin
:PWD=/home/sergeil
:XMODIFIERS=@im=none
:LANG=ru_RU.UTF-8
:MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
:LOADEDMODULES=
:KDEDIRS=/usr
:SELINUX_LEVEL_REQUESTED=
:SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
:SHLVL=1
:HOME=/home/sergeil
:LANGUAGE=
:LOGNAME=sergeil
:CVS_RSH=ssh
:'SSH_CONNECTION=10.119.100.6 47920 10.119.100.100 22'
:MODULESHOME=/usr/share/Modules
:'LESSOPEN=||/usr/bin/lesspipe.sh %s'
:XDG_RUNTIME_DIR=/run/user/sergeil
:CCACHE_HASHDIR=
:'module=() {  eval `/usr/bin/modulecmd bash $*`\n}'
:_=/usr/lib/openmpi/bin/orted

var_log_messages:
:Feb 17 21:59:08 homedesk kernel: [ 9244.904591] orted[31254]: segfault at 99cd240 ip b76ca46a sp bf962450 error 4 in libopen-rte.so.3.0.0[b7688000+ba000]
:Feb 17 21:59:08 homedesk abrt[31268]: Saved core dump of pid 31254 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-21:59:08-31254 (876544 bytes)
:Feb 17 21:59:57 homedesk kernel: [ 9293.922359] orted[31330]: segfault at 8f58f78 ip b774246a sp bffd1580 error 4 in libopen-rte.so.3.0.0[b7700000+ba000]
:Feb 17 21:59:57 homedesk abrt[31331]: Saved core dump of pid 31330 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-21:59:57-31330 (876544 bytes)
:Feb 17 22:00:12 homedesk kernel: [ 9308.925487] orted[31368]: segfault at 8677240 ip b769546a sp bf97ac60 error 4 in libopen-rte.so.3.0.0[b7653000+ba000]
:Feb 17 22:00:12 homedesk abrt[31382]: Not dumping repeating crash in '/usr/lib/openmpi/bin/orted'
:Feb 17 22:00:40 homedesk kernel: [ 9336.712256] orted[31407]: segfault at 9cad358 ip b76e846a sp bfbc1f10 error 4 in libopen-rte.so.3.0.0[b76a6000+ba000]
:Feb 17 22:00:40 homedesk abrt[31421]: Saved core dump of pid 31407 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-22:00:40-31407 (876544 bytes)
:Feb 17 22:04:02 homedesk kernel: [ 9538.880986] orted[31550]: segfault at 857a358 ip b76f546a sp bfadfdc0 error 4 in libopen-rte.so.3.0.0[b76b3000+ba000]
:Feb 17 22:04:02 homedesk abrt[31564]: Saved core dump of pid 31550 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-22:04:02-31550 (876544 bytes)
:Feb 17 22:04:32 homedesk gnome-keyring-daemon[31599]: unsupported key algorithm in certificate: 1.2.840.10045.2.1
:Feb 17 22:04:32 homedesk gnome-keyring-daemon[31599]: unsupported key algorithm in certificate: 1.2.840.10045.2.1
:Feb 17 22:04:32 homedesk gnome-keyring-daemon[31599]: unsupported key algorithm in certificate: 1.2.840.10045.2.1
:Feb 17 22:04:32 homedesk gnome-keyring-daemon[31599]: unsupported key algorithm in certificate: 1.2.840.10045.2.1
:Feb 17 22:04:32 homedesk gnome-keyring-daemon[31599]: unsupported key algorithm in certificate: 1.2.840.10045.2.1
:Feb 17 22:11:48 homedesk kernel: [10005.217518] orted[32104]: segfault at 8908f78 ip b76a546a sp bf889d90 error 4 in libopen-rte.so.3.0.0[b7663000+ba000]
:Feb 17 22:11:48 homedesk abrt[32105]: Saved core dump of pid 32104 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-22:11:48-32104 (876544 bytes)
:Feb 17 22:13:33 homedesk kernel: [10110.389053] orted[32241]: segfault at 9967fa0 ip b76ff46a sp bff56a20 error 4 in libopen-rte.so.3.0.0[b76bd000+ba000]
:Feb 17 22:13:34 homedesk abrt[32242]: Saved core dump of pid 32241 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-22:13:33-32241 (876544 bytes)
:Feb 17 22:13:50 homedesk kernel: [10126.617901] orted[32288]: segfault at 89b2fa0 ip b771d46a sp bf965140 error 4 in libopen-rte.so.3.0.0[b76db000+ba000]
:Feb 17 22:13:50 homedesk abrt[32289]: Not dumping repeating crash in '/usr/lib/openmpi/bin/orted'
:Feb 17 22:14:07 homedesk kernel: [10144.267499] orted[32296]: segfault at 9336380 ip b765e46a sp bf8a3c80 error 4 in libopen-rte.so.3.0.0[b761c000+ba000]
:Feb 17 22:14:07 homedesk abrt[32310]: Saved core dump of pid 32296 (/usr/lib/openmpi/bin/orted) to /var/spool/abrt/ccpp-2012-02-17-22:14:07-32296 (876544 bytes)

Comment 1 Sergei LITVINENKO 2012-02-17 20:29:23 UTC
Created attachment 563979 [details]
File: maps

Comment 2 Sergei LITVINENKO 2012-02-17 20:29:25 UTC
Created attachment 563980 [details]
File: smolt_data

Comment 3 Sergei LITVINENKO 2012-02-17 20:29:27 UTC
Created attachment 563981 [details]
File: backtrace

Comment 4 Fedora End Of Life 2013-01-16 13:34:57 UTC
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 5 Fedora End Of Life 2013-02-13 14:25:29 UTC
Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.