Bug 510475

Summary: clustered qpidd startup - abort because of unhandled exception
Product: Red Hat Enterprise MRG Reporter: Frantisek Reznicek <freznice>
Component: qpid-cppAssignee: Andrew Stitcher <astitcher>
Status: CLOSED ERRATA QA Contact: Frantisek Reznicek <freznice>
Severity: high Docs Contact:
Priority: urgent    
Version: 1.1.2CC: atwright, esammons, gsim, tao
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Under certain circumstances, an unhandled exception may have prevented a clustered qpidd service from starting. With this update, all exception are now handled properly, and the qpidd service always starts as expected.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-14 15:59:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 591292, 602268    
Bug Blocks:    
Attachments:
Description Flags
bz510475 reproducer none

Description Frantisek Reznicek 2009-07-09 12:29:54 UTC 
Description of problem:
While using bug 506758 reproducer I triggered one clustered qpidd node abort (se core file below). It seems to be because an thrown exception:
  terminate called after throwing an instance of 'qpid::Exception'
    what():  Invalid argument (qpid/sys/posix/Mutex.h:120)

Scenario:
  Starting 4 node cluster while 75 running subscribe clients in memory waiting for cluster.

Version-Release number of selected component (if applicable):
[root@mrg-qe-01 bz506758]# rpm -qa | grep -E '(qpid|rhm|openais)' | sort -u
openais-0.80.3-22.el5_3.8
openais-debuginfo-0.80.3-22.el5_3.8
python-qpid-0.5.752581-3.el5
qpidc-0.5.752581-25.el5
qpidc-debuginfo-0.5.752581-25.el5
qpidc-devel-0.5.752581-25.el5
qpidc-perftest-0.5.752581-25.el5
qpidc-rdma-0.5.752581-25.el5
qpidc-ssl-0.5.752581-25.el5
qpidd-0.5.752581-25.el5
qpidd-acl-0.5.752581-25.el5
qpidd-cluster-0.5.752581-25.el5
qpidd-devel-0.5.752581-25.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-25.el5
qpidd-ssl-0.5.752581-25.el5
qpidd-xml-0.5.752581-25.el5
qpid-java-client-0.5.751061-8.el5
qpid-java-common-0.5.751061-8.el5
rhm-0.5.3206-6.el5
rhm-docs-0.5.756148-1.el5


How reproducible:
very rarely (<2%)

Steps to Reproduce:
1. follow bug 506758 steps or download https://bugzilla.redhat.com/attachment.cgi?id=350930 reproducer
2. repeat starting (keeping running for about 10 minutes) the test until one of the nodes aborts
  
Actual results:
The qpidd broker aborts because of unhandled exception

Expected results:
The qpidd broker should not abort

Additional info (qpidd log file and backtrace):

qpidd log file
...
2009-jul-09 13:14:19 error 10.34.33.54:29922(READY/error) channel error 958336 on 10.34.33.54:29922-25908(local): not-found: Queue not found: queue-015 (qpid/broker/SessionAdapter.cpp:742) (unresolved: 10.34.33.54:29922 10.34.33.54:29939 10.34.33.54:29957 10.34.33.54:30174 )
2009-jul-09 13:14:19 error Execution exception: not-found: Queue not found: queue-036 (qpid/broker/SessionAdapter.cpp:742)
2009-jul-09 13:14:19 error 10.34.33.54:29922(READY/error) channel error 959084 on 10.34.33.54:29922-25909(local): not-found: Queue not found: queue-036 (qpid/broker/SessionAdapter.cpp:742) (unresolved: 10.34.33.54:29922 10.34.33.54:29939 10.34.33.54:29957 10.34.33.54:30174 )
2009-jul-09 13:14:19 error Execution exception: not-found: Queue not found: queue-032 (qpid/broker/SessionAdapter.cpp:742)
2009-jul-09 13:14:19 error 10.34.33.54:29922(READY/error) channel error 959473 on 10.34.33.54:29922-25903(local): not-found: Queue not found: queue-032 (qpid/broker/SessionAdapter.cpp:742) (unresolved: 10.34.33.54:29922 10.34.33.54:29939 10.34.33.54:29957 10.34.33.54:30174 )
2009-jul-09 13:14:19 error Execution exception: not-found: Queue not found: queue-066 (qpid/broker/SessionAdapter.cpp:742)
2009-jul-09 13:14:19 error 10.34.33.54:29922(READY/error) channel error 959478 on 10.34.33.54:29922-25879(local): not-found: Queue not found: queue-066 (qpid/broker/SessionAdapter.cpp:742) (unresolved: 10.34.33.54:29922 10.34.33.54:29939 10.34.33.54:29957 10.34.33.54:30174 )
2009-jul-09 13:14:19 error Execution exception: not-found: Queue not found: queue-016 (qpid/broker/SessionAdapter.cpp:742)
2009-jul-09 13:14:19 error 10.34.33.54:29922(READY/error) channel error 959486 on 10.34.33.54:29922-25910(local): not-found: Queue not found: queue-016 (qpid/broker/SessionAdapter.cpp:742) (unresolved: 10.34.33.54:29922 10.34.33.54:29939 10.34.33.54:29957 10.34.33.54:30174 )
2009-jul-09 13:14:19 warning Client closed connection with 501: Connection is not open.
terminate called after throwing an instance of 'qpid::Exception'
  what():  Invalid argument (qpid/sys/posix/Mutex.h:120)



[root@mrg-qe-01 bz506758]# rpm -q qpidd
qpidd-0.5.752581-25.el5
[root@mrg-qe-01 bz506758]# gdb `which qpidd` core.29922
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidbroker.so.0
Reading symbols from /usr/lib/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidcommon.so.0
Reading symbols from /usr/lib/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib/libboost_program_options.so.2
Reading symbols from /usr/lib/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib/libboost_filesystem.so.2
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/acl.so
Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/ssl.so
Reading symbols from /usr/lib/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libsslcommon.so.0
Reading symbols from /usr/lib/libnss3.so...done.
Loaded symbols for /usr/lib/libnss3.so
Reading symbols from /usr/lib/libssl3.so...done.
Loaded symbols for /usr/lib/libssl3.so
Reading symbols from /usr/lib/libnspr4.so...done.
Loaded symbols for /usr/lib/libnspr4.so
Reading symbols from /usr/lib/libnssutil3.so...done.
Loaded symbols for /usr/lib/libnssutil3.so
Reading symbols from /usr/lib/libplc4.so...done.
Loaded symbols for /usr/lib/libplc4.so
Reading symbols from /usr/lib/libplds4.so...done.
Loaded symbols for /usr/lib/libplds4.so
Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so
Reading symbols from /usr/lib/qpid/daemon/msgstore.so...done.
Loaded symbols for /usr/lib/qpid/daemon/msgstore.so
Reading symbols from /usr/lib/libdb_cxx-4.3.so...done.
Loaded symbols for /usr/lib/libdb_cxx-4.3.so
Reading symbols from /usr/lib/libaio.so.1...done.
Loaded symbols for /usr/lib/libaio.so.1
Reading symbols from /usr/lib/qpid/daemon/rdma.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/rdma.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/rdma.so
Reading symbols from /usr/lib/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib/librdmawrap.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/librdmawrap.so.0
Reading symbols from /usr/lib/librdmacm.so.1...done.
Loaded symbols for /usr/lib/librdmacm.so.1
Reading symbols from /usr/lib/libibverbs.so.1...done.
Loaded symbols for /usr/lib/libibverbs.so.1
Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/cluster.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/cluster.so
Reading symbols from /usr/lib/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib/openais/libcpg.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib/openais/libcpg.so.2
Reading symbols from /usr/lib/libcman.so.2...done.
Loaded symbols for /usr/lib/libcman.so.2
Reading symbols from /usr/lib/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib/libqpidclient.so.0
Reading symbols from /usr/lib/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/client/sslconnector.so
Reading symbols from /usr/lib/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/rdmaconnector.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/client/rdmaconnector.so
Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/xml.so
Reading symbols from /usr/lib/libxerces-c.so.28...done.
Loaded symbols for /usr/lib/libxerces-c.so.28
Reading symbols from /usr/lib/libxqilla.so.3...done.
Loaded symbols for /usr/lib/libxqilla.so.3
Reading symbols from /usr/lib/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib/sasl2/libplain.so.2
Reading symbols from /usr/lib/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib/sasl2/libsasldb.so.2
Reading symbols from /usr/lib/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib/sasl2/liblogin.so.2
Reading symbols from /usr/lib/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib/sasl2/libanonymous.so.2
Core was generated by `qpidd -p 5672 --auth no --log-enable info+ --cluster-name mrg-qe-01.lab.eng.brq'.
Program terminated with signal 6, Aborted.
[New process 29929]
[New process 29936]
[New process 29935]
[New process 29934]
[New process 29933]
[New process 29932]
[New process 29931]
[New process 29930]
[New process 29927]
[New process 29926]
[New process 29925]
[New process 29924]
[New process 29922]
#0  0x00e02410 in __kernel_vsyscall ()
(gdb) info threads
  13 process 29922  0x00e02410 in __kernel_vsyscall ()
  12 process 29924  0x00e02410 in __kernel_vsyscall ()
  11 process 29925  0x00e02410 in __kernel_vsyscall ()
  10 process 29926  0x00e02410 in __kernel_vsyscall ()
  9 process 29927  0x00e02410 in __kernel_vsyscall ()
  8 process 29930  0x0046bd31 in qpid::cluster::Connection::decode (this=0xad4cf190, buffer=0xae010020 "\017\001", size=366)
    at qpid/cluster/Connection.cpp:214
  7 process 29931  0x00e02410 in __kernel_vsyscall ()
  6 process 29932  0x00e02410 in __kernel_vsyscall ()
  5 process 29933  0x00e02410 in __kernel_vsyscall ()
  4 process 29934  0x00cb8bf1 in qpid::broker::SessionState::handleContent (this=0xac6c0fd8, frame=@0xb20debd8, id=@0xb20de2cc)
    at qpid/broker/SessionState.cpp:233
  3 process 29935  0x00e02410 in __kernel_vsyscall ()
  2 process 29936  0x00e02410 in __kernel_vsyscall ()
* 1 process 29929  0x00e02410 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 13 (process 29922):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00a47ac9 in qpid::sys::Poller::wait (this=0x8d5eaf8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00a48af6 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#5  0x00be7991 in qpid::broker::Broker::run (this=0x8d6c260) at qpid/broker/Broker.cpp:319
#6  0x0804dd44 in QpiddBroker::execute (this=0xbfe99805, options=0x8d6a838) at posix/QpiddBroker.cpp:166
#7  0x0804c6c7 in main (argc=11, argv=0xbfe998b4) at qpidd.cpp:77

Thread 12 (process 29924):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x006828c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x005e1b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00cc0dbe in qpid::broker::Timer::run (this=0xb7ec00b0) at qpid/sys/posix/Condition.h:69
#4  0x00a3da51 in runRunnable (p=0xb7ec00b0) at qpid/sys/posix/Thread.cpp:35
#5  0x0067e49b in start_thread () from /lib/libpthread.so.0
#6  0x005d542e in clone () from /lib/libc.so.6

Thread 11 (process 29925):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x006828c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x005e1b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00cc0dbe in qpid::broker::Timer::run (this=0x8d6c464) at qpid/sys/posix/Condition.h:69
#4  0x00a3da51 in runRunnable (p=0x8d6c464) at qpid/sys/posix/Thread.cpp:35
#5  0x0067e49b in start_thread () from /lib/libpthread.so.0
#6  0x005d542e in clone () from /lib/libc.so.6

Thread 10 (process 29926):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x006828c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x005e1b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00cc0dbe in qpid::broker::Timer::run (this=0x8d6c4e4) at qpid/sys/posix/Condition.h:69
#4  0x00a3da51 in runRunnable (p=0x8d6c4e4) at qpid/sys/posix/Thread.cpp:35
---Type <return> to continue, or q <return> to quit---
#5  0x0067e49b in start_thread () from /lib/libpthread.so.0
#6  0x005d542e in clone () from /lib/libc.so.6

Thread 9 (process 29927):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x006828c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x005e1b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00cc0dbe in qpid::broker::Timer::run (this=0x8d72380) at qpid/sys/posix/Condition.h:69
#4  0x00a3da51 in runRunnable (p=0x8d72380) at qpid/sys/posix/Thread.cpp:35
#5  0x0067e49b in start_thread () from /lib/libpthread.so.0
#6  0x005d542e in clone () from /lib/libc.so.6

Thread 8 (process 29930):
#0  0x0046bd31 in qpid::cluster::Connection::decode (this=0xad4cf190, buffer=0xae010020 "\017\001", size=366)
    at qpid/cluster/Connection.cpp:214
#1  0x0046f211 in qpid::cluster::ConnectionCodec::decode (this=0xad57ede8, buffer=0xae010020 "\017\001", size=366)
    at qpid/cluster/ConnectionCodec.cpp:64
#2  0x00a97ee6 in qpid::sys::AsynchIOHandler::readbuff (this=0xacd253e0, buff=0xae9b4010) at qpid/sys/AsynchIOHandler.cpp:113
#3  0x00ce6024 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=
      {obj_ptr = 0xae118c00, const_obj_ptr = 0xae118c00, func_ptr = 0xae118c00, data = ""}, a0=@0xae118ae0, a1=0xae9b4010)
    at /usr/include/boost/bind/mem_fn_template.hpp:252
#4  0x00a39f2b in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (this=0xae118b5c, a0=@0xae118ae0, a1=0xae9b4010) at /usr/include/boost/function/function_template.hpp:576
#5  0x00a337ce in qpid::sys::posix::AsynchIO::readable (this=0xae118ae0, h=@0xae118ae4) at qpid/sys/posix/AsynchIO.cpp:447
#6  0x00a387cd in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0xae118c78, const_obj_ptr = 0xae118c78, func_ptr = 0xae118c78, data = "x"}, a0=@0xae118ae4)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#7  0x00a9c214 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0xae118aec, a0=@0xae118ae4) at /usr/include/boost/function/function_template.hpp:576
#8  0x00a9b8f9 in qpid::sys::DispatchHandle::processEvent (this=0xae118ae4, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#9  0x00a48b14 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/Poller.h:122
---Type <return> to continue, or q <return> to quit---
#10 0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#11 0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#12 0x0067e49b in start_thread () from /lib/libpthread.so.0
#13 0x005d542e in clone () from /lib/libc.so.6

Thread 7 (process 29931):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00a47ac9 in qpid::sys::Poller::wait (this=0x8d5eaf8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00a48af6 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#5  0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#6  0x0067e49b in start_thread () from /lib/libpthread.so.0
#7  0x005d542e in clone () from /lib/libc.so.6

Thread 6 (process 29932):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00a47ac9 in qpid::sys::Poller::wait (this=0x8d5eaf8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00a48af6 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#5  0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#6  0x0067e49b in start_thread () from /lib/libpthread.so.0
#7  0x005d542e in clone () from /lib/libc.so.6

Thread 5 (process 29933):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00a47ac9 in qpid::sys::Poller::wait (this=0x8d5eaf8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00a48af6 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#5  0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#6  0x0067e49b in start_thread () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#7  0x005d542e in clone () from /lib/libc.so.6

Thread 4 (process 29934):
#0  0x00cb8bf1 in qpid::broker::SessionState::handleContent (this=0xac6c0fd8, frame=@0xb20debd8, id=@0xb20de2cc)
    at qpid/broker/SessionState.cpp:233
#1  0x00cb9b73 in qpid::broker::SessionState::handleIn (this=0xac6c0fd8, frame=@0xb20debd8) at qpid/broker/SessionState.cpp:326
#2  0x00cb9c7b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xac6c10d8, t=@0xb20debd8) at qpid/framing/Handler.h:67
#3  0x00a68c15 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xac65a640, f=@0xb20debd8) at qpid/amqp_0_10/SessionHandler.cpp:91
#4  0x00cb9c7b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xac65a648, t=@0xb20debd8) at qpid/framing/Handler.h:67
#5  0x00c1a2f0 in qpid::broker::Connection::received (this=0xaebec7f0, frame=@0xb20debd8) at qpid/framing/Handler.h:42
#6  0x0046ce68 in qpid::cluster::Connection::deliveredFrame (this=0xaebec760, f=@0xb20debc8) at qpid/cluster/Connection.cpp:160
#7  0x0043e24d in qpid::cluster::Cluster::processFrame (this=0x8d728e0, e=@0xb20debc8, l=@0xb20dec18) at qpid/cluster/Cluster.cpp:440
#8  0x0043fa8f in qpid::cluster::Cluster::deliveredFrame (this=0x8d728e0, efConst=@0x94305e8) at qpid/cluster/Cluster.cpp:420
#9  0x004489ad in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::cluster::Cluster, qpid::cluster::EventFrame const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::Cluster*>, boost::arg<1> > >, void, qpid::cluster::EventFrame const&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d72768, const_obj_ptr = 0x8d72768, func_ptr = 0x8d72768, data = "h"}, a0=@0x94305e8)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#10 0x0044cb84 in boost::function1<void, qpid::cluster::EventFrame const&, std::allocator<void> >::operator() (this=0x8d72c54,
    a0=@0x94305e8) at /usr/include/boost/function/function_template.hpp:576
#11 0x00454064 in qpid::cluster::PollableQueue<qpid::cluster::EventFrame>::handleBatch (this=0x8d72bc4, values=@0x8d72c3c)
    at qpid/cluster/PollableQueue.h:52
#12 0x00448ac8 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, boost::_mfi::mf1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::PollableQueue<qpid::cluster::EventFrame>*>, boost::arg<1> > >, __gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d73388, const_obj_ptr = 0x8d73388, func_ptr = 0x8d73388, data = "\210"}, a0=@0x8d72c3c)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#13 0x0044c90b in boost::function1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame---Type <return> to continue, or q <return> to quit---
, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&, std::allocator<void> >::operator() (this=0x8d72c0c, a0=@0x8d72c3c) at /usr/include/boost/function/function_template.hpp:576
#14 0x0045398d in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::process (this=0x8d72bc4) at qpid/sys/PollableQueue.h:153
#15 0x00455308 in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::dispatch (this=0x8d72bc4, cond=@0x8d72c18)
    at qpid/sys/PollableQueue.h:138
#16 0x00448b0d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::EventFrame>, qpid::sys::PollableCondition&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<qpid::cluster::EventFrame>*>, boost::arg<1> > >, void, qpid::sys::PollableCondition&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d73758, const_obj_ptr = 0x8d73758, func_ptr = 0x8d73758, data = "X"}, a0=@0x8d72c18)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#17 0x00a458e7 in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() (
    this=0x8d734f8, a0=@0x8d72c18) at /usr/include/boost/function/function_template.hpp:576
#18 0x00a41b47 in qpid::sys::PollableConditionPrivate::dispatch (this=0x8d734f0) at qpid/sys/posix/PollableCondition.cpp:89
#19 0x00a4487d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d73860, const_obj_ptr = 0x8d73860, func_ptr = 0x8d73860, data = "`"}, a0=@0x8d737b0)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#20 0x00a9c214 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x8d737b8, a0=@0x8d737b0) at /usr/include/boost/function/function_template.hpp:576
#21 0x00a9b8f9 in qpid::sys::DispatchHandle::processEvent (this=0x8d737b0, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#22 0x00a48b14 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/Poller.h:122
#23 0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#24 0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#25 0x0067e49b in start_thread () from /lib/libpthread.so.0
#26 0x005d542e in clone () from /lib/libc.so.6

Thread 3 (process 29935):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d6ceb in semop () from /lib/libc.so.6
#2  0x0029173a in openais_msg_send_reply_receive (ipc_context=<value optimized out>, iov=<value optimized out>,
    iov_len=<value optimized out>, res_msg=Could not find the frame base for "openais_msg_send_reply_receive".
) at util.c:623
#3  0x00292357 in cpg_mcast_joined (handle=<value optimized out>, guarantee=<value optimized out>, iovec=<value optimized out>,
    iov_len=<value optimized out>) at cpg.c:513
#4  0x00471420 in qpid::cluster::Cpg::mcast (this=0x8d72918, iov=0xb14ddd68, iovLen=1) at qpid/cluster/Cpg.cpp:124
---Type <return> to continue, or q <return> to quit---
#5  0x0048da99 in qpid::cluster::Multicaster::sendMcast (this=0x8d72a0c, values=@0x8d72aac) at qpid/cluster/Multicaster.cpp:79
#6  0x0048f8a8 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, boost::_mfi::mf1<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, qpid::cluster::Multicaster, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::Multicaster*>, boost::arg<1> > >, __gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d726e8, const_obj_ptr = 0x8d726e8, func_ptr = 0x8d726e8, data = "�"}, a0=@0x8d72aac)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#7  0x0044ca4b in boost::function1<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&, std::allocator<void> >::operator() (this=0x8d72a7c, a0=@0x8d72aac) at /usr/include/boost/function/function_template.hpp:576
#8  0x00453add in qpid::sys::PollableQueue<qpid::cluster::Event>::process (this=0x8d72a34) at qpid/sys/PollableQueue.h:153
#9  0x00455758 in qpid::sys::PollableQueue<qpid::cluster::Event>::dispatch (this=0x8d72a34, cond=@0x8d72a88)
    at qpid/sys/PollableQueue.h:138
#10 0x00448a7d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::Event>, qpid::sys::PollableCondition&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<qpid::cluster::Event>*>, boost::arg<1> > >, void, qpid::sys::PollableCondition&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d72728, const_obj_ptr = 0x8d72728, func_ptr = 0x8d72728, data = "("}, a0=@0x8d72a88)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#11 0x00a458e7 in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() (
    this=0x8d70130, a0=@0x8d72a88) at /usr/include/boost/function/function_template.hpp:576
#12 0x00a41b47 in qpid::sys::PollableConditionPrivate::dispatch (this=0x8d70128) at qpid/sys/posix/PollableCondition.cpp:89
#13 0x00a4487d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0x8d72830, const_obj_ptr = 0x8d72830, func_ptr = 0x8d72830, data = "0"}, a0=@0x8d72780)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#14 0x00a9c214 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x8d72788, a0=@0x8d72780) at /usr/include/boost/function/function_template.hpp:576
#15 0x00a9b8f9 in qpid::sys::DispatchHandle::processEvent (this=0x8d72780, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#16 0x00a48b14 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/Poller.h:122
#17 0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#18 0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#19 0x0067e49b in start_thread () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#20 0x005d542e in clone () from /lib/libc.so.6

Thread 2 (process 29936):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x005d5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00a47ac9 in qpid::sys::Poller::wait (this=0x8d5eaf8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00a48af6 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#5  0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#6  0x0067e49b in start_thread () from /lib/libpthread.so.0
#7  0x005d542e in clone () from /lib/libc.so.6

Thread 1 (process 29929):
#0  0x00e02410 in __kernel_vsyscall ()
#1  0x0052cd80 in raise () from /lib/libc.so.6
#2  0x0052e691 in abort () from /lib/libc.so.6
#3  0x001c6aa0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x001c4505 in ?? () from /usr/lib/libstdc++.so.6
#5  0x001c4542 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x001c467a in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x00a9b866 in qpid::sys::DispatchHandle::processEvent (this=0xae50ec7c, type=qpid::sys::Poller::READABLE)
    at qpid/sys/posix/Mutex.h:120
#8  0x00a48b14 in qpid::sys::Poller::run (this=0x8d5eaf8) at qpid/sys/Poller.h:122
#9  0x00a98bd4 in qpid::sys::Dispatcher::run (this=0xbfe99524) at qpid/sys/Dispatcher.cpp:37
#10 0x00a3da51 in runRunnable (p=0xbfe99524) at qpid/sys/posix/Thread.cpp:35
#11 0x0067e49b in start_thread () from /lib/libpthread.so.0
#12 0x005d542e in clone () from /lib/libc.so.6
Comment 1 Frantisek Reznicek 2009-07-09 13:20:45 UTC 
The occurence of this issue seems to be higher than stated above.

I triggered it once more now on RHEL 5.3 x86_64:

[root@mrg-qe-02 bz506758]# gdb `which qpidd` core.2003
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /usr/lib64/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.0
Reading symbols from /usr/lib64/libqpidcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.0
Reading symbols from /usr/lib64/libboost_program_options.so.2...done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so
Reading symbols from /usr/lib64/qpid/daemon/rdma.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/rdma.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/rdma.so
Reading symbols from /usr/lib64/librdmawrap.so.0...Reading symbols from /usr/lib/debug/usr/lib64/librdmawrap.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/librdmawrap.so.0
Reading symbols from /usr/lib64/librdmacm.so.1...done.
Loaded symbols for /usr/lib64/librdmacm.so.1
Reading symbols from /usr/lib64/libibverbs.so.1...done.
Loaded symbols for /usr/lib64/libibverbs.so.1
Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/cluster.so
Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/openais/libcpg.so.2
Reading symbols from /usr/lib64/libcman.so.2...done.
Loaded symbols for /usr/lib64/libcman.so.2
Reading symbols from /usr/lib64/libqpidclient.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.0
Reading symbols from /usr/lib64/qpid/client/rdmaconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/rdmaconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/rdmaconnector.so
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.0.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.0
Reading symbols from /usr/lib64/libnss3.so...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/xml.so
Reading symbols from /usr/lib64/libxerces-c.so.28...done.
Loaded symbols for /usr/lib64/libxerces-c.so.28
Reading symbols from /usr/lib64/libxqilla.so.3...done.
Loaded symbols for /usr/lib64/libxqilla.so.3
Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/acl.so
Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/ssl.so
Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...done.
Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so
Reading symbols from /usr/lib64/libdb_cxx-4.3.so...done.
Loaded symbols for /usr/lib64/libdb_cxx-4.3.so
Reading symbols from /usr/lib64/libaio.so.1...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Core was generated by `qpidd -p 5672 --auth no --log-enable info+ --cluster-name mrg-qe-02.lab.eng.brq'.
Program terminated with signal 6, Aborted.
[New process 2015]
[New process 2016]
[New process 2014]
[New process 2013]
[New process 2012]
[New process 2011]
[New process 2010]
[New process 2009]
[New process 2008]
[New process 2006]
[New process 2005]
[New process 2004]
[New process 2003]
#0  0x00000036e0430215 in raise () from /lib64/libc.so.6
(gdb) info threads
  13 process 2003  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
  12 process 2004  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11 process 2005  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 process 2006  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 process 2008  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 process 2009  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
  7 process 2010  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
  6 process 2011  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
  5 process 2012  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
  4 process 2013  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
  3 process 2014  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
  2 process 2016  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
* 1 process 2015  0x00000036e0430215 in raise () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 13 (process 2003):
#0  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b238bd1dc95 in qpid::sys::Poller::modFd (this=0x1a986740, handle=<value optimized out>, dir=qpid::sys::Poller::NONE)
    at qpid/sys/epoll/EpollPoller.cpp:336
#2  0x00002b238bd68a8f in qpid::sys::DispatchHandle::processEvent (this=0x1a98d4e0, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:479
#3  0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#4  0x00002b238b857b9e in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:319
#5  0x00000000004069b8 in QpiddBroker::execute (this=<value optimized out>, options=0x1a96ed30) at posix/QpiddBroker.cpp:166
#6  0x00000000004054a8 in main (argc=11, argv=0x7fff1f331c48) at qpidd.cpp:77

Thread 12 (process 2004):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b238b920139 in qpid::broker::Timer::run (this=0x2b238f73d140) at qpid/sys/posix/Condition.h:69
#2  0x00002b238bd14cea in runRunnable (p=0x2b238f73d174) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 11 (process 2005):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b238b920139 in qpid::broker::Timer::run (this=0x1a989d98) at qpid/sys/posix/Condition.h:69
#2  0x00002b238bd14cea in runRunnable (p=0x1a989dcc) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 10 (process 2006):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b238b920139 in qpid::broker::Timer::run (this=0x1a989e60) at qpid/sys/posix/Condition.h:69
#2  0x00002b238bd14cea in runRunnable (p=0x1a989e94) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 9 (process 2008):
#0  0x00000036e100ab00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b238b920139 in qpid::broker::Timer::run (this=0x1a996af0) at qpid/sys/posix/Condition.h:69
---Type <return> to continue, or q <return> to quit---
#2  0x00002b238bd14cea in runRunnable (p=0x1a996b24) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#4  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 8 (process 2009):
#0  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b238bd1dc95 in qpid::sys::Poller::modFd (this=0x1a986740, handle=<value optimized out>, dir=qpid::sys::Poller::NONE)
    at qpid/sys/epoll/EpollPoller.cpp:336
#2  0x00002b238bd68a8f in qpid::sys::DispatchHandle::processEvent (this=0x2aaab4b27bb8, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:479
#3  0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#4  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#5  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 7 (process 2010):
#0  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b238bd1e0dd in qpid::sys::Poller::wait (this=0x1a986740, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00002b238bd1ec87 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 6 (process 2011):
#0  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b238bd1dc95 in qpid::sys::Poller::modFd (this=0x1a986740, handle=<value optimized out>, dir=qpid::sys::Poller::INOUT)
    at qpid/sys/epoll/EpollPoller.cpp:336
#2  0x00002b238bd68414 in qpid::sys::DispatchHandle::rewatchWrite (this=0x2aaab40b82e8) at qpid/sys/DispatchHandle.cpp:187
#3  0x00002b238b8515cf in qpid::amqp_0_10::Connection::send (this=0x2aaab09017e8, f=@0x462b3960) at qpid/amqp_0_10/Connection.cpp:127
#4  0x00002b238cae2496 in qpid::cluster::OutputInterceptor::send (this=0x2aaab01ad0f0, f=@0x462b3960)
    at qpid/cluster/OutputInterceptor.cpp:54
#5  0x00002b238bd582de in qpid::framing::Proxy::send (this=0x2aaab08d1f48, b=@0x462b39a0) at qpid/framing/Proxy.cpp:37
#6  0x00002b238bcd0394 in qpid::framing::AMQP_AllProxy::Connection::openOk (this=0x2aaab08d1f48, knownHosts=@0x462b3a30)
    at gen/qpid/framing/AMQP_AllProxy.cpp:194
#7  0x00002b238b892782 in qpid::broker::ConnectionHandler::Handler::open (this=0x2aaab08d1f40)
---Type <return> to continue, or q <return> to quit---
    at qpid/broker/ConnectionHandler.cpp:191
#8  0x00002b238bcf4d70 in qpid::framing::AMQP_AllOperations::ConnectionHandler::Invoker::visit (this=0x462b3b30, body=@0x2aaaac645210)
    at gen/qpid/framing/ConnectionOpenBody.h:74
#9  0x00002b238b8933ce in qpid::framing::invoke<qpid::framing::AMQP_AllOperations::ConnectionHandler> (target=<value optimized out>,
    body=@0x28) at qpid/framing/Invoker.h:67
#10 0x00002b238b88ffea in qpid::broker::ConnectionHandler::handle (this=0x2aaab01ad2d0, frame=@0x462b42d0)
    at qpid/broker/ConnectionHandler.cpp:69
#11 0x00002b238b889ace in qpid::broker::Connection::received (this=0x2aaab01ad188, frame=@0x462b42d0)
    at qpid/broker/Connection.cpp:143
#12 0x00002b238cabddbc in qpid::cluster::Connection::deliveredFrame (this=0x2aaab01ad090, f=@0x462b42c0)
    at qpid/cluster/Connection.cpp:160
#13 0x00002b238ca94e69 in qpid::cluster::Cluster::processFrame (this=0x1a98c030, e=@0x462b42c0, l=@0x462b4330)
    at qpid/cluster/Cluster.cpp:440
#14 0x00002b238ca96385 in qpid::cluster::Cluster::deliveredFrame (this=0x1a98c030, efConst=<value optimized out>)
    at qpid/cluster/Cluster.cpp:420
#15 0x00002b238caa2c7a in boost::function1<void, qpid::cluster::EventFrame const&, std::allocator<void> >::operator() (this=0x0,
    a0=@0x3) at /usr/include/boost/function/function_template.hpp:576
#16 0x00002b238caa990b in qpid::cluster::PollableQueue<qpid::cluster::EventFrame>::handleBatch (this=0x1a98c4d0, values=@0x1a98c588)
    at qpid/cluster/PollableQueue.h:52
#17 0x00002b238ca9f496 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, boost::_mfi::mf1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::PollableQueue<qpid::cluster::EventFrame>*>, boost::arg<1> > >, __gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>::invoke (
    function_obj_ptr=<value optimized out>, a0=@0x3) at /usr/include/boost/bind/mem_fn_template.hpp:149
#18 0x00002b238caa247a in boost::function1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&, std::allocator<void> >::operator() (this=0x0, a0=@0x3) at /usr/include/boost/function/function_template.hpp:576
#19 0x00002b238caa91ec in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::process (this=0x1a98c4d0)
    at qpid/sys/PollableQueue.h:153
#20 0x00002b238caaa70d in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::dispatch (this=0x1a98c4d0, cond=@0x1a98c540)
    at qpid/sys/PollableQueue.h:138
#21 0x00002b238bd1c10f in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() (
    this=0x0, a0=@0x3) at /usr/include/boost/function/function_template.hpp:576
---Type <return> to continue, or q <return> to quit---
#22 0x00002b238bd6ad07 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x0, a0=@0x3) at /usr/include/boost/function/function_template.hpp:576
#23 0x00002b238bd68a0b in qpid::sys::DispatchHandle::processEvent (this=0x1a98dd50, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#24 0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#25 0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#26 0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#27 0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 5 (process 2012):
#0  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b238bd1dc95 in qpid::sys::Poller::modFd (this=0x1a986740, handle=<value optimized out>, dir=qpid::sys::Poller::NONE)
    at qpid/sys/epoll/EpollPoller.cpp:336
#2  0x00002b238bd68a8f in qpid::sys::DispatchHandle::processEvent (this=0x1a98cbc0, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:479
#3  0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#4  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#5  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 4 (process 2013):
#0  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b238bd1e0dd in qpid::sys::Poller::wait (this=0x1a986740, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00002b238bd1ec87 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#5  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 3 (process 2014):
#0  0x00000036e04d3498 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b238bd1e0dd in qpid::sys::Poller::wait (this=0x1a986740, timeout=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:439
#2  0x00002b238bd1ec87 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/epoll/EpollPoller.cpp:405
#3  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#4  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 2 (process 2016):
#0  0x00000036e04d341a in epoll_ctl () from /lib64/libc.so.6
#1  0x00002b238bd1dc95 in qpid::sys::Poller::modFd (this=0x1a986740, handle=<value optimized out>, dir=qpid::sys::Poller::NONE)
    at qpid/sys/epoll/EpollPoller.cpp:336
#2  0x00002b238bd68a8f in qpid::sys::DispatchHandle::processEvent (this=0x2aaab8035d38, type=qpid::sys::Poller::WRITABLE)
    at qpid/sys/DispatchHandle.cpp:479
#3  0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#4  0x00002b238bd14cea in runRunnable (p=0x5) at qpid/sys/posix/Thread.cpp:35
#5  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#6  0x00000036e04d30ad in clone () from /lib64/libc.so.6

Thread 1 (process 2015):
#0  0x00000036e0430215 in raise () from /lib64/libc.so.6
#1  0x00000036e0431cc0 in abort () from /lib64/libc.so.6
#2  0x00000036ecabec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x00000036ecabcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000036ecabcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x00000036ecabceca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00002b238bd688d3 in qpid::sys::DispatchHandle::processEvent (this=<value optimized out>, type=<value optimized out>)
    at qpid/sys/posix/Mutex.h:120
#7  0x00002b238bd1ecb3 in qpid::sys::Poller::run (this=0x1a986740) at qpid/sys/Poller.h:122
#8  0x00002b238bd14cea in runRunnable (p=0x7d3) at qpid/sys/posix/Thread.cpp:35
#9  0x00000036e1006367 in start_thread () from /lib64/libpthread.so.0
#10 0x00000036e04d30ad in clone () from /lib64/libc.so.6
Comment 2 Frantisek Reznicek 2009-07-09 13:28:19 UTC 
Created attachment 351073 [details]
bz510475
reproducer

The issue reproducer
run it using './run.sh 4 100' or './run.sh 4 75'
Comment 3 Andrew Stitcher 2009-07-10 04:04:20 UTC 
I ran the reproducer for nearly 10 hours without seeing any broker drop a core file.

[My test box is a 2 cpu 1.6Ghz box, so a lot less powerful than the test box used by freznice]
Comment 4 Frantisek Reznicek 2009-07-13 08:00:07 UTC 
Another (hopefully more detailed) abort backtrace on qpid*-0.5.752581-26.el5 packages (rhel5.3 i386):

GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done.
done.
...
Reading symbols from /usr/lib/libxqilla.so.3...done.
Loaded symbols for /usr/lib/libxqilla.so.3
Core was generated by `qpidd -p 5672 --auth no --log-enable info+ --cluster-name mrg-qe-01.lab.eng.brq'.
Program terminated with signal 6, Aborted.
[New process 29297]
[New process 29303]
[New process 29302]
[New process 29301]
[New process 29300]
[New process 29299]
[New process 29298]
[New process 29296]
[New process 29294]
[New process 29293]
[New process 29292]
[New process 29291]
[New process 29290]
#0  0x00323410 in __kernel_vsyscall ()
(gdb)
Thread 13 (process 29290):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00b40991 in qpid::broker::Broker::run (this=0x88ed0e8) at qpid/broker/Broker.cpp:319
#6  0x0804dd44 in QpiddBroker::execute (this=0xbfa6f395, options=0x88eb830) at posix/QpiddBroker.cpp:166
#7  0x0804c6c7 in main (argc=11, argv=0xbfa6f444) at qpidd.cpp:77

Thread 12 (process 29291):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x0028a8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00401b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00c19dbe in qpid::broker::Timer::run (this=0xb7f3f0b0) at qpid/sys/posix/Condition.h:69
#4  0x00647a41 in runRunnable (p=0xb7f3f0b0) at qpid/sys/posix/Thread.cpp:35
#5  0x0028649b in start_thread () from /lib/libpthread.so.0
#6  0x003f542e in clone () from /lib/libc.so.6

Thread 11 (process 29292):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x0028a8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00401b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00c19dbe in qpid::broker::Timer::run (this=0x88ed2ec) at qpid/sys/posix/Condition.h:69
#4  0x00647a41 in runRunnable (p=0x88ed2ec) at qpid/sys/posix/Thread.cpp:35
#5  0x0028649b in start_thread () from /lib/libpthread.so.0
#6  0x003f542e in clone () from /lib/libc.so.6

Thread 10 (process 29293):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x0028a8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00401b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00c19dbe in qpid::broker::Timer::run (this=0x88ed36c) at qpid/sys/posix/Condition.h:69
#4  0x00647a41 in runRunnable (p=0x88ed36c) at qpid/sys/posix/Thread.cpp:35
#5  0x0028649b in start_thread () from /lib/libpthread.so.0
#6  0x003f542e in clone () from /lib/libc.so.6

Thread 9 (process 29294):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x0028a8c2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00401b84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00c19dbe in qpid::broker::Timer::run (this=0x88f3478) at qpid/sys/posix/Condition.h:69
#4  0x00647a41 in runRunnable (p=0x88f3478) at qpid/sys/posix/Thread.cpp:35
#5  0x0028649b in start_thread () from /lib/libpthread.so.0
#6  0x003f542e in clone () from /lib/libc.so.6

Thread 8 (process 29296):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 7 (process 29298):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f6ceb in semop () from /lib/libc.so.6
#2  0x0049573a in openais_msg_send_reply_receive (ipc_context=<value optimized out>, iov=<value optimized out>,
    iov_len=<value optimized out>, res_msg=) at util.c:623
#3  0x00496357 in cpg_mcast_joined (handle=<value optimized out>, guarantee=<value optimized out>, iovec=<value optimized out>,
    iov_len=<value optimized out>) at cpg.c:513
#4  0x00871380 in qpid::cluster::Cpg::mcast (this=0x88f3960, iov=0xb3f60d48, iovLen=1) at qpid/cluster/Cpg.cpp:124
#5  0x0088f1d9 in qpid::cluster::Multicaster::sendMcast (this=0x88f3a54, values=@0x88f3af4) at qpid/cluster/Multicaster.cpp:79
#6  0x00890fe8 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, boost::_mfi::mf1<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, qpid::cluster::Multicaster, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::Multicaster*>, boost::arg<1> > >, __gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&>::invoke (function_obj_ptr=
      {obj_ptr = 0x88f3710, const_obj_ptr = 0x88f3710, func_ptr = 0x88f3710, data = "\020"}, a0=@0x88f3af4)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#7  0x0084c8cb in boost::function1<__gnu_cxx::__normal_iterator<qpid::cluster::Event const*, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > >, std::vector<qpid::cluster::Event, std::allocator<qpid::cluster::Event> > const&, std::allocator<void> >::operator() (this=0x88f3ac4, a0=@0x88f3af4) at /usr/include/boost/function/function_template.hpp:576
#8  0x008538ed in qpid::sys::PollableQueue<qpid::cluster::Event>::process (this=0x88f3a7c) at qpid/sys/PollableQueue.h:153
#9  0x008556b8 in qpid::sys::PollableQueue<qpid::cluster::Event>::dispatch (this=0x88f3a7c, cond=@0x88f3ad0)
    at qpid/sys/PollableQueue.h:138
#10 0x0084881d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::Event>, qpid::sys::PollableCondition&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<qpid::cluster::Event>*>, boost::arg<1> > >, void, qpid::sys::PollableCondition&>::invoke (function_obj_ptr=
      {obj_ptr = 0x88f3740, const_obj_ptr = 0x88f3740, func_ptr = 0x88f3740, data = "@"}, a0=@0x88f3ad0)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#11 0x0064f8d7 in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() (
    this=0x88ef3e8, a0=@0x88f3ad0) at /usr/include/boost/function/function_template.hpp:576
#12 0x0064bb37 in qpid::sys::PollableConditionPrivate::dispatch (this=0x88ef3e0) at qpid/sys/posix/PollableCondition.cpp:89
#13 0x0064e86d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0x88f3838, const_obj_ptr = 0x88f3838, func_ptr = 0x88f3838, data = "8"}, a0=@0x88f3788)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#14 0x006a6204 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x88f3790, a0=@0x88f3788) at /usr/include/boost/function/function_template.hpp:576
#15 0x006a58e9 in qpid::sys::DispatchHandle::processEvent (this=0x88f3788, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#16 0x00652b04 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/Poller.h:122
#17 0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#18 0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#19 0x0028649b in start_thread () from /lib/libpthread.so.0
#20 0x003f542e in clone () from /lib/libc.so.6

Thread 6 (process 29299):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 5 (process 29300):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 4 (process 29301):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 3 (process 29302):
#0  0x00216571 in operator delete () from /usr/lib/libstdc++.so.6
#1  0x006a7cdd in ~deque (this=0xb175d248) at /usr/include/c++/4.1.2/ext/new_allocator.h:94
#2  0x006a52b5 in qpid::sys::DispatchHandle::processEvent (this=0x88f42d0, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:383
#3  0x00652b04 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/Poller.h:122
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 2 (process 29303):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x003f5aa6 in epoll_wait () from /lib/libc.so.6
#2  0x00651ab9 in qpid::sys::Poller::wait (this=0x88dfae0, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00652ae6 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#5  0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#6  0x0028649b in start_thread () from /lib/libpthread.so.0
#7  0x003f542e in clone () from /lib/libc.so.6

Thread 1 (process 29297):
#0  0x00323410 in __kernel_vsyscall ()
#1  0x0034cd80 in raise () from /lib/libc.so.6
#2  0x0034e691 in abort () from /lib/libc.so.6
#3  0x00219aa0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x00217505 in ?? () from /usr/lib/libstdc++.so.6
#5  0x00217542 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x0021767a in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x006a5856 in qpid::sys::DispatchHandle::processEvent (this=0xae5eaa6c, type=qpid::sys::Poller::READABLE)
    at qpid/sys/posix/Mutex.h:120
#8  0x00652b04 in qpid::sys::Poller::run (this=0x88dfae0) at qpid/sys/Poller.h:122
#9  0x006a2bc4 in qpid::sys::Dispatcher::run (this=0xbfa6f0b4) at qpid/sys/Dispatcher.cpp:37
#10 0x00647a41 in runRunnable (p=0xbfa6f0b4) at qpid/sys/posix/Thread.cpp:35
#11 0x0028649b in start_thread () from /lib/libpthread.so.0
#12 0x003f542e in clone () from /lib/libc.so.6
(gdb) quit

Please find compressed core file and backtrace (for debugging) stored here:
mrg-qe-01.lab.eng.brq.redhat.com:/root/_/bz506758/bz510475_core.tar.bz2
Comment 6 Andrew Wright 2009-12-23 16:47:57 UTC 
Not sure if it's of any use, but I've seen a very similar crash with a 2-node, MRG 1.2 build, running on RHEL 5.4. Various logs below. Let me know if you'd like the core as well.

We make heavy use of durable subscriptions on a topic exchange, as well as LVQs and ring queues. Particularly the durable topics seem to make cluster failover/rejoin unreliable.

Cheers,
Andrew


TEST [amqmadm@vrtstamqb02 core]$ rpm -qa | grep -E '(qpid|rhm|openais)' | sort -u
openais-0.80.6-8.el5_4.1
python-qpid-0.5.758389-1.el5
qpidc-0.5.752581-34.el5
qpidc-devel-0.5.752581-34.el5
qpidc-perftest-0.5.752581-34.el5
qpidc-rdma-0.5.752581-34.el5
qpidc-ssl-0.5.752581-34.el5
qpidd-0.5.752581-34.el5
qpidd-acl-0.5.752581-34.el5
qpidd-cluster-0.5.752581-34.el5
qpidd-devel-0.5.752581-34.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-34.el5
qpidd-ssl-0.5.752581-34.el5
qpidd-xml-0.5.752581-34.el5
qpid-java-client-0.5.751061-9.el5
qpid-java-common-0.5.751061-9.el5
qpid-java-qman-0.5.751061-9.el5
rhm-0.5.3206-27.el5
rhm-docs-0.5.756148-2.el5
ruby-qpid-0.4.749380-2.el5


Last entry in log:
2717949: 2009-dec-23 11:20:46 trace 10.34.22.65:20831(READY/error) DLVR: Event[10.34.22.65:20831-314 data 18 bytes]
2717950: 2009-dec-23 11:20:46 info 10.34.22.65:20831(READY/error) error 1371767 resolved with 10.34.22.65:20831
2717951: 2009-dec-23 11:20:46 info 10.34.22.65:20831(READY) error 1371767 resolved.
2717952: 2009-dec-23 11:20:46 trace 10.34.22.65:20831(READY) DLVR 1371768: Frame[BEbe; channel=638; {SessionCommandPointBody: command-id=32; command-offset=0; }] data 10.34.22.65:20831-314
2717953: 2009-dec-23 11:20:46 trace 10.34.22.65:20831(READY) DLVR 1371769: Frame[BEbe; channel=638; {MessageSetFlowModeBody: destination=4; flow-mode=1; }] data 10.34.22.65:20831-314
2717954: 2009-dec-23 11:20:46 debug Exception constructed: Invalid argument (qpid/sys/posix/Mutex.h:120)



(gdb) backtrace
#0  0x0000003c7a230265 in raise () from /lib64/libc.so.6
#1  0x0000003c7a231d10 in abort () from /lib64/libc.so.6
#2  0x0000003c7e2bec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x0000003c7e2bcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x0000003c7e2bcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x0000003c7e2bceca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x0000003c7c7ca493 in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.0
#7  0x0000003c7c780b93 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.0
#8  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#9  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6
(gdb) info threads
  6 process 20831  0x0000003c7aa0d174 in __lll_lock_wait () from /lib64/libpthread.so.0
  5 process 20832  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 process 20833  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 process 20834  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 process 20836  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 process 20837  0x0000003c7a230265 in raise () from /lib64/libc.so.6
(gdb) thread apply all bt

Thread 6 (process 20831):
#0  0x0000003c7aa0d174 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x0000003c7aa08ae5 in _L_lock_1127 () from /lib64/libpthread.so.0
#2  0x0000003c7aa089e3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000003c7c7bd6cc in qpid::log::Logger::log () from /usr/lib64/libqpidcommon.so.0
#4  0x0000003c7c7c3d5c in qpid::log::Statement::log () from /usr/lib64/libqpidcommon.so.0
#5  0x00002b74a6d599f6 in qpid::cluster::Cluster::processFrame () from /var/lib/qpid-dev1/modules/cluster.so
#6  0x00002b74a6d5b39d in qpid::cluster::Cluster::deliveredFrame () from /var/lib/qpid-dev1/modules/cluster.so
#7  0x00002b74a6d65d5a in boost::function1<void, qpid::cluster::EventFrame const&, std::allocator<void> >::operator() () from /var/lib/qpid-dev1/modules/cluster.so
#8  0x00002b74a6d6f74b in qpid::cluster::PollableQueue<qpid::cluster::EventFrame>::handleBatch () from /var/lib/qpid-dev1/modules/cluster.so
#9  0x00002b74a6d625d6 in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, boost::_mfi::mf1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::PollableQueue<qpid::cluster::EventFrame>*>, boost::arg<1> > >, __gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&>::invoke ()
   from /var/lib/qpid-dev1/modules/cluster.so
#10 0x00002b74a6d65e6a in boost::function1<__gnu_cxx::__normal_iterator<qpid::cluster::EventFrame const*, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > >, std::vector<qpid::cluster::EventFrame, std::allocator<qpid::cluster::EventFrame> > const&, std::allocator<void> >::operator() () from /var/lib/qpid-dev1/modules/cluster.so
#11 0x00002b74a6d6c5bc in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::process () from /var/lib/qpid-dev1/modules/cluster.so
#12 0x00002b74a6d6d8fd in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::dispatch () from /var/lib/qpid-dev1/modules/cluster.so
#13 0x0000003c7c77dfef in boost::function1<void, qpid::sys::PollableCondition&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.0
#14 0x0000003c7c7cc8c7 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.0
#15 0x0000003c7c7ca5cb in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.0
---Type <return> to continue, or q <return> to quit---
#16 0x0000003c7c780b93 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.0
#17 0x0000003c7cccb75e in qpid::broker::Broker::run () from /usr/lib64/libqpidbroker.so.0
#18 0x000000000040917b in ?? ()
#19 0x0000003c7cd0926e in qpid::broker::Daemon::fork () from /usr/lib64/libqpidbroker.so.0
#20 0x0000000000406a95 in ?? ()
#21 0x00000000004054a8 in __cxa_pure_virtual ()
#22 0x0000003c7a21d994 in __libc_start_main () from /lib64/libc.so.6
#23 0x0000000000404f29 in __cxa_pure_virtual ()
#24 0x00007fffa58499e8 in ?? ()
#25 0x0000000000000000 in ?? ()

Thread 5 (process 20832):
#0  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c7cd96ef9 in qpid::broker::Timer::run () from /usr/lib64/libqpidbroker.so.0
#2  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 20833):
#0  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c7cd96ef9 in qpid::broker::Timer::run () from /usr/lib64/libqpidbroker.so.0
#2  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 3 (process 20834):
#0  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c7cd96ef9 in qpid::broker::Timer::run () from /usr/lib64/libqpidbroker.so.0
#2  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 20836):
#0  0x0000003c7aa0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000003c7cd96ef9 in qpid::broker::Timer::run () from /usr/lib64/libqpidbroker.so.0
#2  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#3  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 20837):
#0  0x0000003c7a230265 in raise () from /lib64/libc.so.6
#1  0x0000003c7a231d10 in abort () from /lib64/libc.so.6
#2  0x0000003c7e2bec44 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib64/libstdc++.so.6
#3  0x0000003c7e2bcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x0000003c7e2bcde3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x0000003c7e2bceca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x0000003c7c7ca493 in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.0
#7  0x0000003c7c780b93 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.0
---Type <return> to continue, or q <return> to quit---
#8  0x0000003c7c776bca in ?? () from /usr/lib64/libqpidcommon.so.0
#9  0x0000003c7aa064a7 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003c7a2d3c2d in clone () from /lib64/libc.so.6
Comment 7 Andrew Stitcher 2010-04-13 18:46:57 UTC 
It's very likely this bug will have been fixed by svn r771338 which occurred on trunk after the 1.1 branch
Comment 8 Frantisek Reznicek 2010-06-09 14:33:54 UTC 
The retest triggered heartbeat issue, see depends on bug list. Waiting for new package set.
Comment 9 Frantisek Reznicek 2010-06-17 10:33:40 UTC 
Heartbeat issue is gone, retesting
Comment 11 Frantisek Reznicek 2010-10-04 11:35:39 UTC 
The issue has been fixed, no such behavior seen over very long on multiple machines run on RHEL 5.5 i386 / x86_64 on packages:

python-qmf-0.7.946106-13.el5
python-qpid-0.7.946106-14.el5
qmf-0.7.946106-17.el5
qmf-devel-0.7.946106-17.el5
qpid-cpp-client-0.7.946106-17.el5
qpid-cpp-client-devel-0.7.946106-17.el5
qpid-cpp-client-devel-docs-0.7.946106-17.el5
qpid-cpp-client-rdma-0.7.946106-17.el5
qpid-cpp-client-ssl-0.7.946106-17.el5
qpid-cpp-mrg-debuginfo-0.7.946106-17.el5
qpid-cpp-server-0.7.946106-17.el5
qpid-cpp-server-cluster-0.7.946106-17.el5
qpid-cpp-server-devel-0.7.946106-17.el5
qpid-cpp-server-rdma-0.7.946106-17.el5
qpid-cpp-server-ssl-0.7.946106-17.el5
qpid-cpp-server-store-0.7.946106-17.el5
qpid-cpp-server-xml-0.7.946106-17.el5
qpid-dotnet-0.4.738274-2.el5
qpid-java-client-0.7.946106-10.el5
qpid-java-common-0.7.946106-10.el5
qpid-tools-0.7.946106-11.el5
ruby-qmf-0.7.946106-17.el5
ruby-qpid-0.7.946106-2.el5

-> VERIFIED
Comment 12 Jaromir Hradilek 2010-10-08 10:37:01 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Under certain circumstances, an unhandled exception may have prevented a clustered qpidd service from starting. With this update, all exception are now handled properly, and the qpidd service always starts as expected.
Comment 14 errata-xmlrpc 2010-10-14 15:59:23 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html