To reproduce, run federation tests in a loop. Sample stack trace below. Thread 4 holds LinkRegistry lock and is waiting for lock on a particular Link object. Thread 3 holds a lock on a particular Link object and is waiting for the LinkRegistry lock. If the Link object whose lock thread 4 is waiting for is the same one as is held by thread 3 this is a deadlock. Thread 3 also holds the ManagementBrokers user lock which further locks out other threads. Thread 4 (Thread 1132931392 (LWP 6221)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad7280946ba in qpid::broker::Link::closed () #4 0x00002ad728099689 in qpid::broker::LinkRegistry::notifyClosed () #5 0x00002ad72806d769 in qpid::broker::Connection::~Connection$delete () #6 0x00002ad7280408b1 in qpid::amqp_0_10::Connection::~Connection$delete () #7 0x00002ad72854c323 in qpid::sys::AsynchIOHandler::~AsynchIOHandler$delete #8 0x00002ad72854ccf0 in qpid::sys::AsynchIOHandler::closedSocket () #9 0x00002ad728500a2f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::Socket const&, std::allocator<boost::function_base> >::operator() () #10 0x00002ad7284fe1f8 in qpid::sys::posix::AsynchIO::writeable () #11 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #12 0x00002ad72854eabc in qpid::sys::DispatchHandle::processEvent () #13 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run () #14 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #15 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #16 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 1143421248 (LWP 6222)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad72809a15d in qpid::broker::LinkRegistry::destroy () #4 0x00002ad728093cd5 in qpid::broker::Link::destroy () #5 0x00002ad7280943cb in qpid::broker::Link::ManagementMethod () #6 0x00002ad72801f714 in qmf::org::apache::qpid::broker::Link::doMethod () #7 0x00002ad7280f06ae in qpid::management::ManagementBroker::handleMethodRequestLH () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0 #8 0x00002ad7280f624c in qpid::management::ManagementBroker::dispatchAgentCommandLH () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0 #9 0x00002ad7280f7aeb in qpid::management::ManagementBroker::dispatchCommand #10 0x00002ad7280faaa8 in qpid::broker::ManagementExchange::route () #11 0x00002ad7280be263 in qpid::broker::SemanticState::route () #12 0x00002ad7280bf761 in qpid::broker::SemanticState::handle () #13 0x00002ad7280d6db9 in qpid::broker::SessionState::handleContent () #14 0x00002ad7280d8d48 in qpid::broker::SessionState::handleIn () #15 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn () #16 0x00002ad72806bead in qpid::broker::Connection::received () #17 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode () #18 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff () #19 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () #20 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable () #21 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #22 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent () #23 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run () #24 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #25 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #26 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 1101461824 (LWP 6218)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad728097ac6 in qpid::broker::LinkRegistry::periodicMaintenance () #4 0x00002ad728097c22 in qpid::broker::LinkRegistry::Periodic::fire () #5 0x00002ad7280df359 in qpid::broker::Timer::run () #6 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #7 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #8 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 1090971968 (LWP 6217)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad7280f3f8c in qpid::management::ManagementBroker::periodicProcessing () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0 #4 0x00002ad7280f4e58 in qpid::management::ManagementBroker::Periodic::fire () #5 0x00002ad7280df359 in qpid::broker::Timer::run () #6 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #7 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #8 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 1153911104 (LWP 6223)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad7280f3afe in qpid::management::ManagementBroker::raiseEvent () #4 0x00002ad7280cca2c in qpid::broker::SessionAdapter::ExchangeHandlerImpl::declare () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0 #5 0x00002ad7284c9975 in qpid::framing::AMQP_ServerOperations::ExchangeHandler::Invoker::visit () from /home/gordon/qpid/cpp/src/.libs/libqpidcommon.so.0 #6 0x00002ad7284cf627 in qpid::framing::AMQP_ServerOperations::Invoker::visit #7 0x00002ad7280d9e7e in qpid::framing::invoke<qpid::broker::SessionAdapter> #8 0x00002ad7280d87a8 in qpid::broker::SessionState::handleCommand () #9 0x00002ad7280d8d88 in qpid::broker::SessionState::handleIn () #10 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn () #11 0x00002ad7280720e1 in qpid::broker::ConnectionHandler::handle () #12 0x00002ad72806be70 in qpid::broker::Connection::received () #13 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode () #14 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff () #15 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () #16 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable () #17 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #18 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent () #19 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run () #20 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #21 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #22 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 1122441536 (LWP 6220)): #0 0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0 #2 0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ad7280f3afe in qpid::management::ManagementBroker::raiseEvent () #4 0x00002ad7280d1c6e in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0 #5 0x00002ad7284c96e6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /home/gordon/qpid/cpp/src/.libs/libqpidcommon.so.0 #6 0x00002ad7284cfd87 in qpid::framing::AMQP_ServerOperations::Invoker::visit #7 0x00002ad7280d9e7e in qpid::framing::invoke<qpid::broker::SessionAdapter> #8 0x00002ad7280d87a8 in qpid::broker::SessionState::handleCommand () #9 0x00002ad7280d8d88 in qpid::broker::SessionState::handleIn () #10 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn () #11 0x00002ad7280720e1 in qpid::broker::ConnectionHandler::handle () #12 0x00002ad72806be70 in qpid::broker::Connection::received () #13 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode () #14 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff () #15 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () #16 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable () #17 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #18 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent () #19 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run () #20 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #21 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #22 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 1111951680 (LWP 6219)): #0 0x000000393700a687 in pthread_cond_timedwait@@GLIBC_2.3.2 () #1 0x00002ad7280df416 in qpid::broker::Timer::run () #2 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable () #3 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0 #4 0x00000039364d1b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 47103588016672 (LWP 6203)): #0 0x00000039370075b5 in pthread_join () from /lib64/libpthread.so.0 #1 0x00002ad728502ca3 in qpid::sys::Thread::join () #2 0x00002ad728041c5d in qpid::broker::Broker::run () #3 0x0000000000406ff9 in QpiddBroker::execute () #4 0x000000000040529c in main () #0 0x00000039370075b5 in pthread_join () from /lib64/libpthread.so.0
This bug was fixed in upstream SVN revision 713161 on 11/11/2008.
No dead-locking observed anymore when ran federation tests in loop. Validated on RHEL 5.2 x86_64 from svn trunk rev 720183 ->VERIFIED
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-0035.html