Bug 471095 - Deadlock between Link and LinkRegistry locks
Deadlock between Link and LinkRegistry locks
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
1.0
All Linux
high Severity high
: 1.1
: ---
Assigned To: Ted Ross
Kim van der Riet
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-11 14:06 EST by Gordon Sim
Modified: 2009-02-04 10:37 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-02-04 10:37:05 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Gordon Sim 2008-11-11 14:06:48 EST
To reproduce, run federation tests in a loop. Sample stack trace below.

Thread 4 holds LinkRegistry lock and is waiting for lock on a
particular Link object. Thread 3 holds a lock on a particular Link
object and is waiting for the LinkRegistry lock. If the Link object
whose lock thread 4 is waiting for is the same one as is held by
thread 3 this is a deadlock.

Thread 3 also holds the ManagementBrokers user lock which further locks out other threads.

Thread 4 (Thread 1132931392 (LWP 6221)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad7280946ba in qpid::broker::Link::closed ()
#4  0x00002ad728099689 in qpid::broker::LinkRegistry::notifyClosed ()
#5  0x00002ad72806d769 in qpid::broker::Connection::~Connection$delete ()
#6  0x00002ad7280408b1 in qpid::amqp_0_10::Connection::~Connection$delete ()
#7  0x00002ad72854c323 in qpid::sys::AsynchIOHandler::~AsynchIOHandler$delete
#8  0x00002ad72854ccf0 in qpid::sys::AsynchIOHandler::closedSocket ()
#9  0x00002ad728500a2f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::Socket const&, std::allocator<boost::function_base> >::operator() ()
#10 0x00002ad7284fe1f8 in qpid::sys::posix::AsynchIO::writeable ()
#11 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#12 0x00002ad72854eabc in qpid::sys::DispatchHandle::processEvent ()
#13 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run ()
#14 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#15 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#16 0x00000039364d1b6d in clone () from /lib64/libc.so.6


Thread 3 (Thread 1143421248 (LWP 6222)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad72809a15d in qpid::broker::LinkRegistry::destroy ()
#4  0x00002ad728093cd5 in qpid::broker::Link::destroy ()
#5  0x00002ad7280943cb in qpid::broker::Link::ManagementMethod ()
#6  0x00002ad72801f714 in qmf::org::apache::qpid::broker::Link::doMethod ()
#7  0x00002ad7280f06ae in qpid::management::ManagementBroker::handleMethodRequestLH () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0
#8  0x00002ad7280f624c in qpid::management::ManagementBroker::dispatchAgentCommandLH () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0
#9  0x00002ad7280f7aeb in qpid::management::ManagementBroker::dispatchCommand
#10 0x00002ad7280faaa8 in qpid::broker::ManagementExchange::route ()
#11 0x00002ad7280be263 in qpid::broker::SemanticState::route ()
#12 0x00002ad7280bf761 in qpid::broker::SemanticState::handle ()
#13 0x00002ad7280d6db9 in qpid::broker::SessionState::handleContent ()
#14 0x00002ad7280d8d48 in qpid::broker::SessionState::handleIn ()
#15 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn ()
#16 0x00002ad72806bead in qpid::broker::Connection::received ()
#17 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode ()
#18 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff ()
#19 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#20 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable ()
#21 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#22 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent ()
#23 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run ()
#24 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#25 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#26 0x00000039364d1b6d in clone () from /lib64/libc.so.6


Thread 7 (Thread 1101461824 (LWP 6218)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad728097ac6 in qpid::broker::LinkRegistry::periodicMaintenance ()
#4  0x00002ad728097c22 in qpid::broker::LinkRegistry::Periodic::fire ()
#5  0x00002ad7280df359 in qpid::broker::Timer::run ()
#6  0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#7  0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#8  0x00000039364d1b6d in clone () from /lib64/libc.so.6



Thread 8 (Thread 1090971968 (LWP 6217)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad7280f3f8c in qpid::management::ManagementBroker::periodicProcessing () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0
#4  0x00002ad7280f4e58 in qpid::management::ManagementBroker::Periodic::fire ()
#5  0x00002ad7280df359 in qpid::broker::Timer::run ()
#6  0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#7  0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#8  0x00000039364d1b6d in clone () from /lib64/libc.so.6



Thread 2 (Thread 1153911104 (LWP 6223)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad7280f3afe in qpid::management::ManagementBroker::raiseEvent ()
#4  0x00002ad7280cca2c in qpid::broker::SessionAdapter::ExchangeHandlerImpl::declare () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0
#5  0x00002ad7284c9975 in qpid::framing::AMQP_ServerOperations::ExchangeHandler::Invoker::visit () from /home/gordon/qpid/cpp/src/.libs/libqpidcommon.so.0
#6  0x00002ad7284cf627 in qpid::framing::AMQP_ServerOperations::Invoker::visit
#7  0x00002ad7280d9e7e in qpid::framing::invoke<qpid::broker::SessionAdapter>
#8  0x00002ad7280d87a8 in qpid::broker::SessionState::handleCommand ()
#9  0x00002ad7280d8d88 in qpid::broker::SessionState::handleIn ()
#10 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn ()
#11 0x00002ad7280720e1 in qpid::broker::ConnectionHandler::handle ()
#12 0x00002ad72806be70 in qpid::broker::Connection::received ()
#13 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode ()
#14 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff ()
#15 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#16 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable ()
#17 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#18 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent ()
#19 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run ()
#20 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#21 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#22 0x00000039364d1b6d in clone () from /lib64/libc.so.6


Thread 5 (Thread 1122441536 (LWP 6220)):
#0  0x000000393700c888 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039370088f6 in _L_mutex_lock_937 () from /lib64/libpthread.so.0
#2  0x0000003937008761 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ad7280f3afe in qpid::management::ManagementBroker::raiseEvent ()
#4  0x00002ad7280d1c6e in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /home/gordon/qpid/cpp/src/.libs/libqpidbroker.so.0
#5  0x00002ad7284c96e6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /home/gordon/qpid/cpp/src/.libs/libqpidcommon.so.0
#6  0x00002ad7284cfd87 in qpid::framing::AMQP_ServerOperations::Invoker::visit
#7  0x00002ad7280d9e7e in qpid::framing::invoke<qpid::broker::SessionAdapter>
#8  0x00002ad7280d87a8 in qpid::broker::SessionState::handleCommand ()
#9  0x00002ad7280d8d88 in qpid::broker::SessionState::handleIn ()
#10 0x00002ad728526079 in qpid::amqp_0_10::SessionHandler::handleIn ()
#11 0x00002ad7280720e1 in qpid::broker::ConnectionHandler::handle ()
#12 0x00002ad72806be70 in qpid::broker::Connection::received ()
#13 0x00002ad72803f97f in qpid::amqp_0_10::Connection::decode ()
#14 0x00002ad72854d843 in qpid::sys::AsynchIOHandler::readbuff ()
#15 0x00002ad7285008af in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ()
#16 0x00002ad7284feb13 in qpid::sys::posix::AsynchIO::readable ()
#17 0x00002ad72855039f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
#18 0x00002ad72854ec1f in qpid::sys::DispatchHandle::processEvent ()
#19 0x00002ad72854e7f8 in qpid::sys::Dispatcher::run ()
#20 0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#21 0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#22 0x00000039364d1b6d in clone () from /lib64/libc.so.6



Thread 6 (Thread 1111951680 (LWP 6219)):
#0  0x000000393700a687 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1  0x00002ad7280df416 in qpid::broker::Timer::run ()
#2  0x00002ad728502bea in qpid::sys::(anonymous namespace)::runRunnable ()
#3  0x00000039370062f7 in start_thread () from /lib64/libpthread.so.0
#4  0x00000039364d1b6d in clone () from /lib64/libc.so.6


Thread 1 (Thread 47103588016672 (LWP 6203)):
#0  0x00000039370075b5 in pthread_join () from /lib64/libpthread.so.0
#1  0x00002ad728502ca3 in qpid::sys::Thread::join ()
#2  0x00002ad728041c5d in qpid::broker::Broker::run ()
#3  0x0000000000406ff9 in QpiddBroker::execute ()
#4  0x000000000040529c in main ()
#0  0x00000039370075b5 in pthread_join () from /lib64/libpthread.so.0
Comment 2 Ted Ross 2008-11-25 08:14:56 EST
This bug was fixed in upstream SVN revision 713161 on 11/11/2008.
Comment 3 Frantisek Reznicek 2008-11-25 08:33:26 EST
No dead-locking observed anymore when ran federation tests in loop.
Validated on RHEL 5.2 x86_64 from svn trunk rev 720183
->VERIFIED
Comment 5 errata-xmlrpc 2009-02-04 10:37:05 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-0035.html

Note You need to log in before you can comment on or make changes to this bug.