+++ This bug was initially created as a clone of Bug #457239 +++ Sometimes a deadlock is achieved by having several connections open against the broker. This is experienced when doing queue_declare management call. To reproduce, start the broker like this: # qpidd --module-dir ../../storage/mrg-1.0/cpp/lib/.libs/ --auth no \ --tpl-wcache-page-size 128 --tpl-jfile-size-pgs 32 Use the reproducer for the bz456272 (mrg-team svn: mrg-team/people/dsommers/bz456272) and run it like this: # python ./bz456272.py -- Backtrace from qpidd -- Thread 9 (Thread 1089722688 (LWP 512)): #0 0x0000003ca980a687 in pthread_cond_timedwait@@GLIBC_2.3.2 () #1 0x00002ae0b98f2692 in qpid::broker::Timer::run () #2 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #3 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 1100212544 (LWP 513)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0b98f262c in qpid::broker::Timer::run () #2 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #3 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 1110702400 (LWP 514)): #0 0x0000003ca980a687 in pthread_cond_timedwait@@GLIBC_2.3.2 () #1 0x00002ae0b98f2692 in qpid::broker::Timer::run () #2 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #3 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 1121192256 (LWP 531)): #0 0x0000003ca980a687 in pthread_cond_timedwait@@GLIBC_2.3.2 () #1 0x00002ae0b98f2692 in qpid::broker::Timer::run () #2 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #3 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 1131682112 (LWP 532)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0ba4a0626 in __db_pthread_mutex_lock () #2 0x00002ae0ba528397 in __lock_get_internal () #3 0x00002ae0ba5284a2 in __lock_get () from /usr/lib64/libdb_cxx-4.3.so #4 0x00002ae0ba508c54 in __db_lget () from /usr/lib64/libdb_cxx-4.3.so #5 0x00002ae0ba4b042e in __bam_search () from /usr/lib64/libdb_cxx-4.3.so #6 0x00002ae0ba4a3987 in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #7 0x00002ae0ba4a4acd in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #8 0x00002ae0ba4fe669 in __db_c_get () from /usr/lib64/libdb_cxx-4.3.so #9 0x00002ae0ba4f9695 in __db_put () from /usr/lib64/libdb_cxx-4.3.so #10 0x00002ae0ba5056c9 in __db_put_pp () from /usr/lib64/libdb_cxx-4.3.so #11 0x00002ae0ba497350 in Db::put () from /usr/lib64/libdb_cxx-4.3.so #12 0x00002ae0ba1beeed in rhm::bdbstore::BdbMessageStore::create () #13 0x00002ae0ba1ce39d in rhm::bdbstore::BdbMessageStore::create () #14 0x00002ae0b98c2c58 in qpid::broker::MessageStoreModule::create () #15 0x00002ae0b9884625 in qpid::broker::Queue::create () #16 0x00002ae0b98e29ea in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /root/svn/qpid/cpp/src/.libs/libqpidbroker.so.0 #17 0x00002ae0b9c74df6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /root/svn/qpid/cpp/src/.libs/libqpidcommon.so.0 #18 0x00002ae0b9c7b497 in qpid::framing::AMQP_ServerOperations::Invoker::visit #19 0x00002ae0b98ec37e in qpid::framing::invoke<qpid::broker::SessionAdapter> #20 0x00002ae0b98e9f5c in qpid::broker::SessionState::handleCommand () #21 0x00002ae0b98ead9c in qpid::broker::SessionState::handle () #22 0x00002ae0b9cb6436 in qpid::amqp_0_10::SessionHandler::handleIn () #23 0x00002ae0b9895681 in qpid::broker::ConnectionHandler::handle () #24 0x00002ae0b988fba0 in qpid::broker::Connection::received () #25 0x00002ae0b986b4a0 in qpid::amqp_0_10::Connection::decode () #26 0x00002ae0b9ce237f in qpid::sys::AsynchIOHandler::readbuff () #27 0x00002ae0b9cb23ea in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIO::BufferBase*, std::allocator<boost::function_base> >::operator() () #28 0x00002ae0b9cacbc9 in qpid::sys::AsynchIO::readable () #29 0x00002ae0b9ce505f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #30 0x00002ae0b9ce4be9 in qpid::sys::DispatchHandle::dispatchCallbacks () #31 0x00002ae0b9ce4ce5 in qpid::sys::Dispatcher::run () #32 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #33 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #34 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 4 (Thread 1142171968 (LWP 533)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0ba4a0626 in __db_pthread_mutex_lock () #2 0x00002ae0ba528397 in __lock_get_internal () #3 0x00002ae0ba5288ff in __lock_vec () from /usr/lib64/libdb_cxx-4.3.so #4 0x00002ae0ba508b70 in __db_lget () from /usr/lib64/libdb_cxx-4.3.so #5 0x00002ae0ba4b0dd5 in __bam_search () from /usr/lib64/libdb_cxx-4.3.so #6 0x00002ae0ba4a3987 in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #7 0x00002ae0ba4a4acd in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #8 0x00002ae0ba4fe669 in __db_c_get () from /usr/lib64/libdb_cxx-4.3.so #9 0x00002ae0ba4f9695 in __db_put () from /usr/lib64/libdb_cxx-4.3.so #10 0x00002ae0ba5056c9 in __db_put_pp () from /usr/lib64/libdb_cxx-4.3.so #11 0x00002ae0ba497350 in Db::put () from /usr/lib64/libdb_cxx-4.3.so #12 0x00002ae0ba1beeed in rhm::bdbstore::BdbMessageStore::create () #13 0x00002ae0ba1ce39d in rhm::bdbstore::BdbMessageStore::create () #14 0x00002ae0b98c2c58 in qpid::broker::MessageStoreModule::create () #15 0x00002ae0b9884625 in qpid::broker::Queue::create () #16 0x00002ae0b98e29ea in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /root/svn/qpid/cpp/src/.libs/libqpidbroker.so.0 #17 0x00002ae0b9c74df6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /root/svn/qpid/cpp/src/.libs/libqpidcommon.so.0 #18 0x00002ae0b9c7b497 in qpid::framing::AMQP_ServerOperations::Invoker::visit #19 0x00002ae0b98ec37e in qpid::framing::invoke<qpid::broker::SessionAdapter> #20 0x00002ae0b98e9f5c in qpid::broker::SessionState::handleCommand () #21 0x00002ae0b98ead9c in qpid::broker::SessionState::handle () #22 0x00002ae0b9cb6436 in qpid::amqp_0_10::SessionHandler::handleIn () #23 0x00002ae0b9895681 in qpid::broker::ConnectionHandler::handle () #24 0x00002ae0b988fba0 in qpid::broker::Connection::received () #25 0x00002ae0b986b4a0 in qpid::amqp_0_10::Connection::decode () #26 0x00002ae0b9ce237f in qpid::sys::AsynchIOHandler::readbuff () #27 0x00002ae0b9cb23ea in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIO::BufferBase*, std::allocator<boost::function_base> >::operator() () #28 0x00002ae0b9cacbc9 in qpid::sys::AsynchIO::readable () #29 0x00002ae0b9ce505f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #30 0x00002ae0b9ce4be9 in qpid::sys::DispatchHandle::dispatchCallbacks () #31 0x00002ae0b9ce4ce5 in qpid::sys::Dispatcher::run () #32 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #33 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #34 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 1152661824 (LWP 534)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0ba4a0626 in __db_pthread_mutex_lock () #2 0x00002ae0ba528397 in __lock_get_internal () #3 0x00002ae0ba5284a2 in __lock_get () from /usr/lib64/libdb_cxx-4.3.so #4 0x00002ae0ba508c54 in __db_lget () from /usr/lib64/libdb_cxx-4.3.so #5 0x00002ae0ba4b042e in __bam_search () from /usr/lib64/libdb_cxx-4.3.so #6 0x00002ae0ba4a3987 in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #7 0x00002ae0ba4a4acd in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #8 0x00002ae0ba4fe669 in __db_c_get () from /usr/lib64/libdb_cxx-4.3.so #9 0x00002ae0ba4f9695 in __db_put () from /usr/lib64/libdb_cxx-4.3.so #10 0x00002ae0ba5056c9 in __db_put_pp () from /usr/lib64/libdb_cxx-4.3.so #11 0x00002ae0ba497350 in Db::put () from /usr/lib64/libdb_cxx-4.3.so #12 0x00002ae0ba1beeed in rhm::bdbstore::BdbMessageStore::create () #13 0x00002ae0ba1ce39d in rhm::bdbstore::BdbMessageStore::create () #14 0x00002ae0b98c2c58 in qpid::broker::MessageStoreModule::create () #15 0x00002ae0b9884625 in qpid::broker::Queue::create () #16 0x00002ae0b98e29ea in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /root/svn/qpid/cpp/src/.libs/libqpidbroker.so.0 #17 0x00002ae0b9c74df6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /root/svn/qpid/cpp/src/.libs/libqpidcommon.so.0 #18 0x00002ae0b9c7b497 in qpid::framing::AMQP_ServerOperations::Invoker::visit #19 0x00002ae0b98ec37e in qpid::framing::invoke<qpid::broker::SessionAdapter> #20 0x00002ae0b98e9f5c in qpid::broker::SessionState::handleCommand () #21 0x00002ae0b98ead9c in qpid::broker::SessionState::handle () #22 0x00002ae0b9cb6436 in qpid::amqp_0_10::SessionHandler::handleIn () #23 0x00002ae0b9895681 in qpid::broker::ConnectionHandler::handle () #24 0x00002ae0b988fba0 in qpid::broker::Connection::received () #25 0x00002ae0b986b4a0 in qpid::amqp_0_10::Connection::decode () #26 0x00002ae0b9ce237f in qpid::sys::AsynchIOHandler::readbuff () #27 0x00002ae0b9cb23ea in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIO::BufferBase*, std::allocator<boost::function_base> >::operator() () #28 0x00002ae0b9cacbc9 in qpid::sys::AsynchIO::readable () #29 0x00002ae0b9ce505f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #30 0x00002ae0b9ce4be9 in qpid::sys::DispatchHandle::dispatchCallbacks () #31 0x00002ae0b9ce4ce5 in qpid::sys::Dispatcher::run () #32 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #33 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #34 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 1163151680 (LWP 535)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0ba4a0626 in __db_pthread_mutex_lock () #2 0x00002ae0ba528397 in __lock_get_internal () #3 0x00002ae0ba5284a2 in __lock_get () from /usr/lib64/libdb_cxx-4.3.so #4 0x00002ae0ba508c54 in __db_lget () from /usr/lib64/libdb_cxx-4.3.so #5 0x00002ae0ba4b0311 in __bam_search () from /usr/lib64/libdb_cxx-4.3.so #6 0x00002ae0ba4b27a1 in __bam_split () from /usr/lib64/libdb_cxx-4.3.so #7 0x00002ae0ba4a4101 in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #8 0x00002ae0ba4ffcb6 in __db_c_put () from /usr/lib64/libdb_cxx-4.3.so #9 0x00002ae0ba4f95c7 in __db_put () from /usr/lib64/libdb_cxx-4.3.so #10 0x00002ae0ba5056c9 in __db_put_pp () from /usr/lib64/libdb_cxx-4.3.so #11 0x00002ae0ba497350 in Db::put () from /usr/lib64/libdb_cxx-4.3.so #12 0x00002ae0ba1beeed in rhm::bdbstore::BdbMessageStore::create () #13 0x00002ae0ba1ce39d in rhm::bdbstore::BdbMessageStore::create () #14 0x00002ae0b98c2c58 in qpid::broker::MessageStoreModule::create () #15 0x00002ae0b9884625 in qpid::broker::Queue::create () #16 0x00002ae0b98e29ea in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /root/svn/qpid/cpp/src/.libs/libqpidbroker.so.0 #17 0x00002ae0b9c74df6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /root/svn/qpid/cpp/src/.libs/libqpidcommon.so.0 #18 0x00002ae0b9c7b497 in qpid::framing::AMQP_ServerOperations::Invoker::visit #19 0x00002ae0b98ec37e in qpid::framing::invoke<qpid::broker::SessionAdapter> #20 0x00002ae0b98e9f5c in qpid::broker::SessionState::handleCommand () #21 0x00002ae0b98ead9c in qpid::broker::SessionState::handle () #22 0x00002ae0b9cb6436 in qpid::amqp_0_10::SessionHandler::handleIn () #23 0x00002ae0b9895681 in qpid::broker::ConnectionHandler::handle () #24 0x00002ae0b988fba0 in qpid::broker::Connection::received () #25 0x00002ae0b986b4a0 in qpid::amqp_0_10::Connection::decode () #26 0x00002ae0b9ce237f in qpid::sys::AsynchIOHandler::readbuff () #27 0x00002ae0b9cb23ea in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIO::BufferBase*, std::allocator<boost::function_base> >::operator() () #28 0x00002ae0b9cacbc9 in qpid::sys::AsynchIO::readable () #29 0x00002ae0b9ce505f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #30 0x00002ae0b9ce4be9 in qpid::sys::DispatchHandle::dispatchCallbacks () #31 0x00002ae0b9ce4ce5 in qpid::sys::Dispatcher::run () #32 0x00002ae0b9cb293a in qpid::sys::Thread::runRunnable () #33 0x0000003ca98062f7 in start_thread () from /lib64/libpthread.so.0 #34 0x0000003ca90d1b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 47144683237824 (LWP 468)): #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002ae0ba4a0626 in __db_pthread_mutex_lock () #2 0x00002ae0ba528397 in __lock_get_internal () #3 0x00002ae0ba5284a2 in __lock_get () from /usr/lib64/libdb_cxx-4.3.so #4 0x00002ae0ba508c54 in __db_lget () from /usr/lib64/libdb_cxx-4.3.so #5 0x00002ae0ba4b042e in __bam_search () from /usr/lib64/libdb_cxx-4.3.so #6 0x00002ae0ba4a3987 in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #7 0x00002ae0ba4a4acd in __bam_c_init () from /usr/lib64/libdb_cxx-4.3.so #8 0x00002ae0ba4fe669 in __db_c_get () from /usr/lib64/libdb_cxx-4.3.so #9 0x00002ae0ba4f9695 in __db_put () from /usr/lib64/libdb_cxx-4.3.so #10 0x00002ae0ba5056c9 in __db_put_pp () from /usr/lib64/libdb_cxx-4.3.so #11 0x00002ae0ba497350 in Db::put () from /usr/lib64/libdb_cxx-4.3.so #12 0x00002ae0ba1beeed in rhm::bdbstore::BdbMessageStore::create () #13 0x00002ae0ba1ce39d in rhm::bdbstore::BdbMessageStore::create () #14 0x00002ae0b98c2c58 in qpid::broker::MessageStoreModule::create () #15 0x00002ae0b9884625 in qpid::broker::Queue::create () #16 0x00002ae0b98e29ea in qpid::broker::SessionAdapter::QueueHandlerImpl::declare () from /root/svn/qpid/cpp/src/.libs/libqpidbroker.so.0 #17 0x00002ae0b9c74df6 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit () from /root/svn/qpid/cpp/src/.libs/libqpidcommon.so.0 #18 0x00002ae0b9c7b497 in qpid::framing::AMQP_ServerOperations::Invoker::visit #19 0x00002ae0b98ec37e in qpid::framing::invoke<qpid::broker::SessionAdapter> #20 0x00002ae0b98e9f5c in qpid::broker::SessionState::handleCommand () #21 0x00002ae0b98ead9c in qpid::broker::SessionState::handle () #22 0x00002ae0b9cb6436 in qpid::amqp_0_10::SessionHandler::handleIn () #23 0x00002ae0b9895681 in qpid::broker::ConnectionHandler::handle () #24 0x00002ae0b988fba0 in qpid::broker::Connection::received () #25 0x00002ae0b986b4a0 in qpid::amqp_0_10::Connection::decode () #26 0x00002ae0b9ce237f in qpid::sys::AsynchIOHandler::readbuff () #27 0x00002ae0b9cb23ea in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIO::BufferBase*, std::allocator<boost::function_base> >::operator() () #28 0x00002ae0b9cacbc9 in qpid::sys::AsynchIO::readable () #29 0x00002ae0b9ce505f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () #30 0x00002ae0b9ce4be9 in qpid::sys::DispatchHandle::dispatchCallbacks () #31 0x00002ae0b9ce4ce5 in qpid::sys::Dispatcher::run () #32 0x00002ae0b986dfd6 in qpid::broker::Broker::run () #33 0x0000000000407a0b in main () #0 0x0000003ca980a496 in pthread_cond_wait@@GLIBC_2.3.2 () --- Additional comment from davids on 2008-07-30 09:30:46 EDT --- btw ... this deadlock do also not respond to SIGINT --- Additional comment from kim.vdriet on 2008-07-30 12:23:22 EDT --- This looks like a race condition internal to BDB triggered by multiple threads calling Db::put() at the same time on the same database - in this case the BdbMessageStore::queueDb. --- Additional comment from kim.vdriet on 2008-07-31 15:09:48 EDT --- This bug has been difficult to reproduce. Each time it has been reproduced the stack trace confirms the BDB race condition through create(), so a fix has been checked in which places each queue create under a local transaction which will serialize all such requests. This should fix the problem, but only time and repeated testing will confirm that it has really gone. I am marking this as MODIFIED for the time being. This can be reopened if the problem shows up again. --- Additional comment from davids on 2008-08-07 02:52:40 EDT --- I've been trying to reproduce this on the latest mrg-1.0/qpid-0.10 branches for quite some hours by now. This bug seems to be solved to me know. The reproducer simply do not work as expected - it do not reproduce this bug anymore, but behaves nicely without any failures. Neither is any other bugs exposed as well. I've tested it close to 150 runs without a single failure. These runs are done on 2 different computers with 4 cores and 4GB RAM, one computer with AMD CPU and one with Intel CPU.
RHTS test MRG/qpid_broker_jfail_bz456272 was updated to handle possible hang during python code is interpreted (local watchdog implemented in test shell unit). Current MRG/qpid_broker_jfail_bz456272 runs show no more problem like this. Bug going to VERIFIED. (Any single failure in future will of course move it to FAIL_QA) MRG/qpid_broker_jfail_bz456272 test results are unstable, sometimes fails on disconnecting queues from broker. This behavior is under investigation on QA and might be reported to DEV as new bug.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2008-0867.html