Bug 1103094
| Summary: | Memory leak and deadlock together when using qpid.auto_delete_timeout queues | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Pavel Moravec <pmoravec> | |
| Component: | qpid-cpp | Assignee: | Gordon Sim <gsim> | |
| Status: | CLOSED ERRATA | QA Contact: | Leonid Zhaldybin <lzhaldyb> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | high | |||
| Version: | 2.5 | CC: | esammons, gsim, jross, lzhaldyb, mcressma, pmoravec, rrajasek, sauchter | |
| Target Milestone: | 2.5.1 | Keywords: | TestCaseProvided | |
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | qpid-cpp-0.18-25 | Doc Type: | Bug Fix | |
| Doc Text: |
It was discovered that a concurrent bind and delete on a queue could leave a binding in place, even though the queue was no longer accessible. Additionally, the queue lock was held when cancelling the auto-delete timer task. These factors caused memory usage to climb, and holding the lock while cancelling a timer could also result in a deadlock. Threads are now coordinated to ensure a concurrent bind and delete removes all bindings when a queue reaches the deleted state, and the timer task is cancelled outside the lock scope. This ensures no deadlocks or memory leaks when attempting to re-bind a queue that is being auto-deleted by a timer.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1105221 (view as bug list) | Environment: | ||
| Last Closed: | 2014-06-30 10:26:26 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1105221 | |||
| Attachments: | ||||
Btw. this _is_ also a deadlock. When having even number of threads, one can end up with totally unresponsible broker (that consumes a lot of memory, in addition). Hence severity urgent. Valgrind output (one has to increase qpid.auto_delete_timeout, in my case to 4s): ==16625== 5,481 (952 direct, 4,529 indirect) bytes in 1 blocks are definitely lost in loss record 183 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1E3571D: qpid::broker::QueueRegistry::declare(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, boost::shared_ptr<qpid::broker::Exchange>, qpid::framing::FieldTable const&, bool, std::string, std::string) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D87E16: qpid::broker::Broker::createQueue(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, std::string const&, qpid::framing::FieldTable const&, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5F282: qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B2C2E: qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B4830: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E69348: qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17F4590: qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1DACC92: qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1DA5697: qpid::broker::Connection::received(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D7010C: qpid::amqp_0_10::Connection::decode(char const*, unsigned long) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== ==16625== 5,481 (24 direct, 5,457 indirect) bytes in 1 blocks are definitely lost in loss record 184 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1E35774: qpid::broker::QueueRegistry::declare(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, boost::shared_ptr<qpid::broker::Exchange>, qpid::framing::FieldTable const&, bool, std::string, std::string) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D87E16: qpid::broker::Broker::createQueue(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, std::string const&, qpid::framing::FieldTable const&, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5F282: qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B2C2E: qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B4830: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E69348: qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17F4590: qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1DACC92: qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1DA5697: qpid::broker::Connection::received(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D7010C: qpid::amqp_0_10::Connection::decode(char const*, unsigned long) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== ==16625== 5,481 (48 direct, 5,433 indirect) bytes in 1 blocks are definitely lost in loss record 185 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1E079A0: qpid::broker::Queue::addObserver(boost::shared_ptr<qpid::broker::QueueObserver>) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E6C818: qpid::broker::ThresholdAlerts::observe(qpid::broker::Queue&, qpid::management::ManagementAgent&, unsigned long, unsigned long, long) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E6DEC0: qpid::broker::ThresholdAlerts::observe(qpid::broker::Queue&, qpid::management::ManagementAgent&, std::map<std::string, qpid::types::Variant, std::less<std::string>, std::allocator<std::pair<std::string const, qpid::types::Variant> > > const&, unsigned short) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E6E7D4: qpid::broker::ThresholdAlerts::observe(qpid::broker::Queue&, qpid::management::ManagementAgent&, qpid::framing::FieldTable const&, unsigned short) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E0C4B7: qpid::broker::Queue::configureImpl(qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E3588E: qpid::broker::QueueRegistry::declare(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, boost::shared_ptr<qpid::broker::Exchange>, qpid::framing::FieldTable const&, bool, std::string, std::string) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D87E16: qpid::broker::Broker::createQueue(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, std::string const&, qpid::framing::FieldTable const&, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5F282: qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B2C2E: qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B4830: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== ==16625== 5,481 (48 direct, 5,433 indirect) bytes in 1 blocks are definitely lost in loss record 186 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1E079A0: qpid::broker::Queue::addObserver(boost::shared_ptr<qpid::broker::QueueObserver>) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E3D587: qpid::broker::QueueFlowLimit::observe(qpid::broker::Queue&, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E0CDB6: qpid::broker::Queue::configureImpl(qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E3588E: qpid::broker::QueueRegistry::declare(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, boost::shared_ptr<qpid::broker::Exchange>, qpid::framing::FieldTable const&, bool, std::string, std::string) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D87E16: qpid::broker::Broker::createQueue(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, std::string const&, qpid::framing::FieldTable const&, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5F282: qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B2C2E: qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B4830: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E69348: qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17F4590: qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== ==16625== 5,516 (48 direct, 5,468 indirect) bytes in 1 blocks are definitely lost in loss record 187 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1D04B10: std::_Rb_tree<std::string, std::pair<std::string const, qpid::types::Variant>, std::_Select1st<std::pair<std::string const, qpid::types::Variant> >, std::less<std::string>, std::allocator<std::pair<std::string const, qpid::types::Variant> > >::_M_copy(std::_Rb_tree_node<std::pair<std::string const, qpid::types::Variant> > const*, std::_Rb_tree_node<std::pair<std::string const, qpid::types::Variant> >*) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E0CD44: qpid::broker::Queue::configureImpl(qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E3588E: qpid::broker::QueueRegistry::declare(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, boost::shared_ptr<qpid::broker::Exchange>, qpid::framing::FieldTable const&, bool, std::string, std::string) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1D87E16: qpid::broker::Broker::createQueue(std::string const&, bool, bool, qpid::broker::OwnershipToken const*, std::string const&, qpid::framing::FieldTable const&, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5F282: qpid::broker::SessionAdapter::QueueHandlerImpl::declare(std::string const&, std::string const&, bool, bool, bool, bool, qpid::framing::FieldTable const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B2C2E: qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B4830: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::QueueDeclareBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E69348: qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17F4590: qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1DACC92: qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== ==16625== 18,114 (552 direct, 17,562 indirect) bytes in 3 blocks are definitely lost in loss record 193 of 193 ==16625== at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298) ==16625== by 0x3EE1E19202: qpid::broker::Queue::tryAutoDelete(qpid::broker::Broker&, boost::shared_ptr<qpid::broker::Queue>, std::string const&, std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E48453: qpid::broker::SemanticState::cancel(boost::shared_ptr<qpid::broker::SemanticState::ConsumerImpl>) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E4D380: qpid::broker::SemanticState::cancel(std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E5AF6D: qpid::broker::SessionAdapter::MessageHandlerImpl::cancel(std::string const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17B3272: qpid::framing::AMQP_ServerOperations::MessageHandler::Invoker::visit(qpid::framing::MessageCancelBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE17B64B0: qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::MessageCancelBody const&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1E68E67: qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*, qpid::framing::SequenceNumber const&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1E69348: qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE17F4590: qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidcommon.so.8.0.0) ==16625== by 0x3EE1DACC92: qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) ==16625== by 0x3EE1DA5697: qpid::broker::Connection::received(qpid::framing::AMQFrame&) (in /usr/lib64/libqpidbroker.so.8.0.0) The deadlock is addressed upstream by: https://svn.apache.org/r1599765 The memory growth is addressed upstream by: https://svn.apache.org/r1599766 Though in running the reproducer I do seem some initial memory growth even with this fix applied, the growth seems to level out rather than continuing to grow indefinitely. Created attachment 909607 [details]
qpidd log file from the startup till the point a client hangs
Created attachment 909608 [details]
the output of 'strace -p $(pidof qpidd)
Created attachment 909647 [details]
pstack trace of qpidd
Created attachment 909648 [details]
pstack trace of the client
Created attachment 909649 [details]
example of qpidd memory consumption growth over time
Tested on the latest available packages on RHEL5 and RHEL6 (both i386 and x86_64). The reproducer script was running for a few days without triggering a deadlock. The broker does not appear to be leaking memory. This issue has been fixed. Packages used for testing: python-qpid-0.18-12 python-qpid-qmf-0.18-24 qpid-cpp-0.18-25 qpid-qmf-0.18-24 qpid-tools-0.18-10 -> VERIFIED Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0804.html |
Description of problem: There is a memory leak in the broker when a client declares auto-delete queue with qpid.auto_delete_timeout and re-connects just at the time the broker is about to delete the queue. Version-Release number of selected component (if applicable): 0.18-11, 0.18-20, 0.22-41 How reproducible: 100% within some time Steps to Reproduce: 1) Run script: queues=100 sleep=0 while true; do for i in $(seq 1 $queues); do qpid-receive -a "autoDelQueue_${i}; {create:always, node:{ x-declare:{auto-delete:True, arguments:{'qpid.auto_delete_timeout':1}}, x-bindings:[{exchange:'amq.fanout', queue:autoDelQueue_${i}}] }}" -b broker --print-content=no -m 10 & done wait sleep $sleep date done Modify parameters (esp. sleep) such that the loop is executed once every second. 2) Monitor qpid broker memory utilization over time. Actual results: Both VSZ and RSS of qpidd process grows. Expected results: No memory grow. Additional info: Optionally, one can generate some traffic by running: qpid-send -a amq.fanout -m 100000 -b broker --send-rate=50 in parallel. If so, the broker will end up with coredump (manually generated) backtrace like: Thread 4 (Thread 0x7f373424a7a0 (LWP 10281)): #0 0x00007f3731fa23dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f373378c09b in wait (this=0x8a1890) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Condition.h:63 #2 wait (this=0x8a1890) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Monitor.h:41 #3 qpid::sys::TimerTask::cancel (this=0x8a1890) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Timer.cpp:94 #4 0x00007f3733c71345 in qpid::broker::Queue::consume (this=0xf05570, c=..., requestExclusive=false, arguments=..., connectionId= "qpid.10.34.1.121:5672-10.34.1.149:54185", userId="anonymous@QPID") at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:571 #5 0x00007f3733d0ec29 in qpid::broker::SemanticState::consume (this=0x7f37207ac168, tag="autoDelQueue_73", queue=<value optimized out>, ackRequired=true, acquire=true, exclusive=false, resumeId="", resumeTtl=0, arguments=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/SemanticState.cpp:144 #6 0x00007f3733d21f8d in qpid::broker::SessionAdapter::MessageHandlerImpl::subscribe (this=0x7f37207ac358, queueName= "autoDelQueue_73", destination="autoDelQueue_73", acceptMode=0 '\000', acquireMode=<value optimized out>, exclusive=false, resumeId="", resumeTtl=0, arguments=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/SessionAdapter.cpp:415 #7 0x00007f37336e5668 in invoke<qpid::framing::AMQP_ServerOperations::MessageHandler> (this=<value optimized out>, body=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/framing/MessageSubscribeBody.h:120 #8 qpid::framing::AMQP_ServerOperations::MessageHandler::Invoker::visit (this=<value optimized out>, body=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/framing/ServerInvoker.cpp:554 #9 0x00007f37336e93e1 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7fffeadd9140, body=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/framing/ServerInvoker.cpp:243 #10 0x00007f3733d296dc in invoke<qpid::broker::SessionAdapter> (this=<value optimized out>, method=0xec3e60) at /usr/src/debug/qpid-0.22/cpp/src/qpid/framing/Invoker.h:67 #11 qpid::broker::SessionState::handleCommand (this=<value optimized out>, method=0xec3e60) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/SessionState.cpp:175 #12 0x00007f3733d2a6c5 in qpid::broker::SessionState::handleIn (this=0x7f37207abfa0, frame=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/SessionState.cpp:267 #13 0x00007f373375a8d1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f3720200a20, f=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/amqp_0_10/SessionHandler.cpp:93 #14 0x00007f3733ca002b in operator() (this=0x7f37200d3608, frame=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/framing/Handler.h:42 #15 qpid::broker::ConnectionHandler::handle (this=0x7f37200d3608, frame=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/ConnectionHandler.cpp:94 #16 0x00007f3733c98ae8 in qpid::broker::amqp_0_10::Connection::received (this=0x7f37200d3420, frame=...) ---Type <return> to continue, or q <return> to quit--- at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/amqp_0_10/Connection.cpp:198 #17 0x00007f3733c2d70d in qpid::amqp_0_10::Connection::decode (this=0x7f3720752730, buffer=<value optimized out>, size=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/amqp_0_10/Connection.cpp:58 #18 0x00007f3733783980 in qpid::sys::AsynchIOHandler::readbuff (this=0x7f3728acdfd0, buff=0x7f3728a3e280) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/AsynchIOHandler.cpp:135 #19 0x00007f373370e524 in operator() (this=0x7f372873e390, h=...) at /usr/include/boost/function/function_template.hpp:1013 #20 qpid::sys::posix::AsynchIO::readable (this=0x7f372873e390, h=...) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/AsynchIO.cpp:453 #21 0x00007f3733788033 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #22 0x00007f3733787181 in qpid::sys::DispatchHandle::processEvent (this=0x7f372873e398, type=qpid::sys::Poller::READABLE) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/DispatchHandle.cpp:280 #23 0x00007f3733731302 in process (this=0x70fa80) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Poller.h:131 #24 qpid::sys::Poller::run (this=0x70fa80) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/epoll/EpollPoller.cpp:522 #25 0x00007f3733c489a2 in qpid::broker::Broker::run (this=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Broker.cpp:450 #26 0x000000000040a9c4 in qpid::broker::QpiddDaemon::child (this=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/posix/QpiddBroker.cpp:149 #27 0x00007f3733bebcc3 in qpid::broker::Daemon::fork (this=0x7fffeaddb260) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Daemon.cpp:91 #28 0x000000000040756a in qpid::broker::QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/posix/QpiddBroker.cpp:193 #29 0x000000000040cba4 in qpid::broker::run_broker (argc=6, argv=0x7fffeaddb898, hidden=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpidd.cpp:108 #30 0x00007f3731c14cdd in __libc_start_main () from /lib64/libc.so.6 #31 0x0000000000406bf9 in _start () Thread 1 (Thread 0x7f3731281700 (LWP 10282)): #0 0x00007f3731fa4ff4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f3731fa0343 in _L_lock_892 () from /lib64/libpthread.so.0 #2 0x00007f3731fa0227 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f3733c0228c in lock (this=<value optimized out>, l=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Mutex.h:116 #4 qpid::sys::ScopedLock<qpid::sys::Mutex>::ScopedLock (this=<value optimized out>, l=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Mutex.h:33 #5 0x00007f3733c7a0e6 in qpid::broker::Queue::remove (this=0xf05570, maxCount=0, p=..., f=..., type=qpid::broker::REPLICATOR, triggerAutoDelete=false, maxTests=0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:746 #6 0x00007f3733c7acf2 in qpid::broker::Queue::destroyed (this=0xf05570) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:1151 #7 0x00007f3733c7b28b in qpid::broker::Queue::tryAutoDelete (this=0xf05570) at /usr/src/debug/qpid-0.22/cpp/src/qpid/broker/Queue.cpp:1319 #8 0x00007f373378b960 in qpid::sys::Timer::fire (this=<value optimized out>, t=...) ---Type <return> to continue, or q <return> to quit--- at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Timer.cpp:222 #9 0x00007f373378d05f in qpid::sys::Timer::run (this=0x729df0) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/Timer.cpp:156 #10 0x00007f373372661a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at /usr/src/debug/qpid-0.22/cpp/src/qpid/sys/posix/Thread.cpp:35 #11 0x00007f3731f9e7f1 in start_thread () from /lib64/libpthread.so.0 #12 0x00007f3731cdb70d in clone () from /lib64/libc.so.6 (btw isn't that a deadlock?) (btw the referenced queue above did not exist at the time core was generated - at least per qpid-stat -q output)