Bug 590636
| Summary: | qpidd broker crash in qpid::broker::PersistableMessage::~PersistableMessage -> atomic_exchange_and_add() call path | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Frantisek Reznicek <freznice> |
| Component: | qpid-cpp | Assignee: | Gordon Sim <gsim> |
| Status: | CLOSED DUPLICATE | QA Contact: | MRG Quality Engineering <mrgqe-bugs> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | Development | CC: | esammons, gsim |
| Target Milestone: | 1.3 | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2010-05-14 16:20:24 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I believe this is a dup of 590624; its the same codepath, but as got a few frames further before failing. There is in my view a strong chance that both these cases are due to the heap corruption addressed in the resolution of 587505. If the DataTokenImpls are trampled on then the above traces are what you would expect. *** This bug has been marked as a duplicate of bug 590624 *** |
Description of problem: There was observed broker crash during broker perftest performance stress in boost's atomic_exchange_and_add() call initialized by ~PersistableMessage(): Thread 1 (Thread 30166): #0 0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50 #1 weak_release (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:157 #2 boost::detail::weak_count::~weak_count (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/shared_count.hpp:262 #3 0x00002b1cac50661f in ~list (this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at /usr/include/boost/weak_ptr.hpp:27 #4 qpid::broker::PersistableMessage::~PersistableMessage ( this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at qpid/broker/PersistableMessage.cpp:34 #5 0x00002b1cac4fbaa6 in qpid::broker::Message::~Message ( this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at qpid/broker/Message.cpp:59 #6 0x00002b1cade2c63c in ~intrusive_ptr (this=0x2aaaac032840, __in_chrg=<value optimized out>) at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42 This issue was observed on RHEL 5.5 x86_64 OS/arch. Version-Release number of selected component (if applicable): qpid-cpp-*0.7.935473 How reproducible: very hard Steps to Reproduce: 1. run qpid_test_qpidd-perftest_performance and wait for crash launch broker loop the perftest client to sweep the parameters keep running until broker crashes Actual results: qpidd broker crashes. Expected results: qpidd broker should not crash. Additional info: [root@mrg-qe-02 qpid_test_qpidd-perftest_performance_bck]# cat dump_core.30166 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done. done. [New Thread 30180] [New Thread 30179] [New Thread 30178] [New Thread 30177] [New Thread 30176] [New Thread 30175] [New Thread 30174] [New Thread 30173] [New Thread 30172] [New Thread 30171] Reading symbols from /usr/lib64/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidbroker.so.2 Reading symbols from /usr/lib64/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidcommon.so.2 Reading symbols from /usr/lib64/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_program_options.so.1.33.1.debug...done. done. Loaded symbols for /usr/lib64/libboost_program_options.so.2 Reading symbols from /usr/lib64/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_filesystem.so.1.33.1.debug...done. done. Loaded symbols for /usr/lib64/libboost_filesystem.so.2 Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/acl.so Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/xml.so Reading symbols from /usr/lib64/libxerces-c.so.28...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxerces-c.so.28 Reading symbols from /usr/lib64/libxqilla.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxqilla.so.3 Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/cluster.so Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/openais/libcpg.so.2 Reading symbols from /usr/lib64/libcman.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcman.so.2 Reading symbols from /usr/lib64/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/libqpidclient.so.2 Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/client/sslconnector.so Reading symbols from /usr/lib64/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib64/libsslcommon.so.2 Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnss3.so Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl3.so Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnspr4.so Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnssutil3.so Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libplc4.so Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libplds4.so Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/msgstore.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so Reading symbols from /usr/lib64/libdb_cxx-4.3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libdb_cxx-4.3.so Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libaio.so.1 Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/ssl.so Reading symbols from /usr/lib64/qpid/daemon/watchdog.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/watchdog.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/watchdog.so Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done. done. Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'. Program terminated with signal 11, Segmentation fault. #0 0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50 50 ); (gdb) rax 0xffffffff 4294967295 rbx 0x2aaab0080a10 46912586123792 rcx 0x2aaab0000038 46912585596984 rdx 0x2aaab0091970 46912586193264 rsi 0x2aaab008fa60 46912586185312 rdi 0x5858585858585858 6365935209750747224 rbp 0x5858585858585858 0x5858585858585858 rsp 0x7fff8596d048 0x7fff8596d048 r8 0x2aaab0003cb8 46912585612472 r9 0x2aaab00591c0 46912585961920 r10 0x5511bb0 89201584 r11 0x36a1c08740 234641983296 r12 0x2aaab0059040 46912585961536 r13 0x2aaab0058fb0 46912585961392 r14 0x2aaab0058fb0 46912585961392 r15 0x2aaab0059158 46912585961816 rip 0x2b1cac50800d 0x2b1cac50800d <boost::detail::weak_count::~weak_count()+13> eflags 0x10202 [ IF RF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 st0 0 (raw 0x00000000000000000000) st1 0 (raw 0x00000000000000000000) st2 0 (raw 0x00000000000000000000) st3 0 (raw 0x00000000000000000000) st4 0 (raw 0x00000000000000000000) st5 0 (raw 0x00000000000000000000) st6 0 (raw 0x00000000000000000000) st7 0 (raw 0x00000000000000000000) fctrl 0x0 0 fstat 0x0 0 ftag 0x0 0 fiseg 0x0 0 fioff 0x0 0 foseg 0x0 0 fooff 0x0 0 fop 0x0 0 (gdb) Using memory regions provided by the target. There are no memory regions defined. (gdb) From To Syms Read Shared Object Library 0x00002b1cac4291a0 0x00002b1cac5ad0b8 Yes (*) /usr/lib64/libqpidbroker.so.2 0x0000003c5630af10 0x0000003c5640bc38 Yes (*) /usr/lib64/libqpidcommon.so.2 0x0000003674010aa0 0x000000367402dae8 Yes (*) /usr/lib64/libboost_program_options.so.2 0x0000003674404810 0x000000367440cff8 Yes (*) /usr/lib64/libboost_filesystem.so.2 0x00000036b5e01500 0x00000036b5e02918 Yes (*) /lib64/libuuid.so.1 0x00000036a1800e10 0x00000036a1801a08 Yes (*) /lib64/libdl.so.2 0x00000036a2402220 0x00000036a2405cc8 Yes (*) /lib64/librt.so.1 0x00000036b66046e0 0x00000036b6613be8 Yes (*) /usr/lib64/libsasl2.so.2 0x00000036b3a4f430 0x00000036b3ac3058 Yes (*) /usr/lib64/libstdc++.so.6 0x00000036a1403e60 0x00000036a1443e38 Yes (*) /lib64/libm.so.6 0x00000036af201e50 0x00000036af20b018 Yes (*) /lib64/libgcc_s.so.1 0x00000036a101d780 0x00000036a1109ff8 Yes (*) /lib64/libc.so.6 0x00000036a0c00a70 0x00000036a0c1671e Yes (*) /lib64/ld-linux-x86-64.so.2 0x00000036a1c051f0 0x00000036a1c10258 Yes (*) /lib64/libpthread.so.0 0x00000036a54032a0 0x00000036a540e2d8 Yes (*) /lib64/libresolv.so.2 0x00000036b1e009f0 0x00000036b1e06918 Yes (*) /lib64/libcrypt.so.1 0x00002b1cac834bf0 0x00002b1cac853f98 Yes (*) /usr/lib64/qpid/daemon/acl.so 0x00002b1caca667f0 0x00002b1caca70988 Yes (*) /usr/lib64/qpid/daemon/xml.so 0x0000003673773070 0x00000036738f4758 Yes (*) /usr/lib64/libxerces-c.so.28 0x00002b1cacdf9090 0x00002b1cacf84b28 Yes (*) /usr/lib64/libxqilla.so.3 0x00002b1cad2fdda0 0x00002b1cad369528 Yes (*) /usr/lib64/qpid/daemon/cluster.so 0x00002b1cad5a73d0 0x00002b1cad5a9338 Yes (*) /usr/lib64/openais/libcpg.so.2 0x00002b1cad7ab110 0x00002b1cad7adb78 Yes (*) /usr/lib64/libcman.so.2 0x0000003c5685f120 0x0000003c56922078 Yes (*) /usr/lib64/libqpidclient.so.2 0x00002b1cad9b8880 0x00002b1cad9c5248 Yes (*) /usr/lib64/qpid/client/sslconnector.so 0x00002b1cadbd6350 0x00002b1cadbed058 Yes (*) /usr/lib64/libsslcommon.so.2 0x00000030b8c183b0 0x00000030b8cf6f08 Yes (*) /usr/lib64/libnss3.so 0x00000030b98085e0 0x00000030b982b638 Yes (*) /usr/lib64/libssl3.so 0x00000030b800cf30 0x00000030b802b738 Yes (*) /usr/lib64/libnspr4.so 0x00000030b9008340 0x00000030b9012c38 Yes (*) /usr/lib64/libnssutil3.so 0x00000030b8401370 0x00000030b8402978 Yes (*) /usr/lib64/libplc4.so 0x00000030b8800e10 0x00000030b8801c08 Yes (*) /usr/lib64/libplds4.so 0x00000036a2001fd0 0x00000036a200cac8 Yes (*) /usr/lib64/libz.so.1 0x00002b1cade2ad10 0x00002b1cadeba5d8 Yes (*) /usr/lib64/qpid/daemon/msgstore.so 0x00002b1cae1245d0 0x00002b1cae1dd288 Yes (*) /usr/lib64/libdb_cxx-4.3.so 0x00002b1cae408510 0x00002b1cae4086d1 Yes (*) /usr/lib64/libaio.so.1 0x00002b1cae612570 0x00002b1cae61a698 Yes (*) /usr/lib64/qpid/daemon/ssl.so 0x00002b1cae8245e0 0x00002b1cae827c18 Yes (*) /usr/lib64/qpid/daemon/watchdog.so 0x00002b1caea2f620 0x00002b1caea33a68 Yes (*) /usr/lib64/qpid/daemon/replication_exchange.so 0x00002b1caec3cb70 0x00002b1caec42708 Yes (*) /usr/lib64/qpid/daemon/replicating_listener.so (*): Shared library is missing debugging information. (gdb) 11 Thread 30171 0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 10 Thread 30172 0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 9 Thread 30173 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 8 Thread 30174 0x00000036a10c6070 in __write_nocancel () from /lib64/libc.so.6 7 Thread 30175 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 6 Thread 30176 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 5 Thread 30177 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 4 Thread 30178 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 3 Thread 30179 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 2 Thread 30180 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 30166 0x00002b1cac50800d in atomic_exchange_and_add ( this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50 (gdb) Thread 11 (Thread 30171): #0 0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003c564074d8 in qpid::sys::Timer::run (this=0x529fcf0) at ../include/qpid/sys/posix/Condition.h:69 #2 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable ( p=0x529fd24) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 10 (Thread 30172): #0 0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003c564074d8 in qpid::sys::Timer::run (this=0x52a87f0) at ../include/qpid/sys/posix/Condition.h:69 #2 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable ( p=0x52a8824) at qpid/sys/posix/Thread.cpp:35 #3 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 9 (Thread 30173): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 8 (Thread 30174): #0 0x00000036a10c6070 in __write_nocancel () from /lib64/libc.so.6 #1 0x00000036a106a8d4 in __libc_message () from /lib64/libc.so.6 #2 0x00000036a107230f in _int_free () from /lib64/libc.so.6 #3 0x00000036a107276b in free () from /lib64/libc.so.6 #4 0x00002b1cade3f1d2 in deallocate (this=0x52a0710, queue=0x5376c10, txn= 0x4346be00, messageId=..., message=..., newId=176) at /usr/include/c++/4.1.2/ext/new_allocator.h:94 #5 _M_deallocate (this=0x52a0710, queue=0x5376c10, txn=0x4346be00, messageId=..., message=..., newId=176) at /usr/include/c++/4.1.2/bits/stl_vector.h:133 #6 ~_Vector_base (this=0x52a0710, queue=0x5376c10, txn=0x4346be00, messageId=..., message=..., newId=176) at /usr/include/c++/4.1.2/bits/stl_vector.h:119 #7 ~vector (this=0x52a0710, queue=0x5376c10, txn=0x4346be00, messageId=..., message=..., newId=176) at /usr/include/c++/4.1.2/bits/stl_vector.h:272 #8 mrg::msgstore::MessageStoreImpl::store (this=0x52a0710, queue=0x5376c10, txn=0x4346be00, messageId=..., message=..., newId=176) at MessageStoreImpl.cpp:1493 #9 0x00002b1cade4d01b in mrg::msgstore::MessageStoreImpl::enqueue ( this=0x52a0710, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434 #10 0x00002b1cac5035cb in qpid::broker::MessageStoreModule::enqueue ( this=<value optimized out>, ctxt=0x4346afd0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124 #11 0x00002b1cac51260c in qpid::broker::Queue::enqueue (this=0x5376c10, ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>) at qpid/broker/Queue.cpp:753 #12 0x00002b1cac513dc6 in qpid::broker::Queue::deliver (this=0x5376c10, msg=...) at qpid/broker/Queue.cpp:164 #13 0x00002b1cac4b67e3 in qpid::broker::DeliverableMessage::deliverTo ( this=0x4346d050, queue=...) at qpid/broker/DeliverableMessage.cpp:31 #14 0x00002b1cac4ce065 in qpid::broker::Exchange::doRoute (this=0x52a99b0, msg=..., b=...) at qpid/broker/Exchange.cpp:91 #15 0x00002b1cac56ea8c in qpid::broker::TopicExchange::route (this=0x52a99b0, msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321 #16 0x00002b1cac544bf3 in qpid::broker::SemanticState::route ( this=<value optimized out>, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461 #17 0x00002b1cac545b4d in qpid::broker::SemanticState::handle (this=0x5333a78, msg=...) at qpid/broker/SemanticState.cpp:415 #18 0x00002b1cac56a90e in qpid::broker::SessionState::handleContent ( this=0x53338a0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:249 #19 0x00002b1cac56aeb0 in qpid::broker::SessionState::handleIn ( this=0x53338a0, frame=...) at qpid/broker/SessionState.cpp:327 #20 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ( this=0x53336c0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #21 0x00002b1cac4a7ee2 in operator() (this=0x5332a30, frame=...) at qpid/framing/Handler.h:42 #22 qpid::broker::Connection::received (this=0x5332a30, frame=...) at qpid/broker/Connection.cpp:143 #23 0x00002b1cac487d84 in qpid::amqp_0_10::Connection::decode (this=0x5332500, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #24 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff ( this=0x52edf80, buff=0x52edfe0) at qpid/sys/AsynchIOHandler.cpp:135 #25 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x155, a0=..., a1=0x400) at /usr/include/boost/function/function_template.hpp:576 #26 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable ( this=0x52f1610, h=...) at qpid/sys/posix/AsynchIO.cpp:418 #27 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x155, a0=...) at /usr/include/boost/function/function_template.hpp:576 #28 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent ( this=0x52f1618, type=<value optimized out>) at qpid/sys/DispatchHandle.cpp:278 #29 0x0000003c56329fff in process (this=0x5283eb0) at qpid/sys/Poller.h:123 #30 qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:483 #31 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable ( p=0x9d) at qpid/sys/posix/Thread.cpp:35 #32 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #33 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 7 (Thread 30175): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 6 (Thread 30176): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 5 (Thread 30177): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 4 (Thread 30178): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 3 (Thread 30179): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 2 (Thread 30180): #0 0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6 #1 0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524 #2 0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:479 #3 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8) at qpid/sys/posix/Thread.cpp:35 #4 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000036a10d3d1d in clone () from /lib64/libc.so.6 Thread 1 (Thread 30166): #0 0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50 #1 weak_release (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:157 #2 boost::detail::weak_count::~weak_count (this=<value optimized out>, __in_chrg=<value optimized out>) at /usr/include/boost/detail/shared_count.hpp:262 #3 0x00002b1cac50661f in ~list (this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at /usr/include/boost/weak_ptr.hpp:27 #4 qpid::broker::PersistableMessage::~PersistableMessage ( this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at qpid/broker/PersistableMessage.cpp:34 #5 0x00002b1cac4fbaa6 in qpid::broker::Message::~Message ( this=0x2aaab0058fb0, __in_chrg=<value optimized out>) at qpid/broker/Message.cpp:59 #6 0x00002b1cade2c63c in ~intrusive_ptr (this=0x2aaaac032840, __in_chrg=<value optimized out>) at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42 #7 mrg::msgstore::DataTokenImpl::~DataTokenImpl (this=0x2aaaac032840, __in_chrg=<value optimized out>) at DataTokenImpl.cpp:30 #8 0x00002b1cade2dd70 in release (this=<value optimized out>, dtokl=...) at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42 #9 mrg::msgstore::JournalImpl::wr_aio_cb (this=<value optimized out>, dtokl=...) at JournalImpl.cpp:584 #10 0x00002b1cadea04a7 in mrg::journal::wmgr::get_events (this=0x53bd498, state=UNUSED) at jrnl/wmgr.cpp:755 #11 0x00002b1cadea2d7f in mrg::journal::wmgr::write_flush (this=0x53bd498) at jrnl/wmgr.cpp:615 #12 0x00002b1cadea314d in mrg::journal::wmgr::flush_check ( this=0x5858585858585858, res=@0x7fff8596dadc, cont=@0x7fff8596dae2, done=@0x7fff8596dae1) at jrnl/wmgr.cpp:530 #13 0x00002b1cadea4921 in mrg::journal::wmgr::enqueue (this=0x53bd498, data_buff=0x2aaab005ad40, tot_data_len=1115, this_data_len=<value optimized out>, dtokp=0x2aaab000be20, xid_ptr=0x0, xid_len=0, transient=false, external=false) at jrnl/wmgr.cpp:208 #14 0x00002b1cade7f3be in mrg::journal::jcntl::enqueue_data_record ( this=0x53bd138, data_buff=0x2aaab005ad40, tot_data_len=1115, this_data_len=1115, dtokp=0x2aaab000be20, transient=<value optimized out>) at jrnl/jcntl.cpp:206 #15 0x00002b1cade34283 in mrg::msgstore::JournalImpl::enqueue_data_record ( this=0x53bd130, data_buff=0x2aaab008fa60, tot_data_len=46912586193264, this_data_len=46912585596984, dtokp=0x2aaab0003cb8, transient=192) at JournalImpl.cpp:355 #16 0x00002b1cade3f1a7 in mrg::msgstore::MessageStoreImpl::store ( this=0x52a0710, queue=0x53bc6a0, txn=0x7fff8596e210, messageId=..., message=..., newId=<value optimized out>) at MessageStoreImpl.cpp:1474 #17 0x00002b1cade4d01b in mrg::msgstore::MessageStoreImpl::enqueue ( this=0x52a0710, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434 #18 0x00002b1cac5035cb in qpid::broker::MessageStoreModule::enqueue ( this=<value optimized out>, ctxt=0x2aaab008fa60, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124 #19 0x00002b1cac51260c in qpid::broker::Queue::enqueue (this=0x53bc6a0, ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>) at qpid/broker/Queue.cpp:753 #20 0x00002b1cac513dc6 in qpid::broker::Queue::deliver (this=0x53bc6a0, msg=...) at qpid/broker/Queue.cpp:164 #21 0x00002b1cac4b67e3 in qpid::broker::DeliverableMessage::deliverTo ( this=0x7fff8596f460, queue=...) at qpid/broker/DeliverableMessage.cpp:31 #22 0x00002b1cac4ce065 in qpid::broker::Exchange::doRoute (this=0x52a99b0, msg=..., b=...) at qpid/broker/Exchange.cpp:91 #23 0x00002b1cac56ea8c in qpid::broker::TopicExchange::route (this=0x52a99b0, msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321 #24 0x00002b1cac544bf3 in qpid::broker::SemanticState::route ( this=<value optimized out>, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461 #25 0x00002b1cac545b4d in qpid::broker::SemanticState::handle (this=0x52f1178, msg=...) at qpid/broker/SemanticState.cpp:415 #26 0x00002b1cac56a90e in qpid::broker::SessionState::handleContent ( this=0x52f0fa0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:249 #27 0x00002b1cac56aeb0 in qpid::broker::SessionState::handleIn ( this=0x52f0fa0, frame=...) at qpid/broker/SessionState.cpp:327 #28 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn ( this=0x52ee210, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #29 0x00002b1cac4a7ee2 in operator() (this=0x52ecdb0, frame=...) at qpid/framing/Handler.h:42 #30 qpid::broker::Connection::received (this=0x52ecdb0, frame=...) at qpid/broker/Connection.cpp:143 #31 0x00002b1cac487d84 in qpid::amqp_0_10::Connection::decode (this=0x52ed770, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #32 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff ( this=0x52ed710, buff=0x52abd70) at qpid/sys/AsynchIOHandler.cpp:135 #33 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0xffffffff, a0=..., a1=0x2aaab0091970) at /usr/include/boost/function/function_template.hpp:576 #34 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable ( this=0x52ab8d0, h=...) at qpid/sys/posix/AsynchIO.cpp:418 #35 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xffffffff, a0=...) at /usr/include/boost/function/function_template.hpp:576 #36 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent ( this=0x52ab8d8, type=<value optimized out>) at qpid/sys/DispatchHandle.cpp:278 #37 0x0000003c56329fff in process (this=0x5283eb0) at qpid/sys/Poller.h:123 #38 qpid::sys::Poller::run (this=0x5283eb0) at qpid/sys/epoll/EpollPoller.cpp:483 #39 0x00002b1cac4942f2 in qpid::broker::Broker::run ( this=<value optimized out>) at qpid/broker/Broker.cpp:334 #40 0x0000000000406ae6 in QpiddBroker::execute (this=<value optimized out>, options=0x5283160) at posix/QpiddBroker.cpp:176 #41 0x00000000004055af in main (argc=23, argv=0x7fff85971cf8) at qpidd.cpp:80 (gdb) quit