Description of problem: There are observed broker aborts during broker vs. perftest performace test. It looks like the broker had no more memory (from the backtrace), but there is no evidence for that. Could you possibly review the backtrace, please? Interesting part of backtrace: Thread 1 (process 25954): ---Type <return> to continue, or q <return> to quit--- #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b138c5 in raise () from /lib/tls/libc.so.6 #2 0x00b15329 in abort () from /lib/tls/libc.so.6 #3 0x00b47dca in __libc_message () from /lib/tls/libc.so.6 #4 0x00b4f423 in _int_malloc () from /lib/tls/libc.so.6 #5 0x00b510a1 in malloc () from /lib/tls/libc.so.6 #6 0x006cb427 in operator new () from /usr/lib/libstdc++.so.6 #7 0x049e4699 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0xb1905e14, __x=0x0, __p=0xb1905e18, __v=@0xb3862e00) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81 #8 0x049e47e5 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0xb1905e14, __v=@0xb3862e00) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_pair.h:85 #9 0x049e3fcb in mrg::journal::enq_map::insert_pfid (this=0xb1905e10, rid=1542206, pfid=1, locked=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_map.h:360 #10 0x049e4571 in mrg::journal::enq_map::insert_pfid (this=0x0, rid=25769803776, pfid=Variable "pfid" is not available. ) at jrnl/enq_map.cpp:67 #11 0x04a21cde in mrg::journal::wmgr::enqueue (this=0xb1906008, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0x6562, xid_ptr=0x0, xid_len=0, transient=Variable "transient" is not available. ) at ./jrnl/data_tok.hpp:146 #12 0x049f365a in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0xb0e32cf0, transient=Variable "transient" is not available. ) at jrnl/jcntl.cpp:206 #13 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0xb0e32cf0, transient=Variable "transient" is not available. ) at JournalImpl.cpp:355 #14 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb38634b0, messageId=@0xb3863500, message=@0xb3863690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462 #15 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430 #16 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at /usr/include/boost/shared_ptr.hpp:252 #17 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb1970fb8}, suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #18 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb3863fb8) at ./boost/detail/atomic_count_gcc.hpp:47 #19 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb3863fb0, queue=@0x0) at /usr/include/boost/shared_ptr.hpp:252 #20 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb3863fb0, b={px = 0xb199dae8, pn = {pi_ = 0xb196f018}}) at /usr/include/boost/shared_ptr.hpp:614 Issue seen on RHEL4.8 i386 only atm. Version-Release number of selected component (if applicable): qpid-cpp-client-0.7.935473-1.el4 qpid-cpp-server-devel-0.7.935473-1.el4 qpid-cpp-client-ssl-0.7.935473-1.el4 qpid-cpp-server-store-0.7.935473-1.el4 qpid-cpp-client-devel-0.7.935473-1.el4 qpid-cpp-server-ssl-0.7.935473-1.el4 qpid-cpp-server-xml-0.7.935473-1.el4 qpid-cpp-server-0.7.935473-1.el4 qpid-cpp-mrg-debuginfo-0.7.935473-1.el4 qpid-cpp-client-devel-docs-0.7.935473-1.el4 How reproducible: very hard (<10%) Steps to Reproduce: 1. take MRG/Messaging/qpid_test_qpidd-perftest_performance test The test loops perftest where perftest parameters are swept/variated with different parameter combinations 2. run the test 'TEST_PARAM_RHTS_TEST_MRG_QPID_TEST_QPIDD_PERFTEST_PERFORMANCE_LOOPS=10 make rb' 3. wait for abort/faulire Actual results: Broker rarely aborts. Expected results: Broker should not abort. Additional info: [22:31:48] mrg_qpidd_start: qpidd launched normal bg way (port:40821,log:qpidd.transcript.log,params: --auth no --log-enable info+ --num-jfiles 64 --jf ile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 64 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 16 --mgmt-enable 0 --data-dir /root/MRG/Messag ing/qpid_test_qpidd-perftest_performance/rhts_qpidd/20100429_111616/b.4938,lockfile:qpidd.lock) [22:32:29] mrg_qpidd_wait_on_settle: qpidd started-up (dur:1sec) [22:32:29] .qpidd settled [22:32:29] launching perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 40821 --mode topic --qt 1 --durable yes --npubs 2 --nsub s 3 --count 200000 --size 1024 --tx 0 --async-commit no ................................................................/root/MRG/Messaging/qpid_common/qpid_test_functions_broker.sh: line 51: 25942 Aborted (core dumped) ${qpidd_wrapper} ${QPIDD_DIR}/${MRG_CPP_BROKER} ${datadir_sw} --port ${port} ${test_params} >${transcript} 2>${transcript} . [22:33:39] .perftest went ok (ecode:0, summary:; dur/to:68.65/740) [22:33:39] .qpidd status check [22:33:39] mrg_qpidd_status: 0 instance[s] running (pids:,ports:) [22:34:16] ..ERROR:qpidd not running fine! (pid[s]:, port[s]:, inst_cnt:0) [22:34:16] .qpidd stop [22:34:16] mrg_qpidd_stop: No qpidd broker found to stop! [22:34:16] .qpidd stopped ok [22:34:16] .remove qpidd parent lock (if needed), wait&log... broker log: 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/ssl.so 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/xml.so 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/acl.so 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so 2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so 2010-04-29 22:31:48 info No message store configured, persistence is disabled. 2010-04-29 22:31:48 info Registered xml exchange 2010-04-29 22:31:49 notice Journal "TplStore": Created 2010-04-29 22:31:49 notice Store module initialized; store-dir=/root/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20100429_111616/b.49 38 2010-04-29 22:31:49 info > Default files per journal: 64 2010-04-29 22:31:49 info > Default journal file size: 64 (wpgs) 2010-04-29 22:31:49 info > Default write cache page size: 64 (Kib) 2010-04-29 22:31:49 info > Default number of write cache pages: 16 2010-04-29 22:31:49 info > TPL files per journal: 64 2010-04-29 22:31:49 info > TPL journal file size: 64 (wpgs) 2010-04-29 22:31:49 info > TPL write cache page size: 16 (Kib) 2010-04-29 22:31:49 info > TPL number of write cache pages: 32 2010-04-29 22:31:49 info Registered replication exchange 2010-04-29 22:31:49 info Most recent persistence id found: 0x0 2010-04-29 22:31:49 info Management not enabled 2010-04-29 22:31:49 notice SASL disabled: No Authentication Performed 2010-04-29 22:31:49 notice Listening on TCP port 40821 2010-04-29 22:31:49 info SSL plugin not enabled, you must set --ssl-cert-db to enable it. 2010-04-29 22:31:49 info Policy file not specified. ACL Disabled, no ACL checking being done! 2010-04-29 22:31:49 notice Broker running 2010-04-29 22:32:29 info Queue "perftest_pub_start": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:29 info Queue "perftest_pub_done": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:29 info Queue "perftest_sub_ready": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:29 info Queue "perftest_sub_done": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:29 notice Journal "anonymous.2bbed682-2176-4e65-af0e-a8219cbcf011": Created 2010-04-29 22:32:29 notice Journal "anonymous.f0437860-d222-409d-8846-f14964f3bb11": Created 2010-04-29 22:32:29 notice Journal "anonymous.9a924f9e-081a-44e0-8e81-e7d449d9c2ef": Created 2010-04-29 22:32:40 info Queue "anonymous.f0437860-d222-409d-8846-f14964f3bb11": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:40 info Queue "anonymous.2bbed682-2176-4e65-af0e-a8219cbcf011": Policy created: type=reject; maxCount=0; maxSize=104857600 2010-04-29 22:32:40 info Queue "anonymous.9a924f9e-081a-44e0-8e81-e7d449d9c2ef": Policy created: type=reject; maxCount=0; maxSize=104857600 perftest log: run 0 - perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 40821 --mode topic --qt 1 --durable yes --npubs 2 --nsubs 3 --count 2 00000 --size 1024 --tx 0 --async-commit no 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:32:29 info Connecting to tcp:localhost:40821 2010-04-29 22:33:38 warning Connection closed 2010-04-29 22:33:38 warning Connection closed 2010-04-29 22:33:38 warning Connection closed 2010-04-29 22:33:38 warning Connection closed 2010-04-29 22:33:38 warning Connection closed 2010-04-29 22:33:38 warning Connection closed PublishThread exception: Connection closed SubscribeThread exception: Connection closedSubscribeThread exception: Connection closedPublishThread exception: Connection closed SubscribeThread exception: Connection closed Controller exception: Connection closed core.25942: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qpidd' [root@mrg-qe-07 qpid_test_qpidd-perftest_performance]# ll core.* -rw------- 1 root root 117088256 Apr 29 22:33 core.25942 [root@mrg-qe-07 qpid_test_qpidd-perftest_performance]# gdb `which qpidd` core.25942 GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'. Program terminated with signal 6, Aborted. Reading symbols from /usr/lib/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libqpidbroker.so.2 Reading symbols from /usr/lib/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libqpidcommon.so.2 Reading symbols from /usr/lib/libboost_program_options.so.1...done. Loaded symbols for /usr/lib/libboost_program_options.so.1 Reading symbols from /usr/lib/libboost_filesystem.so.1...done. Loaded symbols for /usr/lib/libboost_filesystem.so.1 Reading symbols from /lib/tls/libuuid.so.1...done. Loaded symbols for /lib/tls/libuuid.so.1 Reading symbols from /lib/tls/librt.so.1...done. Loaded symbols for /lib/tls/librt.so.1 Reading symbols from /usr/lib/libsasl2.so.2...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /usr/lib/libstdc++.so.6...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/tls/libm.so.6...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/tls/libc.so.6...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/tls/libpthread.so.0...done. Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/ssl.so Reading symbols from /usr/lib/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libsslcommon.so.2 Reading symbols from /usr/lib/libnss3.so...done. Loaded symbols for /usr/lib/libnss3.so Reading symbols from /usr/lib/libssl3.so...done. Loaded symbols for /usr/lib/libssl3.so Reading symbols from /usr/lib/libnspr4.so...done. Loaded symbols for /usr/lib/libnspr4.so Reading symbols from /usr/lib/libnssutil3.so...done. Loaded symbols for /usr/lib/libnssutil3.so Reading symbols from /usr/lib/libplc4.so...done. Loaded symbols for /usr/lib/libplc4.so Reading symbols from /usr/lib/libplds4.so...done. Loaded symbols for /usr/lib/libplds4.so Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/xml.so Reading symbols from /usr/lib/libxerces-c.so.28...done. Loaded symbols for /usr/lib/libxerces-c.so.28 Reading symbols from /usr/lib/libxqilla.so.3...done. Loaded symbols for /usr/lib/libxqilla.so.3 Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/acl.so Reading symbols from /usr/lib/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/msgstore.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/msgstore.so Reading symbols from /usr/lib/tls/i686/libdb_cxx-4.2.so...done. Loaded symbols for /usr/lib/tls/i686/libdb_cxx-4.2.so Reading symbols from /usr/lib/libaio.so.1...done. Loaded symbols for /usr/lib/libaio.so.1 Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) info threads 11 process 25942 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 10 process 25947 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 9 process 25948 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 8 process 25949 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 7 process 25950 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 6 process 25951 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 5 process 25952 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 4 process 25953 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 3 process 25955 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 2 process 25956 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 * 1 process 25954 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 (gdb) thread apply all bt Thread 11 (process 25942): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #5 0x0083f2f0 in qpid::broker::Broker::run (this=0x835cb88) at qpid/broker/Broker.cpp:334 #6 0x08057a15 in QpiddBroker::execute (this=0xbff6fbf7, options=0x834cfb8) at /usr/include/boost/intrusive_ptr.hpp:119 #7 0x08050aea in main (argc=23, argv=0xbff6fe54) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301 #8 0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6 #9 0x0804c571 in _start () Thread 10 (process 25947): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00d5417c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0 #2 0x00bc4b1e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6 #3 0x005323af in qpid::sys::Timer::run (this=0x835cbb0) at ../include/qpid/sys/posix/Condition.h:69 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 9 (process 25948): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00d5417c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0 #2 0x00bc4b1e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6 #3 0x005323af in qpid::sys::Timer::run (this=0x8362eb8) at ../include/qpid/sys/posix/Condition.h:69 #4 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #5 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #6 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 8 (process 25949): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bc4e1e in __lll_mutex_lock_wait () from /lib/tls/libc.so.6 #2 0x00b53b9b in _L_mutex_lock_4191 () from /lib/tls/libc.so.6 #3 0x00000000 in ?? () Thread 7 (process 25950): ---Type <return> to continue, or q <return> to quit--- #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00d565de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0 #2 0x00d5320b in _L_mutex_lock_35 () from /lib/tls/libpthread.so.0 #3 0x0846e568 in ?? () #4 0x00c17ff4 in ?? () from /lib/tls/libc.so.6 #5 0xb1905ddc in ?? () #6 0xb1906008 in ?? () #7 0xb6067028 in ?? () #8 0x00bc4cfe in pthread_mutex_lock () from /lib/tls/libc.so.6 #9 0x00bc4cfe in pthread_mutex_lock () from /lib/tls/libc.so.6 #10 0x049f3602 in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb19713d8, tot_data_len=1115, this_data_len=1115, dtokp=0xb19535a8, transient=false) at ./jrnl/slock.hpp:56 #11 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb19713d8, tot_data_len=1115, this_data_len=1115, dtokp=0xb19535a8, transient=Variable "transient" is not available. ) at JournalImpl.cpp:355 #12 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb60674b0, messageId=@0xb6067500, message=@0xb6067690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462 #13 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb6067690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430 #14 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb6067690, queue=@0xb1905320) at /usr/include/boost/shared_ptr.hpp:252 #15 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb0d28098}, suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #16 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb6067fb8) at ./boost/detail/atomic_count_gcc.hpp:47 #17 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb6067fb0, queue=@0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #18 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb6067fb0, b={px = 0xb0d1b820, pn = {pi_ = 0xb0d457f0}}) at /usr/include/boost/shared_ptr.hpp:614 #19 0x0096a737 in qpid::broker::TopicExchange::route (this=0x8363a08, msg=@0xb6067fb0, routingKey=@0xb6067d20) at ./boost/detail/sp_counted_base_gcc_x86.hpp:66 #20 0x00927ef4 in qpid::broker::SemanticState::route (this=0x83eaa04, msg={p_ = 0xb0d28098}, strategy=@0xb6067fb0) at /usr/include/boost/shared_ptr.hpp:252 #21 0x00928ab1 in qpid::broker::SemanticState::handle (this=0x83eaa04, msg={p_ = 0xb0d28098}) at ./boost/detail/atomic_count_gcc.hpp:47 #22 0x00961b66 in qpid::broker::SessionState::handleContent (this=0x83ea8c8, frame=@0xb60687f0, id=@0xfffffffc) at ./boost/detail/atomic_count_gcc.hpp:47 #23 0x00963fdf in qpid::broker::SessionState::handleIn (this=0x83ea8c8, frame=@0xb60687f0) at qpid/broker/SessionState.cpp:327 #24 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xfffffffc, t=@0xfffffffc) at ./qpid/framing/Handler.h:67 #25 0x004c98dc in qpid::amqp_0_10::SessionHandler::handleIn (this=0x83a9920, f=@0xb60687f0) at qpid/amqp_0_10/SessionHandler.cpp:93 #26 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, ---Type <return> to continue, or q <return> to quit--- &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xfffffffc, t=@0xfffffffc) at ./qpid/framing/Handler.h:67 #27 0x0085ed98 in qpid::broker::Connection::received (this=0x83e9ee0, frame=@0xb60687f0) at ./qpid/framing/Handler.h:42 #28 0x0082bc79 in qpid::amqp_0_10::Connection::decode (this=0x83a9158, buffer=0x83b9ab8 "\a\003\004\f", size=65536) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285 #29 0x009205c4 in qpid::broker::SecureConnection::decode (this=0x8364da8, buffer=0x83b9ab8 "\a\003\004\f", size=0) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285 #30 0x0051fb4c in qpid::sys::AsynchIOHandler::readbuff (this=0x83a6d60, buff=0x83a6818) at qpid/sys/AsynchIOHandler.cpp:135 #31 0x009b7736 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr= {obj_ptr = 0x83a61f0, const_obj_ptr = 0x83a61f0, func_ptr = 0x83a61f0, data = "�"}, a0=@0x83a8ef0, a1=0x83a6818) at /usr/include/boost/bind/mem_fn_template.hpp:252 #32 0x004119c7 in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0xd5aff4, a0=@0xfffffffc, a1=0xfffffffc) at /usr/include/boost/function/function_template.hpp:581 #33 0x004106b2 in qpid::sys::posix::AsynchIO::readable (this=0x83a8ef0, h=@0x83a8ef4) at qpid/sys/posix/AsynchIO.cpp:418 #34 0x0041249c in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr={obj_ptr = 0x83a6268, const_obj_ptr = 0x83a6268, func_ptr = 0x83a6268, data = "h"}, a0=@0xfffffffc) at /usr/include/boost/bind/mem_fn_template.hpp:149 #35 0x00529950 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xd5aff4, a0=@0xfffffffc) at /usr/include/boost/function/function_template.hpp:581 #36 0x005282c4 in qpid::sys::DispatchHandle::processEvent (this=0x83a8ef4, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:289 #37 0x00423758 in qpid::sys::Poller::run (this=0x8358e18) at ./qpid/sys/Poller.h:123 #38 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #39 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #40 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #41 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 6 (process 25951): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #5 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #6 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #7 0x00bb7fae in clone () from /lib/tls/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 5 (process 25952): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #5 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #6 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #7 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 4 (process 25953): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #5 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #6 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #7 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 3 (process 25955): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bb862e in epoll_wait () from /lib/tls/libc.so.6 #2 0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252 #5 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35 #6 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #7 0x00bb7fae in clone () from /lib/tls/libc.so.6 Thread 2 (process 25956): #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00bc4e1e in __lll_mutex_lock_wait () from /lib/tls/libc.so.6 #2 0x00b53b9b in _L_mutex_lock_4191 () from /lib/tls/libc.so.6 #3 0x00000000 in ?? () Thread 1 (process 25954): ---Type <return> to continue, or q <return> to quit--- #0 0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2 #1 0x00b138c5 in raise () from /lib/tls/libc.so.6 #2 0x00b15329 in abort () from /lib/tls/libc.so.6 #3 0x00b47dca in __libc_message () from /lib/tls/libc.so.6 #4 0x00b4f423 in _int_malloc () from /lib/tls/libc.so.6 #5 0x00b510a1 in malloc () from /lib/tls/libc.so.6 #6 0x006cb427 in operator new () from /usr/lib/libstdc++.so.6 #7 0x049e4699 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0xb1905e14, __x=0x0, __p=0xb1905e18, __v=@0xb3862e00) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81 #8 0x049e47e5 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0xb1905e14, __v=@0xb3862e00) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_pair.h:85 #9 0x049e3fcb in mrg::journal::enq_map::insert_pfid (this=0xb1905e10, rid=1542206, pfid=1, locked=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_map.h:360 #10 0x049e4571 in mrg::journal::enq_map::insert_pfid (this=0x0, rid=25769803776, pfid=Variable "pfid" is not available. ) at jrnl/enq_map.cpp:67 #11 0x04a21cde in mrg::journal::wmgr::enqueue (this=0xb1906008, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0x6562, xid_ptr=0x0, xid_len=0, transient=Variable "transient" is not available. ) at ./jrnl/data_tok.hpp:146 #12 0x049f365a in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0xb0e32cf0, transient=Variable "transient" is not available. ) at jrnl/jcntl.cpp:206 #13 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0xb0e32cf0, transient=Variable "transient" is not available. ) at JournalImpl.cpp:355 #14 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb38634b0, messageId=@0xb3863500, message=@0xb3863690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462 #15 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430 #16 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at /usr/include/boost/shared_ptr.hpp:252 #17 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb1970fb8}, suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #18 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb3863fb8) at ./boost/detail/atomic_count_gcc.hpp:47 #19 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb3863fb0, queue=@0x0) at /usr/include/boost/shared_ptr.hpp:252 #20 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb3863fb0, b={px = 0xb199dae8, pn = {pi_ = 0xb196f018}}) at /usr/include/boost/shared_ptr.hpp:614 #21 0x0096a737 in qpid::broker::TopicExchange::route (this=0x8363a08, msg=@0xb3863fb0, routingKey=@0xb3863d20) at ./boost/detail/sp_counted_base_gcc_x86.hpp:66 #22 0x00927ef4 in qpid::broker::SemanticState::route (this=0x83a8ccc, msg={p_ = 0xb1970fb8}, strategy=@0xb3863fb0) ---Type <return> to continue, or q <return> to quit--- at /usr/include/boost/shared_ptr.hpp:252 #23 0x00928ab1 in qpid::broker::SemanticState::handle (this=0x83a8ccc, msg={p_ = 0xb1970fb8}) at ./boost/detail/atomic_count_gcc.hpp:47 #24 0x00961b66 in qpid::broker::SessionState::handleContent (this=0x83a8b90, frame=@0xb38647f0, id=@0x0) at ./boost/detail/atomic_count_gcc.hpp:47 #25 0x00963fdf in qpid::broker::SessionState::handleIn (this=0x83a8b90, frame=@0xb38647f0) at qpid/broker/SessionState.cpp:327 #26 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x0, t=@0x0) at ./qpid/framing/Handler.h:67 #27 0x004c98dc in qpid::amqp_0_10::SessionHandler::handleIn (this=0x83a66f0, f=@0xb38647f0) at qpid/amqp_0_10/SessionHandler.cpp:93 #28 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x0, t=@0x0) at ./qpid/framing/Handler.h:67 #29 0x0085ed98 in qpid::broker::Connection::received (this=0x83a6388, frame=@0xb38647f0) at ./qpid/framing/Handler.h:42 #30 0x0082bc79 in qpid::amqp_0_10::Connection::decode (this=0x83658b0, buffer=0x8385998 "\a\003\004\f", size=65536) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285 #31 0x009205c4 in qpid::broker::SecureConnection::decode (this=0x8364fa0, buffer=0x8385998 "\a\003\004\f", size=25954) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285 #32 0x0051fb4c in qpid::sys::AsynchIOHandler::readbuff (this=0x8364e20, buff=0x8364de8) at qpid/sys/AsynchIOHandler.cpp:135 #33 0x009b7736 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr= {obj_ptr = 0x8364dd0, const_obj_ptr = 0x8364dd0, func_ptr = 0x8364dd0, data = "�"}, a0=@0x8365078, a1=0x8364de8) at /usr/include/boost/bind/mem_fn_template.hpp:252 #34 0x004119c7 in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x6, a0=@0x0, a1=0x0) at /usr/include/boost/function/function_template.hpp:581 #35 0x004106b2 in qpid::sys::posix::AsynchIO::readable (this=0x8365078, h=@0x836507c) at qpid/sys/posix/AsynchIO.cpp:418 #36 0x0041249c in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr={obj_ptr = 0x8364ee8, const_obj_ptr = 0x8364ee8, func_ptr = 0x8364ee8, data = "�"}, a0=@0x0) at /usr/include/boost/bind/mem_fn_template.hpp:149 #37 0x00529950 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x6, a0=@0x0) at /usr/include/boost/function/function_template.hpp:581 #38 0x005282c4 in qpid::sys::DispatchHandle::processEvent (this=0x836507c, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:289 #39 0x00423758 in qpid::sys::Poller::run (this=0x8358e18) at ./qpid/sys/Poller.h:123 #40 0x0052d51e in qpid::sys::Dispatcher::run (this=0x0) at /usr/include/boost/shared_ptr.hpp:252 #41 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0x0) at qpid/sys/posix/Thread.cpp:35 #42 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0 #43 0x00bb7fae in clone () from /lib/tls/libc.so.6
Seen also on RHEL 5.5, multiple cases cases furing the perftest performance test -> How reproducible: moderate (50%) Again no info from qpidd logs about not enough of memory. GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done. done. [New Thread 10382] [New Thread 10381] [New Thread 10380] [New Thread 10379] [New Thread 10378] [New Thread 10377] [New Thread 10376] [New Thread 10375] [New Thread 10374] [New Thread 10369] warning: .dynamic section for "/usr/lib/openais/libcpg.so.2" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libxerces-c.so.28" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libqpidbroker.so.2 Reading symbols from /usr/lib/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libqpidcommon.so.2 Reading symbols from /usr/lib/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib/libboost_program_options.so.1.33.1.debug...done. done. Loaded symbols for /usr/lib/libboost_program_options.so.2 Reading symbols from /usr/lib/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib/libboost_filesystem.so.1.33.1.debug...done. done. Loaded symbols for /usr/lib/libboost_filesystem.so.2 Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libuuid.so.1 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /usr/lib/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/msgstore.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/msgstore.so Reading symbols from /usr/lib/libdb_cxx-4.3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb_cxx-4.3.so Reading symbols from /usr/lib/libaio.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libaio.so.1 Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/ssl.so Reading symbols from /usr/lib/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libsslcommon.so.2 Reading symbols from /usr/lib/libnss3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnss3.so Reading symbols from /usr/lib/libssl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl3.so Reading symbols from /usr/lib/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnspr4.so Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnssutil3.so Reading symbols from /usr/lib/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libplc4.so Reading symbols from /usr/lib/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libplds4.so Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/cluster.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/cluster.so Reading symbols from /usr/lib/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib/openais/libcpg.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/openais/libcpg.so.2 Reading symbols from /usr/lib/libcman.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcman.so.2 Reading symbols from /usr/lib/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidclient.so.2.0.0.debug...done. done. Loaded symbols for /usr/lib/libqpidclient.so.2 Reading symbols from /usr/lib/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/sslconnector.so.debug...done. done. Loaded symbols for /usr/lib/qpid/client/sslconnector.so Reading symbols from /usr/lib/qpid/daemon/watchdog.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/watchdog.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/watchdog.so Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/xml.so Reading symbols from /usr/lib/libxerces-c.so.28...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libxerces-c.so.28 Reading symbols from /usr/lib/libxqilla.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libxqilla.so.3 Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done. done. Loaded symbols for /usr/lib/qpid/daemon/acl.so Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'. Program terminated with signal 6, Aborted. #0 0x00707410 in __kernel_vsyscall () (gdb) eax 0x0 0 ecx 0x288f 10383 edx 0x6 6 ebx 0x2881 10369 esp 0xb23ef4f8 0xb23ef4f8 ebp 0xb23ef510 0xb23ef510 esi 0xb23ef5b0 -1304496720 edi 0x368ff4 3575796 eip 0x707410 0x707410 <__kernel_vsyscall+16> eflags 0x206 [ PF IF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 st0 0 (raw 0x00000000000000000000) st1 0 (raw 0x00000000000000000000) st2 0 (raw 0x00000000000000000000) st3 0 (raw 0x00000000000000000000) st4 0 (raw 0x00000000000000000000) st5 0 (raw 0x00000000000000000000) st6 0 (raw 0x00000000000000000000) st7 0 (raw 0x00000000000000000000) fctrl 0x0 0 fstat 0x0 0 ftag 0x0 0 fiseg 0x0 0 fioff 0x0 0 foseg 0x0 0 fooff 0x0 0 fop 0x0 0 xmm0 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm1 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm2 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm3 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm4 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm5 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm6 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} xmm7 {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000} mxcsr 0x0 [ ] mm0 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm1 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm2 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm3 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm4 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm5 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm6 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} mm7 {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}} (gdb) Using memory regions provided by the target. There are no memory regions defined. (gdb) From To Syms Read Shared Object Library 0x0083bc30 0x009f5554 Yes (*) /usr/lib/libqpidbroker.so.2 0x004aae10 0x005cba24 Yes (*) /usr/lib/libqpidcommon.so.2 0x006bf970 0x006de524 Yes (*) /usr/lib/libboost_program_options.so.2 0x006a39a0 0x006ac614 Yes (*) /usr/lib/libboost_filesystem.so.2 0x077ccee0 0x077ce954 Yes (*) /lib/libuuid.so.1 0x00398a50 0x00399a84 Yes (*) /lib/libdl.so.2 0x003ce880 0x003d2b04 Yes (*) /lib/librt.so.1 0x077da190 0x077ea774 Yes (*) /usr/lib/libsasl2.so.2 0x071dac50 0x07256134 Yes (*) /usr/lib/libstdc++.so.6 0x00372410 0x0038d594 Yes (*) /lib/libm.so.6 0x00b37660 0x00b3ef34 Yes (*) /lib/libgcc_s.so.1 0x0023cc80 0x00337c40 Yes (*) /lib/libc.so.6 0x002097f0 0x0021eebf Yes (*) /lib/ld-linux.so.2 0x003a2210 0x003ada74 Yes (*) /lib/libpthread.so.0 0x0073b0c0 0x00746054 Yes (*) /lib/libresolv.so.2 0x074326e0 0x07439314 Yes (*) /lib/libcrypt.so.1 0x00cf7380 0x00d9a4c4 Yes (*) /usr/lib/qpid/daemon/msgstore.so 0x00bd3240 0x00c9b554 Yes (*) /usr/lib/libdb_cxx-4.3.so 0x00110390 0x001106d0 Yes (*) /usr/lib/libaio.so.1 0x0011a0f0 0x001204f4 Yes (*) /usr/lib/qpid/daemon/replicating_listener.so 0x0012b240 0x00133c64 Yes (*) /usr/lib/qpid/daemon/ssl.so 0x001417a0 0x0015abf4 Yes (*) /usr/lib/libsslcommon.so.2 0x04d7afc0 0x04e5ea04 Yes (*) /usr/lib/libnss3.so 0x04efb480 0x04f21664 Yes (*) /usr/lib/libssl3.so 0x04e9a7a0 0x04ebd3f4 Yes (*) /usr/lib/libnspr4.so 0x04d4db40 0x04d59284 Yes (*) /usr/lib/libnssutil3.so 0x04d64df0 0x04d66704 Yes (*) /usr/lib/libplc4.so 0x00204a30 0x002059a4 Yes (*) /usr/lib/libplds4.so 0x003b96a0 0x003c48d4 Yes (*) /usr/lib/libz.so.1 0x00a91000 0x00b08534 Yes (*) /usr/lib/qpid/daemon/cluster.so 0x00161e80 0x00164024 Yes (*) /usr/lib/openais/libcpg.so.2 0x00166d30 0x00169894 Yes (*) /usr/lib/libcman.so.2 0x00e0e9b0 0x00ee5cc4 Yes (*) /usr/lib/libqpidclient.so.2 0x001724f0 0x001803f4 Yes (*) /usr/lib/qpid/client/sslconnector.so 0x00188f20 0x0018cb54 Yes (*) /usr/lib/qpid/daemon/watchdog.so 0x00193250 0x001985f4 Yes (*) /usr/lib/qpid/daemon/replication_exchange.so 0x007a6a50 0x007b1d44 Yes (*) /usr/lib/qpid/daemon/xml.so 0x010510a0 0x0120e534 Yes (*) /usr/lib/libxerces-c.so.28 0x078fec00 0x07ab41d4 Yes (*) /usr/lib/libxqilla.so.3 0x001a58b0 0x001c7ae4 Yes (*) /usr/lib/qpid/daemon/acl.so (*): Shared library is missing debugging information. (gdb) 11 Thread 10369 0x00707410 in __kernel_vsyscall () 10 Thread 10374 0x00707410 in __kernel_vsyscall () 9 Thread 10375 0x00707410 in __kernel_vsyscall () 8 Thread 10376 0x00707410 in __kernel_vsyscall () 7 Thread 10377 0x00707410 in __kernel_vsyscall () 6 Thread 10378 0x00707410 in __kernel_vsyscall () 5 Thread 10379 0x0032eea4 in _dl_addr () from /lib/libc.so.6 4 Thread 10380 0x00707410 in __kernel_vsyscall () 3 Thread 10381 0x00707410 in __kernel_vsyscall () 2 Thread 10382 0x00707410 in __kernel_vsyscall () * 1 Thread 10383 0x00707410 in __kernel_vsyscall () (gdb) Thread 11 (Thread 10369): #0 0x00707410 in __kernel_vsyscall () #1 0x002f9486 in epoll_wait () from /lib/libc.so.6 #2 0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #5 0x008ca692 in qpid::broker::Broker::run (this=0x9cf7e80) at qpid/broker/Broker.cpp:334 #6 0x0804debe in QpiddBroker::execute (this=0xbfef2b75, options=0x9cf3608) at posix/QpiddBroker.cpp:176 #7 0x0804c821 in main (argc=23, argv=0xbfef2c24) at qpidd.cpp:80 Thread 10 (Thread 10374): #0 0x00707410 in __kernel_vsyscall () #1 0x003a7ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00305524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x005c6eb4 in qpid::sys::Timer::run (this=0x9cf7ea8) at ../include/qpid/sys/posix/Condition.h:69 #4 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0x9cf7ea8) at qpid/sys/posix/Thread.cpp:35 #5 0x003a3832 in start_thread () from /lib/libpthread.so.0 #6 0x002f8e0e in clone () from /lib/libc.so.6 Thread 9 (Thread 10375): #0 0x00707410 in __kernel_vsyscall () #1 0x003a7ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00305524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x005c6eb4 in qpid::sys::Timer::run (this=0x9ce4d78) at ../include/qpid/sys/posix/Condition.h:69 #4 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0x9ce4d78) at qpid/sys/posix/Thread.cpp:35 #5 0x003a3832 in start_thread () from /lib/libpthread.so.0 #6 0x002f8e0e in clone () from /lib/libc.so.6 Thread 8 (Thread 10376): #0 0x00707410 in __kernel_vsyscall () #1 0x003aa6e9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x003a5d9f in _L_lock_885 () from /lib/libpthread.so.0 #3 0x003a5c66 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x00305646 in pthread_mutex_lock () from /lib/libc.so.6 #5 0x00d47b5d in mrg::journal::slock::slock(._4 *) (this=0xb69f8278, m=0x9e02524) at jrnl/slock.hpp:56 #6 0x00d540e7 in mrg::journal::jcntl::dequeue_data_record (this=0x9e020c4, dtokp=0xb0c3e6f8, txn_coml_commit=false) at jrnl/jcntl.cpp:304 #7 0x00d00fc1 in mrg::msgstore::JournalImpl::dequeue_data_record ( this=0x9e020c0, dtokp=0xb0c3e6f8, txn_coml_commit=false) at JournalImpl.cpp:416 #8 0x00d1bc75 in mrg::msgstore::MessageStoreImpl::async_dequeue ( this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1545 #9 0x00d1c39b in mrg::msgstore::MessageStoreImpl::dequeue (this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1520 #10 0x0093a0cd in qpid::broker::MessageStoreModule::dequeue (this=0x9ce3930, ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:131 #11 0x00947c6b in qpid::broker::Queue::dequeue (this=0x9e017b0, ctxt=0x0, msg=...) at qpid/broker/Queue.cpp:784 #12 0x009786e5 in qpid::broker::SemanticState::ConsumerImpl::deliver ( this=0xb18dcc60, msg=...) at qpid/broker/SemanticState.cpp:323 #13 0x00951698 in qpid::broker::Queue::dispatch (this=0x9e017b0, c=...) at qpid/broker/Queue.cpp:399 #14 0x00978c4c in qpid::broker::SemanticState::ConsumerImpl::doOutput ( this=0xb18dcc60) at qpid/broker/SemanticState.cpp:685 #15 0x005b8c08 in qpid::sys::AggregateOutput::doOutput (this=0xb1844968) at qpid/sys/AggregateOutput.cpp:59 #16 0x008d612c in qpid::broker::Connection::doOutput (this=0xb1844930) at qpid/broker/Connection.cpp:282 #17 0x008b1b9e in qpid::amqp_0_10::Connection::canEncode (this=0xb1803af8) at qpid/amqp_0_10/Connection.cpp:67 #18 0x0097681b in qpid::broker::SecureConnection::canEncode (this=0xb1803800) at qpid/broker/SecureConnection.cpp:58 #19 0x005bbd19 in qpid::sys::AsynchIOHandler::idle (this=0xb1803740) at qpid/sys/AsynchIOHandler.cpp:210 #20 0x009f388d in operator() (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/mem_fn_template.hpp:149 #21 operator()<boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list1<qpid::sys::AsynchIO&> > (function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 #22 operator()<qpid::sys::AsynchIO> (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/bind_template.hpp:32 #23 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list2<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1> > >, void, qpid::sys::AsynchIO&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:136 #24 0x004bf737 in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xb1803978, a0=...) at /usr/include/boost/function/function_template.hpp:576 #25 0x004bcb56 in qpid::sys::posix::AsynchIO::writeable (this=0xb1803898, h=...) at qpid/sys/posix/AsynchIO.cpp:536 #26 0x004be56d in operator() (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/mem_fn_template.hpp:149 #27 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > ( function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 #28 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/bind_template.hpp:32 #29 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:136 #30 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xb18038b0, a0=...) at /usr/include/boost/function/function_template.hpp:576 #31 0x005bfc99 in qpid::sys::DispatchHandle::processEvent (this=0xb180389c, type=WRITABLE) at qpid/sys/DispatchHandle.cpp:285 #32 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123 #33 qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:483 #34 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #35 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #36 0x003a3832 in start_thread () from /lib/libpthread.so.0 #37 0x002f8e0e in clone () from /lib/libc.so.6 Thread 7 (Thread 10377): #0 0x00707410 in __kernel_vsyscall () #1 0x002f9486 in epoll_wait () from /lib/libc.so.6 #2 0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #5 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #6 0x003a3832 in start_thread () from /lib/libpthread.so.0 #7 0x002f8e0e in clone () from /lib/libc.so.6 Thread 6 (Thread 10378): #0 0x00707410 in __kernel_vsyscall () #1 0x002f9486 in epoll_wait () from /lib/libc.so.6 #2 0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #5 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #6 0x003a3832 in start_thread () from /lib/libpthread.so.0 #7 0x002f8e0e in clone () from /lib/libc.so.6 Thread 5 (Thread 10379): #0 0x0032eea4 in _dl_addr () from /lib/libc.so.6 #1 0x0030c4f5 in backtrace_symbols_fd () from /lib/libc.so.6 #2 0x002882e7 in __libc_message () from /lib/libc.so.6 #3 0x002905a5 in _int_free () from /lib/libc.so.6 #4 0x002909e9 in free () from /lib/libc.so.6 #5 0x0724e581 in operator delete(void*) () from /usr/lib/libstdc++.so.6 #6 0x00d1c892 in deallocate (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=..., message=..., newId=false) at /usr/include/c++/4.1.2/ext/new_allocator.h:94 #7 _M_deallocate (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=..., message=..., newId=false) at /usr/include/c++/4.1.2/bits/stl_vector.h:133 #8 ~_Vector_base (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=..., message=..., newId=false) at /usr/include/c++/4.1.2/bits/stl_vector.h:119 #9 ~vector (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=..., message=..., newId=false) at /usr/include/c++/4.1.2/bits/stl_vector.h:272 #10 mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=..., message=..., newId=false) at MessageStoreImpl.cpp:1493 #11 0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434 #12 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930, ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124 #13 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9dbe590, ctxt=0x0, msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #14 0x0094e628 in qpid::broker::Queue::deliver (this=0x9dbe590, msg=...) at qpid/broker/Queue.cpp:164 #15 0x008e457e in qpid::broker::DeliverableMessage::deliverTo ( this=0xb4bf5040, queue=...) at qpid/broker/DeliverableMessage.cpp:31 #16 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=..., b=...) at qpid/broker/Exchange.cpp:91 #17 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321 #18 0x009817ff in qpid::broker::SemanticState::route (this=0x9d3bef4, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461 #19 0x00982b9c in qpid::broker::SemanticState::handle (this=0x9d3bef4, msg=...) at qpid/broker/SemanticState.cpp:415 #20 0x009a8306 in qpid::broker::SessionState::handleContent (this=0x9d3bdb8, frame=..., id=...) at qpid/broker/SessionState.cpp:249 #21 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0x9d3bdb8, frame=...) at qpid/broker/SessionState.cpp:327 #22 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0x9d3becc, t=...) at qpid/framing/Handler.h:67 #23 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x9ce7738, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #24 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0x9ce7740, t=...) at qpid/framing/Handler.h:67 #25 0x008d41b0 in operator() (this=0x9ce7378, frame=...) at qpid/framing/Handler.h:42 #26 qpid::broker::Connection::received (this=0x9ce7378, frame=...) at qpid/broker/Connection.cpp:143 #27 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0x9ce8190, buffer=0x9d099c0 "\a\003\004\f", size=65536) at qpid/amqp_0_10/Connection.cpp:58 #28 0x009767e4 in qpid::broker::SecureConnection::decode (this=0x9d3bd70, buffer=0x9d099c0 "\a\003\004\f", size=10) at qpid/broker/SecureConnection.cpp:42 #29 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9ce5ac0, buff=0x9ce7e58) at qpid/sys/AsynchIOHandler.cpp:135 #30 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9ce7e58) at /usr/include/boost/bind/mem_fn_template.hpp:252 #31 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=..., a1=0x9ce7e58) at /usr/include/boost/bind.hpp:348 #32 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> ( function_obj_ptr=..., a0=..., a1=0x9ce7e58) at /usr/include/boost/bind/bind_template.hpp:44 #33 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9ce7e58) at /usr/include/boost/function/function_template.hpp:136 #34 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x9ce6c3c, a0=..., a1=0x9ce7e58) at /usr/include/boost/function/function_template.hpp:576 #35 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9ce6b98, h=...) at qpid/sys/posix/AsynchIO.cpp:418 #36 0x004be56d in operator() (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/mem_fn_template.hpp:149 #37 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > ( function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 #38 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/bind_template.hpp:32 #39 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:136 #40 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9ce6ba4, a0=...) at /usr/include/boost/function/function_template.hpp:576 #41 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9ce6b9c, type=READABLE) at qpid/sys/DispatchHandle.cpp:278 #42 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123 #43 qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:483 #44 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #45 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #46 0x003a3832 in start_thread () from /lib/libpthread.so.0 #47 0x002f8e0e in clone () from /lib/libc.so.6 Thread 4 (Thread 10380): #0 0x00707410 in __kernel_vsyscall () #1 0x002f9486 in epoll_wait () from /lib/libc.so.6 #2 0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #5 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #6 0x003a3832 in start_thread () from /lib/libpthread.so.0 #7 0x002f8e0e in clone () from /lib/libc.so.6 Thread 3 (Thread 10381): #0 0x00707410 in __kernel_vsyscall () #1 0x003aa6e9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x003a5d9f in _L_lock_885 () from /lib/libpthread.so.0 #3 0x003a5c66 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x00305646 in pthread_mutex_lock () from /lib/libc.so.6 #5 0x00d47b5d in mrg::journal::slock::slock(._4 *) (this=0xb37f2138, m=0x9e02524) at jrnl/slock.hpp:56 #6 0x00d54487 in mrg::journal::jcntl::enqueue_data_record (this=0x9e020c4, data_buff=0xb0a6ebe8, tot_data_len=1115, this_data_len=1115, dtokp=0xb0a6a0b0, transient=false) at jrnl/jcntl.cpp:204 #7 0x00d01a9c in mrg::msgstore::JournalImpl::enqueue_data_record ( this=0x9e020c0, data_buff=0xb0a6ebe8, tot_data_len=1115, this_data_len=1115, dtokp=0xb0a6a0b0, transient=false) at JournalImpl.cpp:355 #8 0x00d1c86c in mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558, queue=0x9e017b0, txn=0xb37f2548, messageId=..., message=..., newId=false) at MessageStoreImpl.cpp:1474 #9 0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434 #10 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930, ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124 #11 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9e017b0, ctxt=0x0, msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #12 0x0094e628 in qpid::broker::Queue::deliver (this=0x9e017b0, msg=...) at qpid/broker/Queue.cpp:164 #13 0x008e457e in qpid::broker::DeliverableMessage::deliverTo ( this=0xb37f3040, queue=...) at qpid/broker/DeliverableMessage.cpp:31 #14 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=..., b=...) at qpid/broker/Exchange.cpp:91 #15 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321 #16 0x009817ff in qpid::broker::SemanticState::route (this=0xb1800b6c, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461 #17 0x00982b9c in qpid::broker::SemanticState::handle (this=0xb1800b6c, msg=...) at qpid/broker/SemanticState.cpp:415 #18 0x009a8306 in qpid::broker::SessionState::handleContent (this=0xb1800a30, frame=..., id=...) at qpid/broker/SessionState.cpp:249 #19 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0xb1800a30, frame=...) at qpid/broker/SessionState.cpp:327 #20 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1800b44, t=...) at qpid/framing/Handler.h:67 #21 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xb1800548, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #22 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1800550, t=...) at qpid/framing/Handler.h:67 #23 0x008d41b0 in operator() (this=0x9d7d108, frame=...) at qpid/framing/Handler.h:42 #24 qpid::broker::Connection::received (this=0x9d7d108, frame=...) at qpid/broker/Connection.cpp:143 #25 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0x9d3c2d0, buffer=0x9d6ccf0 "\a\003\004\f", size=65536) at qpid/amqp_0_10/Connection.cpp:58 #26 0x009767e4 in qpid::broker::SecureConnection::decode (this=0x9ce7d68, buffer=0x9d6ccf0 "\a\003\004\f", size=128) at qpid/broker/SecureConnection.cpp:42 #27 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9d39e08, buff=0x9d39e58) at qpid/sys/AsynchIOHandler.cpp:135 #28 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9d39e58) at /usr/include/boost/bind/mem_fn_template.hpp:252 #29 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=..., a1=0x9d39e58) at /usr/include/boost/bind.hpp:348 #30 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> ( function_obj_ptr=..., a0=..., a1=0x9d39e58) at /usr/include/boost/bind/bind_template.hpp:44 #31 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9d39e58) at /usr/include/boost/function/function_template.hpp:136 #32 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x9d3c1bc, a0=..., a1=0x9d39e58) at /usr/include/boost/function/function_template.hpp:576 #33 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9d3c118, h=...) at qpid/sys/posix/AsynchIO.cpp:418 #34 0x004be56d in operator() (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/mem_fn_template.hpp:149 #35 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > ( function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 #36 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/bind_template.hpp:32 #37 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:136 #38 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9d3c124, a0=...) at /usr/include/boost/function/function_template.hpp:576 #39 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9d3c11c, type=READABLE) at qpid/sys/DispatchHandle.cpp:278 #40 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123 #41 qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:483 #42 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #43 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #44 0x003a3832 in start_thread () from /lib/libpthread.so.0 #45 0x002f8e0e in clone () from /lib/libc.so.6 Thread 2 (Thread 10382): #0 0x00707410 in __kernel_vsyscall () #1 0x002f9486 in epoll_wait () from /lib/libc.so.6 #2 0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:524 #3 0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:479 #4 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #5 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #6 0x003a3832 in start_thread () from /lib/libpthread.so.0 #7 0x002f8e0e in clone () from /lib/libc.so.6 Thread 1 (Thread 10383): #0 0x00707410 in __kernel_vsyscall () #1 0x0024fdf0 in raise () from /lib/libc.so.6 #2 0x00251701 in abort () from /lib/libc.so.6 #3 0x0028828b in __libc_message () from /lib/libc.so.6 #4 0x002912dd in _int_malloc () from /lib/libc.so.6 #5 0x00292e97 in malloc () from /lib/libc.so.6 #6 0x0724fab7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6 #7 0x00d4781a in allocate (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...) at /usr/include/c++/4.1.2/ext/new_allocator.h:88 #8 _M_get_node (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...) at /usr/include/c++/4.1.2/bits/stl_tree.h:358 #9 _M_create_node (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...) at /usr/include/c++/4.1.2/bits/stl_tree.h:367 #10 std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...) at /usr/include/c++/4.1.2/bits/stl_tree.h:819 #11 0x00d479ba in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0x9e020fc, __v=...) at /usr/include/c++/4.1.2/bits/stl_tree.h:927 #12 0x00d45908 in mrg::journal::enq_map::insert_pfid (this=0x9e020f8, rid=1623396, pfid=8, locked=false) at /usr/include/c++/4.1.2/bits/stl_map.h:396 #13 0x00d45ebb in mrg::journal::enq_map::insert_pfid (this=0x9e020f8, rid=1623396, pfid=8) at jrnl/enq_map.cpp:67 #14 0x00d802a8 in mrg::journal::wmgr::enqueue (this=0x9e022f0, data_buff=0xb18e2c80, tot_data_len=1115, this_data_len=1115, dtokp=0x288f, xid_ptr=0x0, xid_len=0, transient=false, external=false) at jrnl/wmgr.cpp:200 #15 0x00d544d5 in mrg::journal::jcntl::enqueue_data_record (this=0x9e020c4, data_buff=0xb18e2c80, tot_data_len=1115, this_data_len=1115, dtokp=0xb0d027b0, transient=false) at jrnl/jcntl.cpp:206 #16 0x00d01a9c in mrg::msgstore::JournalImpl::enqueue_data_record ( this=0x9e020c0, data_buff=0xb18e2c80, tot_data_len=1115, this_data_len=1115, dtokp=0xb0d027b0, transient=false) at JournalImpl.cpp:355 #17 0x00d1c86c in mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558, queue=0x9e017b0, txn=0xb23f0548, messageId=..., message=..., newId=false) at MessageStoreImpl.cpp:1474 #18 0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434 #19 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930, ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124 #20 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9e017b0, ctxt=0x0, msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753 #21 0x0094e628 in qpid::broker::Queue::deliver (this=0x9e017b0, msg=...) at qpid/broker/Queue.cpp:164 #22 0x008e457e in qpid::broker::DeliverableMessage::deliverTo ( this=0xb23f1040, queue=...) at qpid/broker/DeliverableMessage.cpp:31 #23 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=..., b=...) at qpid/broker/Exchange.cpp:91 #24 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321 #25 0x009817ff in qpid::broker::SemanticState::route (this=0xb180205c, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461 #26 0x00982b9c in qpid::broker::SemanticState::handle (this=0xb180205c, msg=...) at qpid/broker/SemanticState.cpp:415 #27 0x009a8306 in qpid::broker::SessionState::handleContent (this=0xb1801f20, frame=..., id=...) at qpid/broker/SessionState.cpp:249 #28 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0xb1801f20, frame=...) at qpid/broker/SessionState.cpp:327 #29 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1802034, t=...) at qpid/framing/Handler.h:67 #30 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xb1801918, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #31 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1801920, t=...) at qpid/framing/Handler.h:67 #32 0x008d41b0 in operator() (this=0xb18014c8, frame=...) at qpid/framing/Handler.h:42 #33 qpid::broker::Connection::received (this=0xb18014c8, frame=...) at qpid/broker/Connection.cpp:143 #34 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0xb1801020, buffer=0x9d9e338 "\a\003\004\f", size=65536) at qpid/amqp_0_10/Connection.cpp:58 #35 0x009767e4 in qpid::broker::SecureConnection::decode (this=0xb1800740, buffer=0x9d9e338 "\a\003\004\f", size=10383) at qpid/broker/SecureConnection.cpp:42 #36 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9d3cad0, buff=0x9d3cb20) at qpid/sys/AsynchIOHandler.cpp:135 #37 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9d3cb20) at /usr/include/boost/bind/mem_fn_template.hpp:252 #38 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=..., a1=0x9d3cb20) at /usr/include/boost/bind.hpp:348 #39 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> ( function_obj_ptr=..., a0=..., a1=0x9d3cb20) at /usr/include/boost/bind/bind_template.hpp:44 #40 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9d3cb20) at /usr/include/boost/function/function_template.hpp:136 #41 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() ( this=0x9d7d84c, a0=..., a1=0x9d3cb20) at /usr/include/boost/function/function_template.hpp:576 #42 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9d7d7a8, h=...) at qpid/sys/posix/AsynchIO.cpp:418 #43 0x004be56d in operator() (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/mem_fn_template.hpp:149 #44 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > ( function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 #45 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...) at /usr/include/boost/bind/bind_template.hpp:32 #46 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:136 #47 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9d7d7b4, a0=...) at /usr/include/boost/function/function_template.hpp:576 #48 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9d7d7ac, type=READABLE) at qpid/sys/DispatchHandle.cpp:278 #49 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123 #50 qpid::sys::Poller::run (this=0x9cf58c8) at qpid/sys/epoll/EpollPoller.cpp:483 #51 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878) at qpid/sys/Dispatcher.cpp:37 #52 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878) at qpid/sys/posix/Thread.cpp:35 #53 0x003a3832 in start_thread () from /lib/libpthread.so.0 #54 0x002f8e0e in clone () from /lib/libc.so.6
Created attachment 410310 [details] The related aborts Please find attached related broker aborts, generally from different broker calls ending in new() or in free()
I can generate a very similar crash easily (on mrg15, qpid-cpp-server-0.7.935473-1.el5) with: while perftest --mode topic --npubs 2 --nsubs 4 --summary --durable true --sub-ack 100; do true; done There is no question of running out of memory in this case. Further, if I run the same test agaisnt the broker without loading the store module (--no-moule-dir), the test continues happily (ran >20 iterations without failure with no store, all 5 attempts with store crashed on first iteration). Core was generated by `/usr/sbin/qpidd --auth no'. Program terminated with signal 6, Aborted. [New process 5049] [New process 5050] [New process 5048] [New process 5047] [New process 5046] [New process 5045] [New process 5044] #0 0x00000038f9c30265 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00000038f9c30265 in raise () from /lib64/libc.so.6 #1 0x00000038f9c31d10 in abort () from /lib64/libc.so.6 #2 0x00000038f9c6a84b in __libc_message () from /lib64/libc.so.6 #3 0x00000038f9c72f39 in _int_malloc () from /lib64/libc.so.6 #4 0x00000038f9c74bee in malloc () from /lib64/libc.so.6 #5 0x00000038fccbd17d in operator new () from /usr/lib64/libstdc++.so.6 #6 0x00002afb3aeb33b4 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert () from /usr/lib64/qpid/daemon/msgstore.so #7 0x00002afb3aeb34c7 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique () from /usr/lib64/qpid/daemon/msgstore.so #8 0x00002afb3aeb1c73 in mrg::journal::enq_map::insert_pfid () from /usr/lib64/qpid/daemon/msgstore.so #9 0x00002afb3aee5a01 in mrg::journal::wmgr::enqueue () from /usr/lib64/qpid/daemon/msgstore.so #10 0x00002afb3aec03be in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so #11 0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so #12 0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so #13 0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so #14 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2 #15 0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2 #16 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2 #17 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2 #18 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2 #19 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2 #20 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2 #21 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2 #22 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2 #23 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2 #24 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2 #25 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2 #26 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2 #27 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2 #28 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #29 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2 #30 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #31 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2 #32 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #33 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #34 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- full thread dump: (gdb) thread apply all bt Thread 7 (process 5044): #0 0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2 #2 0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #3 0x00000039d290a2f2 in qpid::broker::Broker::run () from /usr/lib64/libqpidbroker.so.2 #4 0x0000000000406ae6 in ?? () #5 0x00000000004055af in __cxa_pure_virtual () #6 0x00000038f9c1d994 in __libc_start_main () from /lib64/libc.so.6 #7 0x0000000000405019 in __cxa_pure_virtual () #8 0x00007fffcacd64e8 in ?? () #9 0x0000000000000000 in ?? () Thread 6 (process 5045): #0 0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000039d24074d8 in qpid::sys::Timer::run () from /usr/lib64/libqpidcommon.so.2 #2 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #3 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #4 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 Thread 5 (process 5046): #0 0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000039d24074d8 in qpid::sys::Timer::run () from /usr/lib64/libqpidcommon.so.2 #2 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #3 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #4 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 Thread 4 (process 5047): #0 0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2 #2 0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #3 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #4 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #5 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 Thread 3 (process 5048): #0 0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0 #2 0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002afb3aeb365a in mrg::journal::slock::slock () from /usr/lib64/qpid/daemon/msgstore.so #4 0x00002afb3aec0382 in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so ---Type <return> to continue, or q <return> to quit--- #5 0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so #6 0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so #7 0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so #8 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2 #9 0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2 #10 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2 #11 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2 #12 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2 #13 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2 #14 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2 #15 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2 #16 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2 #17 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2 #18 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2 #19 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2 #20 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2 #21 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2 #22 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #23 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2 #24 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #25 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2 #26 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #27 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #28 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #29 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 Thread 2 (process 5050): #0 0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6 #1 0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2 #2 0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #3 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #4 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #5 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 Thread 1 (process 5049): #0 0x00000038f9c30265 in raise () from /lib64/libc.so.6 #1 0x00000038f9c31d10 in abort () from /lib64/libc.so.6 #2 0x00000038f9c6a84b in __libc_message () from /lib64/libc.so.6 #3 0x00000038f9c72f39 in _int_malloc () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #4 0x00000038f9c74bee in malloc () from /lib64/libc.so.6 #5 0x00000038fccbd17d in operator new () from /usr/lib64/libstdc++.so.6 #6 0x00002afb3aeb33b4 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert () from /usr/lib64/qpid/daemon/msgstore.so #7 0x00002afb3aeb34c7 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique () from /usr/lib64/qpid/daemon/msgstore.so #8 0x00002afb3aeb1c73 in mrg::journal::enq_map::insert_pfid () from /usr/lib64/qpid/daemon/msgstore.so #9 0x00002afb3aee5a01 in mrg::journal::wmgr::enqueue () from /usr/lib64/qpid/daemon/msgstore.so #10 0x00002afb3aec03be in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so #11 0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so #12 0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so #13 0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so #14 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2 #15 0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2 #16 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2 #17 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2 #18 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2 #19 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2 #20 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2 #21 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2 #22 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2 #23 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2 #24 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2 #25 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2 #26 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2 #27 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2 #28 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #29 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2 #30 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2 #31 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2 #32 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2 #33 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2 #34 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0 #35 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6 (gdb)
This is due to buffer overrun when encoding a message for writing to the store. The reason for the overrun is that the message reports the wrong size and the buffer is therefore created too small. This in turn is a result of the size of the header frame sometimes being reported as 0 when the frane is also being concurrently accessed be a subscriber thread. The bug was introduced in http://svn.apache.org/viewvc?view=revision&revision=736810. If you call the non-const getBody() concurrently with encodedSize() you can get a zero size reported even if you don't actually modify the frame in any way.
I believe this is fixed by r944329 which reverts r736810.
The issue has been fixed (no aborts / crashes detected), tested in four extended week runs on RHEL 4.8 / 5.5 i386 / x86_64 on packages: python-qpid-0.7.946106-1.el5 python-saslwrapper-0.1.934605-2.el5 qpid-cpp-client-0.7.946106-2.el5 qpid-cpp-client-devel-0.7.946106-2.el5 qpid-cpp-client-devel-docs-0.7.946106-2.el5 qpid-cpp-client-ssl-0.7.946106-2.el5 qpid-cpp-mrg-debuginfo-0.7.946106-2.el5 qpid-cpp-server-0.7.946106-2.el5 qpid-cpp-server-cluster-0.7.946106-2.el5 qpid-cpp-server-devel-0.7.946106-2.el5 qpid-cpp-server-ssl-0.7.946106-2.el5 qpid-cpp-server-store-0.7.946106-2.el5 qpid-cpp-server-xml-0.7.946106-2.el5 qpid-java-client-0.7.946106-3.el5 qpid-java-common-0.7.946106-3.el5 qpid-tests-0.7.946106-1.el5 qpid-tools-0.7.946106-4.el5 ruby-qpid-0.7.946106-2.el5 ruby-saslwrapper-0.1.934605-2.el5 saslwrapper-0.1.934605-2.el5 saslwrapper-devel-0.1.934605-2.el5 -> VERIFIED