Bug 587505 - msgstore: broker abort in mrg::journal::enq_map::insert_pfid()->std::_Rb_tree()->new()
msgstore: broker abort in mrg::journal::enq_map::insert_pfid()->std::_Rb_tree...
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
All Linux
urgent Severity urgent
: 1.3
: ---
Assigned To: Kim van der Riet
Frantisek Reznicek
:
Depends On:
Blocks: 484048
  Show dependency treegraph
 
Reported: 2010-04-30 01:23 EDT by Frantisek Reznicek
Modified: 2015-11-15 20:12 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
The related aborts (105.40 KB, application/x-tbz)
2010-04-30 02:06 EDT, Frantisek Reznicek
no flags Details

  None (edit)
Description Frantisek Reznicek 2010-04-30 01:23:37 EDT
Description of problem:

There are observed broker aborts during broker vs. perftest performace test. It looks like the broker had no more memory (from the backtrace), but there is no evidence for that.

Could you possibly review the backtrace, please?

Interesting part of backtrace:


  Thread 1 (process 25954):
  ---Type <return> to continue, or q <return> to quit---
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00b138c5 in raise () from /lib/tls/libc.so.6
  #2  0x00b15329 in abort () from /lib/tls/libc.so.6
  #3  0x00b47dca in __libc_message () from /lib/tls/libc.so.6
  #4  0x00b4f423 in _int_malloc () from /lib/tls/libc.so.6
  #5  0x00b510a1 in malloc () from /lib/tls/libc.so.6
  #6  0x006cb427 in operator new () from /usr/lib/libstdc++.so.6
  #7  0x049e4699 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0xb1905e14, __x=0x0, __p=0xb1905e18, __v=@0xb3862e00)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81
  #8  0x049e47e5 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0xb1905e14, __v=@0xb3862e00)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_pair.h:85
  #9  0x049e3fcb in mrg::journal::enq_map::insert_pfid (this=0xb1905e10, rid=1542206, pfid=1, locked=false)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_map.h:360
  #10 0x049e4571 in mrg::journal::enq_map::insert_pfid (this=0x0, rid=25769803776, pfid=Variable "pfid" is not available.
  ) at jrnl/enq_map.cpp:67
  #11 0x04a21cde in mrg::journal::wmgr::enqueue (this=0xb1906008, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0x6562,
      xid_ptr=0x0, xid_len=0, transient=Variable "transient" is not available.
  ) at ./jrnl/data_tok.hpp:146
  #12 0x049f365a in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058,
      dtokp=0xb0e32cf0, transient=Variable "transient" is not available.
  ) at jrnl/jcntl.cpp:206
  #13 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058,
      dtokp=0xb0e32cf0, transient=Variable "transient" is not available.
  ) at JournalImpl.cpp:355
  #14 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb38634b0, messageId=@0xb3863500,
      message=@0xb3863690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462
  #15 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430
  #16 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320)
      at /usr/include/boost/shared_ptr.hpp:252
  #17 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb1970fb8}, suppressPolicyCheck=false)
      at qpid/broker/Queue.cpp:753
  #18 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb3863fb8) at ./boost/detail/atomic_count_gcc.hpp:47
  #19 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb3863fb0, queue=@0x0) at /usr/include/boost/shared_ptr.hpp:252
  #20 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb3863fb0, b={px = 0xb199dae8, pn = {pi_ = 0xb196f018}})
      at /usr/include/boost/shared_ptr.hpp:614

Issue seen on RHEL4.8 i386 only atm.

Version-Release number of selected component (if applicable):
qpid-cpp-client-0.7.935473-1.el4
qpid-cpp-server-devel-0.7.935473-1.el4
qpid-cpp-client-ssl-0.7.935473-1.el4
qpid-cpp-server-store-0.7.935473-1.el4
qpid-cpp-client-devel-0.7.935473-1.el4
qpid-cpp-server-ssl-0.7.935473-1.el4
qpid-cpp-server-xml-0.7.935473-1.el4
qpid-cpp-server-0.7.935473-1.el4
qpid-cpp-mrg-debuginfo-0.7.935473-1.el4
qpid-cpp-client-devel-docs-0.7.935473-1.el4


How reproducible:
very hard (<10%)

Steps to Reproduce:
1. take MRG/Messaging/qpid_test_qpidd-perftest_performance test
The test loops perftest where perftest parameters are swept/variated with different parameter combinations
2. run the test  'TEST_PARAM_RHTS_TEST_MRG_QPID_TEST_QPIDD_PERFTEST_PERFORMANCE_LOOPS=10 make rb'
3. wait for abort/faulire
  
Actual results:
Broker rarely aborts.

Expected results:
Broker should not abort.

Additional info:
  
  
  
  [22:31:48] mrg_qpidd_start: qpidd launched normal bg way (port:40821,log:qpidd.transcript.log,params: --auth no --log-enable info+ --num-jfiles 64 --jf
  ile-size-pgs 64 --wcache-page-size 64 --tpl-num-jfiles 64 --tpl-jfile-size-pgs 64 --tpl-wcache-page-size 16 --mgmt-enable 0 --data-dir /root/MRG/Messag
  ing/qpid_test_qpidd-perftest_performance/rhts_qpidd/20100429_111616/b.4938,lockfile:qpidd.lock)
  [22:32:29] mrg_qpidd_wait_on_settle: qpidd started-up (dur:1sec)
  [22:32:29] .qpidd settled
  [22:32:29] launching perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 40821 --mode topic --qt 1 --durable yes --npubs 2 --nsub
  s 3 --count 200000 --size 1024 --tx 0 --async-commit no
  ................................................................/root/MRG/Messaging/qpid_common/qpid_test_functions_broker.sh: line 51: 25942 Aborted
                (core dumped) ${qpidd_wrapper} ${QPIDD_DIR}/${MRG_CPP_BROKER} ${datadir_sw} --port ${port} ${test_params} >${transcript} 2>${transcript}
  .
  [22:33:39] .perftest went ok  (ecode:0, summary:; dur/to:68.65/740)
  [22:33:39] .qpidd status check
  [22:33:39] mrg_qpidd_status: 0 instance[s] running (pids:,ports:)
  [22:34:16] ..ERROR:qpidd not running fine! (pid[s]:, port[s]:, inst_cnt:0)
  [22:34:16] .qpidd stop
  [22:34:16] mrg_qpidd_stop: No qpidd broker found to stop!
  [22:34:16] .qpidd stopped ok
  [22:34:16] .remove qpidd parent lock (if needed), wait&log...
  
  broker log:
  
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/ssl.so
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/xml.so
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/acl.so
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/msgstore.so
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/replicating_listener.so
  2010-04-29 22:31:48 info Loaded Module: /usr/lib/qpid/daemon/replication_exchange.so
  2010-04-29 22:31:48 info No message store configured, persistence is disabled.
  2010-04-29 22:31:48 info Registered xml exchange
  2010-04-29 22:31:49 notice Journal "TplStore": Created
  2010-04-29 22:31:49 notice Store module initialized; store-dir=/root/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd/20100429_111616/b.49
  38
  2010-04-29 22:31:49 info > Default files per journal: 64
  2010-04-29 22:31:49 info > Default journal file size: 64 (wpgs)
  2010-04-29 22:31:49 info > Default write cache page size: 64 (Kib)
  2010-04-29 22:31:49 info > Default number of write cache pages: 16
  2010-04-29 22:31:49 info > TPL files per journal: 64
  2010-04-29 22:31:49 info > TPL journal file size: 64 (wpgs)
  2010-04-29 22:31:49 info > TPL write cache page size: 16 (Kib)
  2010-04-29 22:31:49 info > TPL number of write cache pages: 32
  2010-04-29 22:31:49 info Registered replication exchange
  2010-04-29 22:31:49 info Most recent persistence id found: 0x0
  2010-04-29 22:31:49 info Management not enabled
  2010-04-29 22:31:49 notice SASL disabled: No Authentication Performed
  2010-04-29 22:31:49 notice Listening on TCP port 40821
  2010-04-29 22:31:49 info SSL plugin not enabled, you must set --ssl-cert-db to enable it.
  2010-04-29 22:31:49 info Policy file not specified. ACL Disabled, no ACL checking being done!
  2010-04-29 22:31:49 notice Broker running
  2010-04-29 22:32:29 info Queue "perftest_pub_start": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:29 info Queue "perftest_pub_done": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:29 info Queue "perftest_sub_ready": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:29 info Queue "perftest_sub_done": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:29 notice Journal "anonymous.2bbed682-2176-4e65-af0e-a8219cbcf011": Created
  2010-04-29 22:32:29 notice Journal "anonymous.f0437860-d222-409d-8846-f14964f3bb11": Created
  2010-04-29 22:32:29 notice Journal "anonymous.9a924f9e-081a-44e0-8e81-e7d449d9c2ef": Created
  2010-04-29 22:32:40 info Queue "anonymous.f0437860-d222-409d-8846-f14964f3bb11": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:40 info Queue "anonymous.2bbed682-2176-4e65-af0e-a8219cbcf011": Policy created: type=reject; maxCount=0; maxSize=104857600
  2010-04-29 22:32:40 info Queue "anonymous.9a924f9e-081a-44e0-8e81-e7d449d9c2ef": Policy created: type=reject; maxCount=0; maxSize=104857600
  
  perftest log:
  
  run 0 - perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 40821 --mode topic --qt 1 --durable yes --npubs 2 --nsubs 3 --count 2
  00000 --size 1024 --tx 0 --async-commit no
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:32:29 info Connecting to tcp:localhost:40821
  2010-04-29 22:33:38 warning Connection closed
  2010-04-29 22:33:38 warning Connection closed
  2010-04-29 22:33:38 warning Connection closed
  2010-04-29 22:33:38 warning Connection closed
  2010-04-29 22:33:38 warning Connection closed
  2010-04-29 22:33:38 warning Connection closed
  PublishThread exception: Connection closed
  SubscribeThread exception: Connection closedSubscribeThread exception:
  Connection closedPublishThread exception: Connection closed
  SubscribeThread exception: Connection closed
  Controller exception: Connection closed
  
  
  core.25942: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qpidd'
  [root@mrg-qe-07 qpid_test_qpidd-perftest_performance]# ll core.*
  -rw-------  1 root root 117088256 Apr 29 22:33 core.25942
  [root@mrg-qe-07 qpid_test_qpidd-perftest_performance]# gdb `which qpidd` core.25942
  GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".
  
  Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
  Program terminated with signal 6, Aborted.
  Reading symbols from /usr/lib/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libqpidbroker.so.2
  Reading symbols from /usr/lib/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libqpidcommon.so.2
  Reading symbols from /usr/lib/libboost_program_options.so.1...done.
  Loaded symbols for /usr/lib/libboost_program_options.so.1
  Reading symbols from /usr/lib/libboost_filesystem.so.1...done.
  Loaded symbols for /usr/lib/libboost_filesystem.so.1
  Reading symbols from /lib/tls/libuuid.so.1...done.
  Loaded symbols for /lib/tls/libuuid.so.1
  Reading symbols from /lib/tls/librt.so.1...done.
  Loaded symbols for /lib/tls/librt.so.1
  Reading symbols from /usr/lib/libsasl2.so.2...done.
  Loaded symbols for /usr/lib/libsasl2.so.2
  Reading symbols from /lib/libdl.so.2...done.
  Loaded symbols for /lib/libdl.so.2
  Reading symbols from /lib/libresolv.so.2...done.
  Loaded symbols for /lib/libresolv.so.2
  Reading symbols from /lib/libcrypt.so.1...done.
  Loaded symbols for /lib/libcrypt.so.1
  Reading symbols from /usr/lib/libstdc++.so.6...done.
  Loaded symbols for /usr/lib/libstdc++.so.6
  Reading symbols from /lib/tls/libm.so.6...done.
  Loaded symbols for /lib/tls/libm.so.6
  Reading symbols from /lib/libgcc_s.so.1...done.
  Loaded symbols for /lib/libgcc_s.so.1
  Reading symbols from /lib/tls/libc.so.6...done.
  Loaded symbols for /lib/tls/libc.so.6
  Reading symbols from /lib/tls/libpthread.so.0...done.
  Loaded symbols for /lib/tls/libpthread.so.0
  Reading symbols from /lib/ld-linux.so.2...done.
  Loaded symbols for /lib/ld-linux.so.2
  Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/ssl.so
  Reading symbols from /usr/lib/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libsslcommon.so.2
  Reading symbols from /usr/lib/libnss3.so...done.
  Loaded symbols for /usr/lib/libnss3.so
  Reading symbols from /usr/lib/libssl3.so...done.
  Loaded symbols for /usr/lib/libssl3.so
  Reading symbols from /usr/lib/libnspr4.so...done.
  Loaded symbols for /usr/lib/libnspr4.so
  Reading symbols from /usr/lib/libnssutil3.so...done.
  Loaded symbols for /usr/lib/libnssutil3.so
  Reading symbols from /usr/lib/libplc4.so...done.
  Loaded symbols for /usr/lib/libplc4.so
  Reading symbols from /usr/lib/libplds4.so...done.
  Loaded symbols for /usr/lib/libplds4.so
  Reading symbols from /usr/lib/libz.so.1...done.
  Loaded symbols for /usr/lib/libz.so.1
  Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/xml.so
  Reading symbols from /usr/lib/libxerces-c.so.28...done.
  Loaded symbols for /usr/lib/libxerces-c.so.28
  Reading symbols from /usr/lib/libxqilla.so.3...done.
  Loaded symbols for /usr/lib/libxqilla.so.3
  Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/acl.so
  Reading symbols from /usr/lib/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/msgstore.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/msgstore.so
  Reading symbols from /usr/lib/tls/i686/libdb_cxx-4.2.so...done.
  Loaded symbols for /usr/lib/tls/i686/libdb_cxx-4.2.so
  Reading symbols from /usr/lib/libaio.so.1...done.
  Loaded symbols for /usr/lib/libaio.so.1
  Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so
  Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  (gdb) info threads
    11 process 25942  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    10 process 25947  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    9 process 25948  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    8 process 25949  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    7 process 25950  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    6 process 25951  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    5 process 25952  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    4 process 25953  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    3 process 25955  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
    2 process 25956  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  * 1 process 25954  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  (gdb) thread apply all bt
  
  Thread 11 (process 25942):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bb862e in epoll_wait () from /lib/tls/libc.so.6
  #2  0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #5  0x0083f2f0 in qpid::broker::Broker::run (this=0x835cb88) at qpid/broker/Broker.cpp:334
  #6  0x08057a15 in QpiddBroker::execute (this=0xbff6fbf7, options=0x834cfb8) at /usr/include/boost/intrusive_ptr.hpp:119
  #7  0x08050aea in main (argc=23, argv=0xbff6fe54) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:301
  #8  0x00b00e93 in __libc_start_main () from /lib/tls/libc.so.6
  #9  0x0804c571 in _start ()
  
  Thread 10 (process 25947):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00d5417c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
  #2  0x00bc4b1e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
  #3  0x005323af in qpid::sys::Timer::run (this=0x835cbb0) at ../include/qpid/sys/posix/Condition.h:69
  #4  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #5  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #6  0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 9 (process 25948):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00d5417c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
  #2  0x00bc4b1e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libc.so.6
  #3  0x005323af in qpid::sys::Timer::run (this=0x8362eb8) at ../include/qpid/sys/posix/Condition.h:69
  #4  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #5  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #6  0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 8 (process 25949):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bc4e1e in __lll_mutex_lock_wait () from /lib/tls/libc.so.6
  #2  0x00b53b9b in _L_mutex_lock_4191 () from /lib/tls/libc.so.6
  #3  0x00000000 in ?? ()
  
  Thread 7 (process 25950):
  ---Type <return> to continue, or q <return> to quit---
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00d565de in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
  #2  0x00d5320b in _L_mutex_lock_35 () from /lib/tls/libpthread.so.0
  #3  0x0846e568 in ?? ()
  #4  0x00c17ff4 in ?? () from /lib/tls/libc.so.6
  #5  0xb1905ddc in ?? ()
  #6  0xb1906008 in ?? ()
  #7  0xb6067028 in ?? ()
  #8  0x00bc4cfe in pthread_mutex_lock () from /lib/tls/libc.so.6
  #9  0x00bc4cfe in pthread_mutex_lock () from /lib/tls/libc.so.6
  #10 0x049f3602 in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb19713d8, tot_data_len=1115, this_data_len=1115,
      dtokp=0xb19535a8, transient=false) at ./jrnl/slock.hpp:56
  #11 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb19713d8, tot_data_len=1115, this_data_len=1115,
      dtokp=0xb19535a8, transient=Variable "transient" is not available.
  ) at JournalImpl.cpp:355
  #12 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb60674b0, messageId=@0xb6067500,
      message=@0xb6067690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462
  #13 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb6067690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430
  #14 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb6067690, queue=@0xb1905320)
      at /usr/include/boost/shared_ptr.hpp:252
  #15 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb0d28098}, suppressPolicyCheck=false)
      at qpid/broker/Queue.cpp:753
  #16 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb6067fb8) at ./boost/detail/atomic_count_gcc.hpp:47
  #17 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb6067fb0, queue=@0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #18 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb6067fb0, b={px = 0xb0d1b820, pn = {pi_ = 0xb0d457f0}})
      at /usr/include/boost/shared_ptr.hpp:614
  #19 0x0096a737 in qpid::broker::TopicExchange::route (this=0x8363a08, msg=@0xb6067fb0, routingKey=@0xb6067d20)
      at ./boost/detail/sp_counted_base_gcc_x86.hpp:66
  #20 0x00927ef4 in qpid::broker::SemanticState::route (this=0x83eaa04, msg={p_ = 0xb0d28098}, strategy=@0xb6067fb0)
      at /usr/include/boost/shared_ptr.hpp:252
  #21 0x00928ab1 in qpid::broker::SemanticState::handle (this=0x83eaa04, msg={p_ = 0xb0d28098}) at ./boost/detail/atomic_count_gcc.hpp:47
  #22 0x00961b66 in qpid::broker::SessionState::handleContent (this=0x83ea8c8, frame=@0xb60687f0, id=@0xfffffffc)
      at ./boost/detail/atomic_count_gcc.hpp:47
  #23 0x00963fdf in qpid::broker::SessionState::handleIn (this=0x83ea8c8, frame=@0xb60687f0) at qpid/broker/SessionState.cpp:327
  #24 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xfffffffc,
      t=@0xfffffffc) at ./qpid/framing/Handler.h:67
  #25 0x004c98dc in qpid::amqp_0_10::SessionHandler::handleIn (this=0x83a9920, f=@0xb60687f0) at qpid/amqp_0_10/SessionHandler.cpp:93
  #26 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, ---Type <return> to continue, or q <return> to quit---
  &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xfffffffc,
      t=@0xfffffffc) at ./qpid/framing/Handler.h:67
  #27 0x0085ed98 in qpid::broker::Connection::received (this=0x83e9ee0, frame=@0xb60687f0) at ./qpid/framing/Handler.h:42
  #28 0x0082bc79 in qpid::amqp_0_10::Connection::decode (this=0x83a9158, buffer=0x83b9ab8 "\a\003\004\f", size=65536)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
  #29 0x009205c4 in qpid::broker::SecureConnection::decode (this=0x8364da8, buffer=0x83b9ab8 "\a\003\004\f", size=0)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
  #30 0x0051fb4c in qpid::sys::AsynchIOHandler::readbuff (this=0x83a6d60, buff=0x83a6818) at qpid/sys/AsynchIOHandler.cpp:135
  #31 0x009b7736 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=
        {obj_ptr = 0x83a61f0, const_obj_ptr = 0x83a61f0, func_ptr = 0x83a61f0, data = "�"}, a0=@0x83a8ef0, a1=0x83a6818)
      at /usr/include/boost/bind/mem_fn_template.hpp:252
  #32 0x004119c7 in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0xd5aff4, a0=@0xfffffffc, a1=0xfffffffc) at /usr/include/boost/function/function_template.hpp:581
  #33 0x004106b2 in qpid::sys::posix::AsynchIO::readable (this=0x83a8ef0, h=@0x83a8ef4) at qpid/sys/posix/AsynchIO.cpp:418
  #34 0x0041249c in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke
      (function_obj_ptr={obj_ptr = 0x83a6268, const_obj_ptr = 0x83a6268, func_ptr = 0x83a6268, data = "h"}, a0=@0xfffffffc)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #35 0x00529950 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xd5aff4,
      a0=@0xfffffffc) at /usr/include/boost/function/function_template.hpp:581
  #36 0x005282c4 in qpid::sys::DispatchHandle::processEvent (this=0x83a8ef4, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:289
  #37 0x00423758 in qpid::sys::Poller::run (this=0x8358e18) at ./qpid/sys/Poller.h:123
  #38 0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #39 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #40 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #41 0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 6 (process 25951):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bb862e in epoll_wait () from /lib/tls/libc.so.6
  #2  0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #5  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #6  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #7  0x00bb7fae in clone () from /lib/tls/libc.so.6
  ---Type <return> to continue, or q <return> to quit---
  
  Thread 5 (process 25952):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bb862e in epoll_wait () from /lib/tls/libc.so.6
  #2  0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #5  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #6  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #7  0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 4 (process 25953):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bb862e in epoll_wait () from /lib/tls/libc.so.6
  #2  0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #5  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #6  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #7  0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 3 (process 25955):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bb862e in epoll_wait () from /lib/tls/libc.so.6
  #2  0x00422140 in qpid::sys::Poller::wait (this=0x8358e18, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004234dc in qpid::sys::Poller::run (this=0x8358e18) at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x0052d51e in qpid::sys::Dispatcher::run (this=0xfffffffc) at /usr/include/boost/shared_ptr.hpp:252
  #5  0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0xfffffffc) at qpid/sys/posix/Thread.cpp:35
  #6  0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #7  0x00bb7fae in clone () from /lib/tls/libc.so.6
  
  Thread 2 (process 25956):
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00bc4e1e in __lll_mutex_lock_wait () from /lib/tls/libc.so.6
  #2  0x00b53b9b in _L_mutex_lock_4191 () from /lib/tls/libc.so.6
  #3  0x00000000 in ?? ()
  
  Thread 1 (process 25954):
  ---Type <return> to continue, or q <return> to quit---
  #0  0x00ad27a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
  #1  0x00b138c5 in raise () from /lib/tls/libc.so.6
  #2  0x00b15329 in abort () from /lib/tls/libc.so.6
  #3  0x00b47dca in __libc_message () from /lib/tls/libc.so.6
  #4  0x00b4f423 in _int_malloc () from /lib/tls/libc.so.6
  #5  0x00b510a1 in malloc () from /lib/tls/libc.so.6
  #6  0x006cb427 in operator new () from /usr/lib/libstdc++.so.6
  #7  0x049e4699 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0xb1905e14, __x=0x0, __p=0xb1905e18, __v=@0xb3862e00)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/ext/new_allocator.h:81
  #8  0x049e47e5 in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0xb1905e14, __v=@0xb3862e00)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_pair.h:85
  #9  0x049e3fcb in mrg::journal::enq_map::insert_pfid (this=0xb1905e10, rid=1542206, pfid=1, locked=false)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_map.h:360
  #10 0x049e4571 in mrg::journal::enq_map::insert_pfid (this=0x0, rid=25769803776, pfid=Variable "pfid" is not available.
  ) at jrnl/enq_map.cpp:67
  #11 0x04a21cde in mrg::journal::wmgr::enqueue (this=0xb1906008, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058, dtokp=0x6562,
      xid_ptr=0x0, xid_len=0, transient=Variable "transient" is not available.
  ) at ./jrnl/data_tok.hpp:146
  #12 0x049f365a in mrg::journal::jcntl::enqueue_data_record (this=0xb1905ddc, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058,
      dtokp=0xb0e32cf0, transient=Variable "transient" is not available.
  ) at jrnl/jcntl.cpp:206
  #13 0x0499480e in mrg::msgstore::JournalImpl::enqueue_data_record (this=0xb1905dd8, data_buff=0xb0e31568, tot_data_len=1058, this_data_len=1058,
      dtokp=0xb0e32cf0, transient=Variable "transient" is not available.
  ) at JournalImpl.cpp:355
  #14 0x049abb95 in mrg::msgstore::MessageStoreImpl::store (this=0x835e280, queue=0xb1905320, txn=0xb38634b0, messageId=@0xb3863500,
      message=@0xb3863690, newId=false) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/stl_vector.h:462
  #15 0x049b6f11 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x835e280, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320) at MessageStoreImpl.cpp:1430
  #16 0x008daa01 in qpid::broker::MessageStoreModule::enqueue (this=0x835dcd0, ctxt=0x0, msg=@0xb3863690, queue=@0xb1905320)
      at /usr/include/boost/shared_ptr.hpp:252
  #17 0x008e6559 in qpid::broker::Queue::enqueue (this=0xb1905320, ctxt=0x0, msg={p_ = 0xb1970fb8}, suppressPolicyCheck=false)
      at qpid/broker/Queue.cpp:753
  #18 0x008f2f9f in qpid::broker::Queue::deliver (this=0xb1905320, msg=@0xb3863fb8) at ./boost/detail/atomic_count_gcc.hpp:47
  #19 0x0087047e in qpid::broker::DeliverableMessage::deliverTo (this=0xb3863fb0, queue=@0x0) at /usr/include/boost/shared_ptr.hpp:252
  #20 0x00890b30 in qpid::broker::Exchange::doRoute (this=0x8363a44, msg=@0xb3863fb0, b={px = 0xb199dae8, pn = {pi_ = 0xb196f018}})
      at /usr/include/boost/shared_ptr.hpp:614
  #21 0x0096a737 in qpid::broker::TopicExchange::route (this=0x8363a08, msg=@0xb3863fb0, routingKey=@0xb3863d20)
      at ./boost/detail/sp_counted_base_gcc_x86.hpp:66
  #22 0x00927ef4 in qpid::broker::SemanticState::route (this=0x83a8ccc, msg={p_ = 0xb1970fb8}, strategy=@0xb3863fb0)
  ---Type <return> to continue, or q <return> to quit---
      at /usr/include/boost/shared_ptr.hpp:252
  #23 0x00928ab1 in qpid::broker::SemanticState::handle (this=0x83a8ccc, msg={p_ = 0xb1970fb8}) at ./boost/detail/atomic_count_gcc.hpp:47
  #24 0x00961b66 in qpid::broker::SessionState::handleContent (this=0x83a8b90, frame=@0xb38647f0, id=@0x0) at ./boost/detail/atomic_count_gcc.hpp:47
  #25 0x00963fdf in qpid::broker::SessionState::handleIn (this=0x83a8b90, frame=@0xb38647f0) at qpid/broker/SessionState.cpp:327
  #26 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x0, t=@0x0)
      at ./qpid/framing/Handler.h:67
  #27 0x004c98dc in qpid::amqp_0_10::SessionHandler::handleIn (this=0x83a66f0, f=@0xb38647f0) at qpid/amqp_0_10/SessionHandler.cpp:93
  #28 0x0096521b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x0, t=@0x0)
      at ./qpid/framing/Handler.h:67
  #29 0x0085ed98 in qpid::broker::Connection::received (this=0x83a6388, frame=@0xb38647f0) at ./qpid/framing/Handler.h:42
  #30 0x0082bc79 in qpid::amqp_0_10::Connection::decode (this=0x83658b0, buffer=0x8385998 "\a\003\004\f", size=65536)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
  #31 0x009205c4 in qpid::broker::SecureConnection::decode (this=0x8364fa0, buffer=0x8385998 "\a\003\004\f", size=25954)
      at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/memory:285
  #32 0x0051fb4c in qpid::sys::AsynchIOHandler::readbuff (this=0x8364e20, buff=0x8364de8) at qpid/sys/AsynchIOHandler.cpp:135
  #33 0x009b7736 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=
        {obj_ptr = 0x8364dd0, const_obj_ptr = 0x8364dd0, func_ptr = 0x8364dd0, data = "�"}, a0=@0x8365078, a1=0x8364de8)
      at /usr/include/boost/bind/mem_fn_template.hpp:252
  #34 0x004119c7 in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0x6, a0=@0x0, a1=0x0) at /usr/include/boost/function/function_template.hpp:581
  #35 0x004106b2 in qpid::sys::posix::AsynchIO::readable (this=0x8365078, h=@0x836507c) at qpid/sys/posix/AsynchIO.cpp:418
  #36 0x0041249c in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke
      (function_obj_ptr={obj_ptr = 0x8364ee8, const_obj_ptr = 0x8364ee8, func_ptr = 0x8364ee8, data = "�"}, a0=@0x0)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #37 0x00529950 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x6, a0=@0x0)
      at /usr/include/boost/function/function_template.hpp:581
  #38 0x005282c4 in qpid::sys::DispatchHandle::processEvent (this=0x836507c, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:289
  #39 0x00423758 in qpid::sys::Poller::run (this=0x8358e18) at ./qpid/sys/Poller.h:123
  #40 0x0052d51e in qpid::sys::Dispatcher::run (this=0x0) at /usr/include/boost/shared_ptr.hpp:252
  #41 0x00415211 in qpid::sys::(anonymous namespace)::runRunnable (p=0x0) at qpid/sys/posix/Thread.cpp:35
  #42 0x00d515cc in start_thread () from /lib/tls/libpthread.so.0
  #43 0x00bb7fae in clone () from /lib/tls/libc.so.6
Comment 1 Frantisek Reznicek 2010-04-30 01:42:05 EDT
Seen also on RHEL 5.5, multiple cases cases furing the perftest performance test -> 
  How reproducible:
  moderate (50%)

Again no info from qpidd logs about not enough of memory.

  GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i386-redhat-linux-gnu".
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>...
  Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done.
  done.
  [New Thread 10382]
  [New Thread 10381]
  [New Thread 10380]
  [New Thread 10379]
  [New Thread 10378]
  [New Thread 10377]
  [New Thread 10376]
  [New Thread 10375]
  [New Thread 10374]
  [New Thread 10369]
  
  warning: .dynamic section for "/usr/lib/openais/libcpg.so.2" is not at the expected address
  
  warning: difference appears to be caused by prelink, adjusting expectations
  
  warning: .dynamic section for "/usr/lib/libxerces-c.so.28" is not at the expected address
  
  warning: difference appears to be caused by prelink, adjusting expectations
  Reading symbols from /usr/lib/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libqpidbroker.so.2
  Reading symbols from /usr/lib/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidcommon.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libqpidcommon.so.2
  Reading symbols from /usr/lib/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib/libboost_program_options.so.1.33.1.debug...done.
  done.
  Loaded symbols for /usr/lib/libboost_program_options.so.2
  Reading symbols from /usr/lib/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib/libboost_filesystem.so.1.33.1.debug...done.
  done.
  Loaded symbols for /usr/lib/libboost_filesystem.so.2
  Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
  Loaded symbols for /lib/libuuid.so.1
  Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
  Loaded symbols for /lib/libdl.so.2
  Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
  Loaded symbols for /lib/librt.so.1
  Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libsasl2.so.2
  Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libstdc++.so.6
  Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
  Loaded symbols for /lib/libm.so.6
  Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
  Loaded symbols for /lib/libgcc_s.so.1
  Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
  Loaded symbols for /lib/libc.so.6
  Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
  Loaded symbols for /lib/ld-linux.so.2
  Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
  Loaded symbols for /lib/libpthread.so.0
  Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
  Loaded symbols for /lib/libresolv.so.2
  Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
  Loaded symbols for /lib/libcrypt.so.1
  Reading symbols from /usr/lib/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/msgstore.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/msgstore.so
  Reading symbols from /usr/lib/libdb_cxx-4.3.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libdb_cxx-4.3.so
  Reading symbols from /usr/lib/libaio.so.1...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libaio.so.1
  Reading symbols from /usr/lib/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replicating_listener.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/replicating_listener.so
  Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/ssl.so
  Reading symbols from /usr/lib/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib/libsslcommon.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libsslcommon.so.2
  Reading symbols from /usr/lib/libnss3.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libnss3.so
  Reading symbols from /usr/lib/libssl3.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libssl3.so
  Reading symbols from /usr/lib/libnspr4.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libnspr4.so
  Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libnssutil3.so
  Reading symbols from /usr/lib/libplc4.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libplc4.so
  Reading symbols from /usr/lib/libplds4.so...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libplds4.so
  Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libz.so.1
  Reading symbols from /usr/lib/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/cluster.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/cluster.so
  Reading symbols from /usr/lib/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib/openais/libcpg.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/openais/libcpg.so.2
  Reading symbols from /usr/lib/libcman.so.2...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libcman.so.2
  Reading symbols from /usr/lib/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib/libqpidclient.so.2.0.0.debug...done.
  done.
  Loaded symbols for /usr/lib/libqpidclient.so.2
  Reading symbols from /usr/lib/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/client/sslconnector.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/client/sslconnector.so
  Reading symbols from /usr/lib/qpid/daemon/watchdog.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/watchdog.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/watchdog.so
  Reading symbols from /usr/lib/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/replication_exchange.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/replication_exchange.so
  Reading symbols from /usr/lib/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/xml.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/xml.so
  Reading symbols from /usr/lib/libxerces-c.so.28...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libxerces-c.so.28
  Reading symbols from /usr/lib/libxqilla.so.3...(no debugging symbols found)...done.
  Loaded symbols for /usr/lib/libxqilla.so.3
  Reading symbols from /usr/lib/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/acl.so.debug...done.
  done.
  Loaded symbols for /usr/lib/qpid/daemon/acl.so
  Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
  Program terminated with signal 6, Aborted.
  #0  0x00707410 in __kernel_vsyscall ()
  (gdb) eax            0x0        0
  ecx            0x288f   10383
  edx            0x6      6
  ebx            0x2881   10369
  esp            0xb23ef4f8       0xb23ef4f8
  ebp            0xb23ef510       0xb23ef510
  esi            0xb23ef5b0       -1304496720
  edi            0x368ff4 3575796
  eip            0x707410 0x707410 <__kernel_vsyscall+16>
  eflags         0x206    [ PF IF ]
  cs             0x73     115
  ss             0x7b     123
  ds             0x7b     123
  es             0x7b     123
  fs             0x0      0
  gs             0x33     51
  st0            0        (raw 0x00000000000000000000)
  st1            0        (raw 0x00000000000000000000)
  st2            0        (raw 0x00000000000000000000)
  st3            0        (raw 0x00000000000000000000)
  st4            0        (raw 0x00000000000000000000)
  st5            0        (raw 0x00000000000000000000)
  st6            0        (raw 0x00000000000000000000)
  st7            0        (raw 0x00000000000000000000)
  fctrl          0x0      0
  fstat          0x0      0
  ftag           0x0      0
  fiseg          0x0      0
  fioff          0x0      0
  foseg          0x0      0
  fooff          0x0      0
  fop            0x0      0
  xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0},
    v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0},
    uint128 = 0x00000000000000000000000000000000}
  mxcsr          0x0      [ ]
  mm0            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm1            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm2            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm3            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm4            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm5            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm6            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  mm7            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0,
      0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
  (gdb) Using memory regions provided by the target.
  There are no memory regions defined.
  (gdb) From        To          Syms Read   Shared Object Library
  0x0083bc30  0x009f5554  Yes (*)     /usr/lib/libqpidbroker.so.2
  0x004aae10  0x005cba24  Yes (*)     /usr/lib/libqpidcommon.so.2
  0x006bf970  0x006de524  Yes (*)     /usr/lib/libboost_program_options.so.2
  0x006a39a0  0x006ac614  Yes (*)     /usr/lib/libboost_filesystem.so.2
  0x077ccee0  0x077ce954  Yes (*)     /lib/libuuid.so.1
  0x00398a50  0x00399a84  Yes (*)     /lib/libdl.so.2
  0x003ce880  0x003d2b04  Yes (*)     /lib/librt.so.1
  0x077da190  0x077ea774  Yes (*)     /usr/lib/libsasl2.so.2
  0x071dac50  0x07256134  Yes (*)     /usr/lib/libstdc++.so.6
  0x00372410  0x0038d594  Yes (*)     /lib/libm.so.6
  0x00b37660  0x00b3ef34  Yes (*)     /lib/libgcc_s.so.1
  0x0023cc80  0x00337c40  Yes (*)     /lib/libc.so.6
  0x002097f0  0x0021eebf  Yes (*)     /lib/ld-linux.so.2
  0x003a2210  0x003ada74  Yes (*)     /lib/libpthread.so.0
  0x0073b0c0  0x00746054  Yes (*)     /lib/libresolv.so.2
  0x074326e0  0x07439314  Yes (*)     /lib/libcrypt.so.1
  0x00cf7380  0x00d9a4c4  Yes (*)     /usr/lib/qpid/daemon/msgstore.so
  0x00bd3240  0x00c9b554  Yes (*)     /usr/lib/libdb_cxx-4.3.so
  0x00110390  0x001106d0  Yes (*)     /usr/lib/libaio.so.1
  0x0011a0f0  0x001204f4  Yes (*)     /usr/lib/qpid/daemon/replicating_listener.so
  0x0012b240  0x00133c64  Yes (*)     /usr/lib/qpid/daemon/ssl.so
  0x001417a0  0x0015abf4  Yes (*)     /usr/lib/libsslcommon.so.2
  0x04d7afc0  0x04e5ea04  Yes (*)     /usr/lib/libnss3.so
  0x04efb480  0x04f21664  Yes (*)     /usr/lib/libssl3.so
  0x04e9a7a0  0x04ebd3f4  Yes (*)     /usr/lib/libnspr4.so
  0x04d4db40  0x04d59284  Yes (*)     /usr/lib/libnssutil3.so
  0x04d64df0  0x04d66704  Yes (*)     /usr/lib/libplc4.so
  0x00204a30  0x002059a4  Yes (*)     /usr/lib/libplds4.so
  0x003b96a0  0x003c48d4  Yes (*)     /usr/lib/libz.so.1
  0x00a91000  0x00b08534  Yes (*)     /usr/lib/qpid/daemon/cluster.so
  0x00161e80  0x00164024  Yes (*)     /usr/lib/openais/libcpg.so.2
  0x00166d30  0x00169894  Yes (*)     /usr/lib/libcman.so.2
  0x00e0e9b0  0x00ee5cc4  Yes (*)     /usr/lib/libqpidclient.so.2
  0x001724f0  0x001803f4  Yes (*)     /usr/lib/qpid/client/sslconnector.so
  0x00188f20  0x0018cb54  Yes (*)     /usr/lib/qpid/daemon/watchdog.so
  0x00193250  0x001985f4  Yes (*)     /usr/lib/qpid/daemon/replication_exchange.so
  0x007a6a50  0x007b1d44  Yes (*)     /usr/lib/qpid/daemon/xml.so
  0x010510a0  0x0120e534  Yes (*)     /usr/lib/libxerces-c.so.28
  0x078fec00  0x07ab41d4  Yes (*)     /usr/lib/libxqilla.so.3
  0x001a58b0  0x001c7ae4  Yes (*)     /usr/lib/qpid/daemon/acl.so
  (*): Shared library is missing debugging information.
  (gdb)   11 Thread 10369  0x00707410 in __kernel_vsyscall ()
    10 Thread 10374  0x00707410 in __kernel_vsyscall ()
    9 Thread 10375  0x00707410 in __kernel_vsyscall ()
    8 Thread 10376  0x00707410 in __kernel_vsyscall ()
    7 Thread 10377  0x00707410 in __kernel_vsyscall ()
    6 Thread 10378  0x00707410 in __kernel_vsyscall ()
    5 Thread 10379  0x0032eea4 in _dl_addr () from /lib/libc.so.6
    4 Thread 10380  0x00707410 in __kernel_vsyscall ()
    3 Thread 10381  0x00707410 in __kernel_vsyscall ()
    2 Thread 10382  0x00707410 in __kernel_vsyscall ()
  * 1 Thread 10383  0x00707410 in __kernel_vsyscall ()
  (gdb)
  Thread 11 (Thread 10369):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x002f9486 in epoll_wait () from /lib/libc.so.6
  #2  0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #5  0x008ca692 in qpid::broker::Broker::run (this=0x9cf7e80)
      at qpid/broker/Broker.cpp:334
  #6  0x0804debe in QpiddBroker::execute (this=0xbfef2b75, options=0x9cf3608)
      at posix/QpiddBroker.cpp:176
  #7  0x0804c821 in main (argc=23, argv=0xbfef2c24) at qpidd.cpp:80
  
  Thread 10 (Thread 10374):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x003a7ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib/libpthread.so.0
  #2  0x00305524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
  #3  0x005c6eb4 in qpid::sys::Timer::run (this=0x9cf7ea8)
      at ../include/qpid/sys/posix/Condition.h:69
  #4  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0x9cf7ea8)
      at qpid/sys/posix/Thread.cpp:35
  #5  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #6  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 9 (Thread 10375):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x003a7ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
    from /lib/libpthread.so.0
  #2  0x00305524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
  #3  0x005c6eb4 in qpid::sys::Timer::run (this=0x9ce4d78)
      at ../include/qpid/sys/posix/Condition.h:69
  #4  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0x9ce4d78)
      at qpid/sys/posix/Thread.cpp:35
  #5  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #6  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 8 (Thread 10376):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x003aa6e9 in __lll_lock_wait () from /lib/libpthread.so.0
  #2  0x003a5d9f in _L_lock_885 () from /lib/libpthread.so.0
  #3  0x003a5c66 in pthread_mutex_lock () from /lib/libpthread.so.0
  #4  0x00305646 in pthread_mutex_lock () from /lib/libc.so.6
  #5  0x00d47b5d in mrg::journal::slock::slock(._4 *) (this=0xb69f8278,
      m=0x9e02524) at jrnl/slock.hpp:56
  #6  0x00d540e7 in mrg::journal::jcntl::dequeue_data_record (this=0x9e020c4,
      dtokp=0xb0c3e6f8, txn_coml_commit=false) at jrnl/jcntl.cpp:304
  #7  0x00d00fc1 in mrg::msgstore::JournalImpl::dequeue_data_record (
      this=0x9e020c0, dtokp=0xb0c3e6f8, txn_coml_commit=false)
      at JournalImpl.cpp:416
  #8  0x00d1bc75 in mrg::msgstore::MessageStoreImpl::async_dequeue (
      this=0x9cf8558, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1545
  #9  0x00d1c39b in mrg::msgstore::MessageStoreImpl::dequeue (this=0x9cf8558,
      ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1520
  #10 0x0093a0cd in qpid::broker::MessageStoreModule::dequeue (this=0x9ce3930,
      ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:131
  #11 0x00947c6b in qpid::broker::Queue::dequeue (this=0x9e017b0, ctxt=0x0,
      msg=...) at qpid/broker/Queue.cpp:784
  #12 0x009786e5 in qpid::broker::SemanticState::ConsumerImpl::deliver (
      this=0xb18dcc60, msg=...) at qpid/broker/SemanticState.cpp:323
  #13 0x00951698 in qpid::broker::Queue::dispatch (this=0x9e017b0, c=...)
      at qpid/broker/Queue.cpp:399
  #14 0x00978c4c in qpid::broker::SemanticState::ConsumerImpl::doOutput (
      this=0xb18dcc60) at qpid/broker/SemanticState.cpp:685
  #15 0x005b8c08 in qpid::sys::AggregateOutput::doOutput (this=0xb1844968)
      at qpid/sys/AggregateOutput.cpp:59
  #16 0x008d612c in qpid::broker::Connection::doOutput (this=0xb1844930)
      at qpid/broker/Connection.cpp:282
  #17 0x008b1b9e in qpid::amqp_0_10::Connection::canEncode (this=0xb1803af8)
      at qpid/amqp_0_10/Connection.cpp:67
  #18 0x0097681b in qpid::broker::SecureConnection::canEncode (this=0xb1803800)
      at qpid/broker/SecureConnection.cpp:58
  #19 0x005bbd19 in qpid::sys::AsynchIOHandler::idle (this=0xb1803740)
      at qpid/sys/AsynchIOHandler.cpp:210
  #20 0x009f388d in operator() (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #21 operator()<boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list1<qpid::sys::AsynchIO&> > (function_obj_ptr=...,
      a0=...) at /usr/include/boost/bind.hpp:287
  #22 operator()<qpid::sys::AsynchIO> (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/bind_template.hpp:32
  #23 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list2<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1> > >, void, qpid::sys::AsynchIO&>::invoke (function_obj_ptr=..., a0=...)
      at /usr/include/boost/function/function_template.hpp:136
  #24 0x004bf737 in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0xb1803978, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #25 0x004bcb56 in qpid::sys::posix::AsynchIO::writeable (this=0xb1803898,
      h=...) at qpid/sys/posix/AsynchIO.cpp:536
  #26 0x004be56d in operator() (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #27 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
      function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
  #28 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/bind_template.hpp:32
  #29 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...)
      at /usr/include/boost/function/function_template.hpp:136
  #30 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xb18038b0, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #31 0x005bfc99 in qpid::sys::DispatchHandle::processEvent (this=0xb180389c,
      type=WRITABLE) at qpid/sys/DispatchHandle.cpp:285
  #32 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123
  #33 qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:483
  #34 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #35 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #36 0x003a3832 in start_thread () from /lib/libpthread.so.0
  #37 0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 7 (Thread 10377):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x002f9486 in epoll_wait () from /lib/libc.so.6
  #2  0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #5  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #6  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #7  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 6 (Thread 10378):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x002f9486 in epoll_wait () from /lib/libc.so.6
  #2  0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #5  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #6  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #7  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 5 (Thread 10379):
  #0  0x0032eea4 in _dl_addr () from /lib/libc.so.6
  #1  0x0030c4f5 in backtrace_symbols_fd () from /lib/libc.so.6
  #2  0x002882e7 in __libc_message () from /lib/libc.so.6
  #3  0x002905a5 in _int_free () from /lib/libc.so.6
  #4  0x002909e9 in free () from /lib/libc.so.6
  #5  0x0724e581 in operator delete(void*) () from /usr/lib/libstdc++.so.6
  #6  0x00d1c892 in deallocate (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548,
      messageId=..., message=..., newId=false)
      at /usr/include/c++/4.1.2/ext/new_allocator.h:94
  #7  _M_deallocate (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548,
      messageId=..., message=..., newId=false)
      at /usr/include/c++/4.1.2/bits/stl_vector.h:133
  #8  ~_Vector_base (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548,
      messageId=..., message=..., newId=false)
      at /usr/include/c++/4.1.2/bits/stl_vector.h:119
  #9  ~vector (this=0x9cf8558, queue=0x9dbe590, txn=0xb4bf4548, messageId=...,
      message=..., newId=false) at /usr/include/c++/4.1.2/bits/stl_vector.h:272
  #10 mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558, queue=0x9dbe590,
      txn=0xb4bf4548, messageId=..., message=..., newId=false)
      at MessageStoreImpl.cpp:1493
  #11 0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558,
      ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
  #12 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930,
      ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124
  #13 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9dbe590, ctxt=0x0,
      msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753
  #14 0x0094e628 in qpid::broker::Queue::deliver (this=0x9dbe590, msg=...)
      at qpid/broker/Queue.cpp:164
  #15 0x008e457e in qpid::broker::DeliverableMessage::deliverTo (
      this=0xb4bf5040, queue=...) at qpid/broker/DeliverableMessage.cpp:31
  #16 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=...,
      b=...) at qpid/broker/Exchange.cpp:91
  #17 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=...,
      routingKey=...) at qpid/broker/TopicExchange.cpp:321
  #18 0x009817ff in qpid::broker::SemanticState::route (this=0x9d3bef4, msg=...,
      strategy=...) at qpid/broker/SemanticState.cpp:461
  #19 0x00982b9c in qpid::broker::SemanticState::handle (this=0x9d3bef4, msg=...)
      at qpid/broker/SemanticState.cpp:415
  #20 0x009a8306 in qpid::broker::SessionState::handleContent (this=0x9d3bdb8,
      frame=..., id=...) at qpid/broker/SessionState.cpp:249
  #21 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0x9d3bdb8,
      frame=...) at qpid/broker/SessionState.cpp:327
  #22 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0x9d3becc, t=...) at qpid/framing/Handler.h:67
  #23 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x9ce7738,
      f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
  #24 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0x9ce7740, t=...) at qpid/framing/Handler.h:67
  #25 0x008d41b0 in operator() (this=0x9ce7378, frame=...)
      at qpid/framing/Handler.h:42
  #26 qpid::broker::Connection::received (this=0x9ce7378, frame=...)
      at qpid/broker/Connection.cpp:143
  #27 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0x9ce8190,
      buffer=0x9d099c0 "\a\003\004\f", size=65536)
      at qpid/amqp_0_10/Connection.cpp:58
  #28 0x009767e4 in qpid::broker::SecureConnection::decode (this=0x9d3bd70,
      buffer=0x9d099c0 "\a\003\004\f", size=10)
      at qpid/broker/SecureConnection.cpp:42
  #29 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9ce5ac0,
      buff=0x9ce7e58) at qpid/sys/AsynchIOHandler.cpp:135
  #30 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9ce7e58)
      at /usr/include/boost/bind/mem_fn_template.hpp:252
  #31 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=...,
      a1=0x9ce7e58) at /usr/include/boost/bind.hpp:348
  #32 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> (
      function_obj_ptr=..., a0=..., a1=0x9ce7e58)
      at /usr/include/boost/bind/bind_template.hpp:44
  #33 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9ce7e58)
      at /usr/include/boost/function/function_template.hpp:136
  #34 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0x9ce6c3c, a0=..., a1=0x9ce7e58)
      at /usr/include/boost/function/function_template.hpp:576
  #35 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9ce6b98, h=...)
      at qpid/sys/posix/AsynchIO.cpp:418
  #36 0x004be56d in operator() (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #37 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
      function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
  #38 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/bind_template.hpp:32
  #39 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...)
      at /usr/include/boost/function/function_template.hpp:136
  #40 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9ce6ba4, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #41 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9ce6b9c,
      type=READABLE) at qpid/sys/DispatchHandle.cpp:278
  #42 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123
  #43 qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:483
  #44 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #45 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #46 0x003a3832 in start_thread () from /lib/libpthread.so.0
  #47 0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 4 (Thread 10380):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x002f9486 in epoll_wait () from /lib/libc.so.6
  #2  0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #5  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #6  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #7  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 3 (Thread 10381):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x003aa6e9 in __lll_lock_wait () from /lib/libpthread.so.0
  #2  0x003a5d9f in _L_lock_885 () from /lib/libpthread.so.0
  #3  0x003a5c66 in pthread_mutex_lock () from /lib/libpthread.so.0
  #4  0x00305646 in pthread_mutex_lock () from /lib/libc.so.6
  #5  0x00d47b5d in mrg::journal::slock::slock(._4 *) (this=0xb37f2138,
      m=0x9e02524) at jrnl/slock.hpp:56
  #6  0x00d54487 in mrg::journal::jcntl::enqueue_data_record (this=0x9e020c4,
      data_buff=0xb0a6ebe8, tot_data_len=1115, this_data_len=1115,
      dtokp=0xb0a6a0b0, transient=false) at jrnl/jcntl.cpp:204
  #7  0x00d01a9c in mrg::msgstore::JournalImpl::enqueue_data_record (
      this=0x9e020c0, data_buff=0xb0a6ebe8, tot_data_len=1115,
      this_data_len=1115, dtokp=0xb0a6a0b0, transient=false)
      at JournalImpl.cpp:355
  #8  0x00d1c86c in mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558,
      queue=0x9e017b0, txn=0xb37f2548, messageId=..., message=..., newId=false)
      at MessageStoreImpl.cpp:1474
  #9  0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558,
      ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
  #10 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930,
      ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124
  #11 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9e017b0, ctxt=0x0,
      msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753
  #12 0x0094e628 in qpid::broker::Queue::deliver (this=0x9e017b0, msg=...)
      at qpid/broker/Queue.cpp:164
  #13 0x008e457e in qpid::broker::DeliverableMessage::deliverTo (
      this=0xb37f3040, queue=...) at qpid/broker/DeliverableMessage.cpp:31
  #14 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=...,
      b=...) at qpid/broker/Exchange.cpp:91
  #15 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=...,
      routingKey=...) at qpid/broker/TopicExchange.cpp:321
  #16 0x009817ff in qpid::broker::SemanticState::route (this=0xb1800b6c,
      msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461
  #17 0x00982b9c in qpid::broker::SemanticState::handle (this=0xb1800b6c,
      msg=...) at qpid/broker/SemanticState.cpp:415
  #18 0x009a8306 in qpid::broker::SessionState::handleContent (this=0xb1800a30,
      frame=..., id=...) at qpid/broker/SessionState.cpp:249
  #19 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0xb1800a30,
      frame=...) at qpid/broker/SessionState.cpp:327
  #20 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1800b44, t=...) at qpid/framing/Handler.h:67
  #21 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xb1800548,
      f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
  #22 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1800550, t=...) at qpid/framing/Handler.h:67
  #23 0x008d41b0 in operator() (this=0x9d7d108, frame=...)
      at qpid/framing/Handler.h:42
  #24 qpid::broker::Connection::received (this=0x9d7d108, frame=...)
      at qpid/broker/Connection.cpp:143
  #25 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0x9d3c2d0,
      buffer=0x9d6ccf0 "\a\003\004\f", size=65536)
      at qpid/amqp_0_10/Connection.cpp:58
  #26 0x009767e4 in qpid::broker::SecureConnection::decode (this=0x9ce7d68,
      buffer=0x9d6ccf0 "\a\003\004\f", size=128)
      at qpid/broker/SecureConnection.cpp:42
  #27 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9d39e08,
      buff=0x9d39e58) at qpid/sys/AsynchIOHandler.cpp:135
  #28 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9d39e58)
      at /usr/include/boost/bind/mem_fn_template.hpp:252
  #29 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=...,
      a1=0x9d39e58) at /usr/include/boost/bind.hpp:348
  #30 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> (
      function_obj_ptr=..., a0=..., a1=0x9d39e58)
      at /usr/include/boost/bind/bind_template.hpp:44
  #31 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9d39e58)
      at /usr/include/boost/function/function_template.hpp:136
  #32 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0x9d3c1bc, a0=..., a1=0x9d39e58)
      at /usr/include/boost/function/function_template.hpp:576
  #33 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9d3c118, h=...)
      at qpid/sys/posix/AsynchIO.cpp:418
  #34 0x004be56d in operator() (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #35 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
      function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
  #36 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/bind_template.hpp:32
  #37 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...)
      at /usr/include/boost/function/function_template.hpp:136
  #38 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9d3c124, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #39 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9d3c11c,
      type=READABLE) at qpid/sys/DispatchHandle.cpp:278
  #40 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123
  #41 qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:483
  #42 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #43 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #44 0x003a3832 in start_thread () from /lib/libpthread.so.0
  #45 0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 2 (Thread 10382):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x002f9486 in epoll_wait () from /lib/libc.so.6
  #2  0x004ccafa in qpid::sys::Poller::wait (this=0x9cf58c8, timeout=...)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #3  0x004cd666 in qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:479
  #4  0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #5  0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #6  0x003a3832 in start_thread () from /lib/libpthread.so.0
  #7  0x002f8e0e in clone () from /lib/libc.so.6
  
  Thread 1 (Thread 10383):
  #0  0x00707410 in __kernel_vsyscall ()
  #1  0x0024fdf0 in raise () from /lib/libc.so.6
  #2  0x00251701 in abort () from /lib/libc.so.6
  #3  0x0028828b in __libc_message () from /lib/libc.so.6
  #4  0x002912dd in _int_malloc () from /lib/libc.so.6
  #5  0x00292e97 in malloc () from /lib/libc.so.6
  #6  0x0724fab7 in operator new(unsigned int) () from /usr/lib/libstdc++.so.6
  #7  0x00d4781a in allocate (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...)
      at /usr/include/c++/4.1.2/ext/new_allocator.h:88
  #8  _M_get_node (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...)
      at /usr/include/c++/4.1.2/bits/stl_tree.h:358
  #9  _M_create_node (this=0x9e020fc, __x=0x0, __p=0xb0a16280, __v=...)
      at /usr/include/c++/4.1.2/bits/stl_tree.h:367
  #10 std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert (this=0x9e020fc, __x=0x0, __p=0xb0a16280,
      __v=...) at /usr/include/c++/4.1.2/bits/stl_tree.h:819
  #11 0x00d479ba in std::_Rb_tree<unsigned long long, std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long long>, std::allocator<std::pair<unsigned long long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique (this=0x9e020fc, __v=...)
      at /usr/include/c++/4.1.2/bits/stl_tree.h:927
  #12 0x00d45908 in mrg::journal::enq_map::insert_pfid (this=0x9e020f8,
      rid=1623396, pfid=8, locked=false)
      at /usr/include/c++/4.1.2/bits/stl_map.h:396
  #13 0x00d45ebb in mrg::journal::enq_map::insert_pfid (this=0x9e020f8,
      rid=1623396, pfid=8) at jrnl/enq_map.cpp:67
  #14 0x00d802a8 in mrg::journal::wmgr::enqueue (this=0x9e022f0,
      data_buff=0xb18e2c80, tot_data_len=1115, this_data_len=1115, dtokp=0x288f,
      xid_ptr=0x0, xid_len=0, transient=false, external=false)
      at jrnl/wmgr.cpp:200
  #15 0x00d544d5 in mrg::journal::jcntl::enqueue_data_record (this=0x9e020c4,
      data_buff=0xb18e2c80, tot_data_len=1115, this_data_len=1115,
      dtokp=0xb0d027b0, transient=false) at jrnl/jcntl.cpp:206
  #16 0x00d01a9c in mrg::msgstore::JournalImpl::enqueue_data_record (
      this=0x9e020c0, data_buff=0xb18e2c80, tot_data_len=1115,
      this_data_len=1115, dtokp=0xb0d027b0, transient=false)
      at JournalImpl.cpp:355
  #17 0x00d1c86c in mrg::msgstore::MessageStoreImpl::store (this=0x9cf8558,
      queue=0x9e017b0, txn=0xb23f0548, messageId=..., message=..., newId=false)
      at MessageStoreImpl.cpp:1474
  #18 0x00d1d746 in mrg::msgstore::MessageStoreImpl::enqueue (this=0x9cf8558,
      ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
  #19 0x0093a1dd in qpid::broker::MessageStoreModule::enqueue (this=0x9ce3930,
      ctxt=0x0, msg=..., queue=...) at qpid/broker/MessageStoreModule.cpp:124
  #20 0x00948cd1 in qpid::broker::Queue::enqueue (this=0x9e017b0, ctxt=0x0,
      msg=..., suppressPolicyCheck=false) at qpid/broker/Queue.cpp:753
  #21 0x0094e628 in qpid::broker::Queue::deliver (this=0x9e017b0, msg=...)
      at qpid/broker/Queue.cpp:164
  #22 0x008e457e in qpid::broker::DeliverableMessage::deliverTo (
      this=0xb23f1040, queue=...) at qpid/broker/DeliverableMessage.cpp:31
  #23 0x008ff15d in qpid::broker::Exchange::doRoute (this=0x9ce5794, msg=...,
      b=...) at qpid/broker/Exchange.cpp:91
  #24 0x009afa9a in qpid::broker::TopicExchange::route (this=0x9ce5794, msg=...,
      routingKey=...) at qpid/broker/TopicExchange.cpp:321
  #25 0x009817ff in qpid::broker::SemanticState::route (this=0xb180205c,
      msg=..., strategy=...) at qpid/broker/SemanticState.cpp:461
  #26 0x00982b9c in qpid::broker::SemanticState::handle (this=0xb180205c,
      msg=...) at qpid/broker/SemanticState.cpp:415
  #27 0x009a8306 in qpid::broker::SessionState::handleContent (this=0xb1801f20,
      frame=..., id=...) at qpid/broker/SessionState.cpp:249
  #28 0x009a89b3 in qpid::broker::SessionState::handleIn (this=0xb1801f20,
      frame=...) at qpid/broker/SessionState.cpp:327
  #29 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1802034, t=...) at qpid/framing/Handler.h:67
  #30 0x0056ef85 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xb1801918,
      f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
  #31 0x009aaa6b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle (this=0xb1801920, t=...) at qpid/framing/Handler.h:67
  #32 0x008d41b0 in operator() (this=0xb18014c8, frame=...)
      at qpid/framing/Handler.h:42
  #33 qpid::broker::Connection::received (this=0xb18014c8, frame=...)
      at qpid/broker/Connection.cpp:143
  #34 0x008b11ff in qpid::amqp_0_10::Connection::decode (this=0xb1801020,
      buffer=0x9d9e338 "\a\003\004\f", size=65536)
      at qpid/amqp_0_10/Connection.cpp:58
  #35 0x009767e4 in qpid::broker::SecureConnection::decode (this=0xb1800740,
      buffer=0x9d9e338 "\a\003\004\f", size=10383)
      at qpid/broker/SecureConnection.cpp:42
  #36 0x005bc638 in qpid::sys::AsynchIOHandler::readbuff (this=0x9d3cad0,
      buff=0x9d3cb20) at qpid/sys/AsynchIOHandler.cpp:135
  #37 0x009f3854 in operator() (function_obj_ptr=..., a0=..., a1=0x9d3cb20)
      at /usr/include/boost/bind/mem_fn_template.hpp:252
  #38 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (function_obj_ptr=..., a0=...,
      a1=0x9d3cb20) at /usr/include/boost/bind.hpp:348
  #39 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> (
      function_obj_ptr=..., a0=..., a1=0x9d3cb20)
      at /usr/include/boost/bind/bind_template.hpp:44
  #40 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (function_obj_ptr=..., a0=..., a1=0x9d3cb20)
      at /usr/include/boost/function/function_template.hpp:136
  #41 0x004bfc3e in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0x9d7d84c, a0=..., a1=0x9d3cb20)
      at /usr/include/boost/function/function_template.hpp:576
  #42 0x004bd440 in qpid::sys::posix::AsynchIO::readable (this=0x9d7d7a8, h=...)
      at qpid/sys/posix/AsynchIO.cpp:418
  #43 0x004be56d in operator() (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/mem_fn_template.hpp:149
  #44 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
      function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287
  #45 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...)
      at /usr/include/boost/bind/bind_template.hpp:32
  #46 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=..., a0=...)
      at /usr/include/boost/function/function_template.hpp:136
  #47 0x005c3ff4 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x9d7d7b4, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #48 0x005bfbd8 in qpid::sys::DispatchHandle::processEvent (this=0x9d7d7ac,
      type=READABLE) at qpid/sys/DispatchHandle.cpp:278
  #49 0x004cd684 in process (this=0x9cf58c8) at qpid/sys/Poller.h:123
  #50 qpid::sys::Poller::run (this=0x9cf58c8)
      at qpid/sys/epoll/EpollPoller.cpp:483
  #51 0x005c4204 in qpid::sys::Dispatcher::run (this=0xbfef2878)
      at qpid/sys/Dispatcher.cpp:37
  #52 0x004c2311 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfef2878)
      at qpid/sys/posix/Thread.cpp:35
  #53 0x003a3832 in start_thread () from /lib/libpthread.so.0
  #54 0x002f8e0e in clone () from /lib/libc.so.6
Comment 2 Frantisek Reznicek 2010-04-30 02:06:41 EDT
Created attachment 410310 [details]
The related aborts

Please find attached related broker aborts, generally from different broker calls ending in new() or in free()
Comment 3 Gordon Sim 2010-05-13 04:21:40 EDT
I can generate a very similar crash easily (on mrg15, qpid-cpp-server-0.7.935473-1.el5) with:

while perftest --mode topic --npubs 2 --nsubs 4 --summary --durable true --sub-ack 100; do true; done

There is no question of running out of memory in this case. Further, if I run the same test agaisnt the broker without loading the store module (--no-moule-dir), the test continues happily (ran >20 iterations without failure with no store, all 5 attempts with store crashed on first iteration).


Core was generated by `/usr/sbin/qpidd --auth no'.
Program terminated with signal 6, Aborted.
[New process 5049]
[New process 5050]
[New process 5048]
[New process 5047]
[New process 5046]
[New process 5045]
[New process 5044]
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038f9c6a84b in __libc_message () from /lib64/libc.so.6
#3  0x00000038f9c72f39 in _int_malloc () from /lib64/libc.so.6
#4  0x00000038f9c74bee in malloc () from /lib64/libc.so.6
#5  0x00000038fccbd17d in operator new () from /usr/lib64/libstdc++.so.6
#6  0x00002afb3aeb33b4 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert () from /usr/lib64/qpid/daemon/msgstore.so
#7  0x00002afb3aeb34c7 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique () from /usr/lib64/qpid/daemon/msgstore.so
#8  0x00002afb3aeb1c73 in mrg::journal::enq_map::insert_pfid () from /usr/lib64/qpid/daemon/msgstore.so
#9  0x00002afb3aee5a01 in mrg::journal::wmgr::enqueue () from /usr/lib64/qpid/daemon/msgstore.so
#10 0x00002afb3aec03be in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#11 0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#12 0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so
#13 0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so
#14 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2
#15 0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2
#16 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2
#17 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2
#18 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2
#19 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2
#20 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2
#21 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2
#22 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2
#23 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2
#24 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2
#25 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2
#26 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2
#27 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2
#28 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2
#29 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2
#30 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
   from /usr/lib64/libqpidcommon.so.2
#31 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2
#32 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#33 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#34 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---

full thread dump:

(gdb) thread apply all bt

Thread 7 (process 5044):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2
#2  0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#3  0x00000039d290a2f2 in qpid::broker::Broker::run () from /usr/lib64/libqpidbroker.so.2
#4  0x0000000000406ae6 in ?? ()
#5  0x00000000004055af in __cxa_pure_virtual ()
#6  0x00000038f9c1d994 in __libc_start_main () from /lib64/libc.so.6
#7  0x0000000000405019 in __cxa_pure_virtual ()
#8  0x00007fffcacd64e8 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 6 (process 5045):
#0  0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000039d24074d8 in qpid::sys::Timer::run () from /usr/lib64/libqpidcommon.so.2
#2  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 5 (process 5046):
#0  0x00000038fa40af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000039d24074d8 in qpid::sys::Timer::run () from /usr/lib64/libqpidcommon.so.2
#2  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#3  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 5047):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2
#2  0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#3  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 5048):
#0  0x00000038fa40d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000038fa408c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2  0x00000038fa408afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002afb3aeb365a in mrg::journal::slock::slock () from /usr/lib64/qpid/daemon/msgstore.so
#4  0x00002afb3aec0382 in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
---Type <return> to continue, or q <return> to quit--- 
#5  0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#6  0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so
#7  0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so
#8  0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2
#9  0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2
#10 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2
#11 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2
#12 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2
#13 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2
#14 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2
#15 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2
#16 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2
#17 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2
#18 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2
#19 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2
#20 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2
#21 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2
#22 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2
#23 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2
#24 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
   from /usr/lib64/libqpidcommon.so.2
#25 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2
#26 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#27 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#28 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#29 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 5050):
#0  0x00000038f9cd4018 in epoll_wait () from /lib64/libc.so.6
#1  0x00000039d23295af in qpid::sys::Poller::wait () from /usr/lib64/libqpidcommon.so.2
#2  0x00000039d2329fd2 in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#3  0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#4  0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#5  0x00000038f9cd3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 5049):
#0  0x00000038f9c30265 in raise () from /lib64/libc.so.6
#1  0x00000038f9c31d10 in abort () from /lib64/libc.so.6
#2  0x00000038f9c6a84b in __libc_message () from /lib64/libc.so.6
#3  0x00000038f9c72f39 in _int_malloc () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#4  0x00000038f9c74bee in malloc () from /lib64/libc.so.6
#5  0x00000038fccbd17d in operator new () from /usr/lib64/libstdc++.so.6
#6  0x00002afb3aeb33b4 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::_M_insert () from /usr/lib64/qpid/daemon/msgstore.so
#7  0x00002afb3aeb34c7 in std::_Rb_tree<unsigned long, std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct>, std::_Select1st<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, mrg::journal::enq_map::emap_data_struct> > >::insert_unique () from /usr/lib64/qpid/daemon/msgstore.so
#8  0x00002afb3aeb1c73 in mrg::journal::enq_map::insert_pfid () from /usr/lib64/qpid/daemon/msgstore.so
#9  0x00002afb3aee5a01 in mrg::journal::wmgr::enqueue () from /usr/lib64/qpid/daemon/msgstore.so
#10 0x00002afb3aec03be in mrg::journal::jcntl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#11 0x00002afb3ae75283 in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib64/qpid/daemon/msgstore.so
#12 0x00002afb3ae801a7 in mrg::msgstore::MessageStoreImpl::store () from /usr/lib64/qpid/daemon/msgstore.so
#13 0x00002afb3ae8e01b in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib64/qpid/daemon/msgstore.so
#14 0x00000039d29795cb in qpid::broker::MessageStoreModule::enqueue () from /usr/lib64/libqpidbroker.so.2
#15 0x00000039d298860c in qpid::broker::Queue::enqueue () from /usr/lib64/libqpidbroker.so.2
#16 0x00000039d2989dc6 in qpid::broker::Queue::deliver () from /usr/lib64/libqpidbroker.so.2
#17 0x00000039d292c7e3 in qpid::broker::DeliverableMessage::deliverTo () from /usr/lib64/libqpidbroker.so.2
#18 0x00000039d2944065 in qpid::broker::Exchange::doRoute () from /usr/lib64/libqpidbroker.so.2
#19 0x00000039d29e4a8c in qpid::broker::TopicExchange::route () from /usr/lib64/libqpidbroker.so.2
#20 0x00000039d29babf3 in qpid::broker::SemanticState::route () from /usr/lib64/libqpidbroker.so.2
#21 0x00000039d29bbb4d in qpid::broker::SemanticState::handle () from /usr/lib64/libqpidbroker.so.2
#22 0x00000039d29e090e in qpid::broker::SessionState::handleContent () from /usr/lib64/libqpidbroker.so.2
#23 0x00000039d29e0eb0 in qpid::broker::SessionState::handleIn () from /usr/lib64/libqpidbroker.so.2
#24 0x00000039d23b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn () from /usr/lib64/libqpidcommon.so.2
#25 0x00000039d291dee2 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.2
#26 0x00000039d28fdd84 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.2
#27 0x00000039d23fde11 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.2
#28 0x00000039d231de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.2
#29 0x00000039d231c3b3 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.2
#30 0x00000039d2404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ()
   from /usr/lib64/libqpidcommon.so.2
#31 0x00000039d23ffd2f in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.2
#32 0x00000039d2329fff in qpid::sys::Poller::run () from /usr/lib64/libqpidcommon.so.2
#33 0x00000039d23201ca in ?? () from /usr/lib64/libqpidcommon.so.2
#34 0x00000038fa406617 in start_thread () from /lib64/libpthread.so.0
#35 0x00000038f9cd3c2d in clone () from /lib64/libc.so.6
(gdb)
Comment 4 Gordon Sim 2010-05-13 12:20:50 EDT
This is due to buffer overrun when encoding a message for writing to the store. The reason for the overrun is that the message reports the wrong size and the buffer is therefore created too small. This in turn is a result of the size of the header frame sometimes being reported as 0 when the frane is also being concurrently accessed be a subscriber thread. The bug was introduced in http://svn.apache.org/viewvc?view=revision&revision=736810. If you call the non-const getBody() concurrently with encodedSize() you can get a zero size reported even if you don't actually modify the frame in any way.
Comment 5 Gordon Sim 2010-05-14 12:09:27 EDT
I believe this is fixed by r944329 which reverts r736810.
Comment 6 Frantisek Reznicek 2010-06-14 04:32:25 EDT
The issue has been fixed (no aborts / crashes detected), tested in four extended week runs on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qpid-0.7.946106-1.el5
python-saslwrapper-0.1.934605-2.el5
qpid-cpp-client-0.7.946106-2.el5
qpid-cpp-client-devel-0.7.946106-2.el5
qpid-cpp-client-devel-docs-0.7.946106-2.el5
qpid-cpp-client-ssl-0.7.946106-2.el5
qpid-cpp-mrg-debuginfo-0.7.946106-2.el5
qpid-cpp-server-0.7.946106-2.el5
qpid-cpp-server-cluster-0.7.946106-2.el5
qpid-cpp-server-devel-0.7.946106-2.el5
qpid-cpp-server-ssl-0.7.946106-2.el5
qpid-cpp-server-store-0.7.946106-2.el5
qpid-cpp-server-xml-0.7.946106-2.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-4.el5
ruby-qpid-0.7.946106-2.el5
ruby-saslwrapper-0.1.934605-2.el5
saslwrapper-0.1.934605-2.el5
saslwrapper-devel-0.1.934605-2.el5

-> VERIFIED

Note You need to log in before you can comment on or make changes to this bug.