Description of problem:

There was observed broker crash during broker perftest performance stress in boost's atomic_exchange_and_add() call initialized by ~PersistableMessage():

Thread 1 (Thread 30166):
#0  0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>,
    __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50
#1  weak_release (this=<value optimized out>, __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:157
#2  boost::detail::weak_count::~weak_count (this=<value optimized out>,
    __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/shared_count.hpp:262
#3  0x00002b1cac50661f in ~list (this=0x2aaab0058fb0,
    __in_chrg=<value optimized out>) at /usr/include/boost/weak_ptr.hpp:27
#4  qpid::broker::PersistableMessage::~PersistableMessage (
    this=0x2aaab0058fb0, __in_chrg=<value optimized out>)
    at qpid/broker/PersistableMessage.cpp:34
#5  0x00002b1cac4fbaa6 in qpid::broker::Message::~Message (
    this=0x2aaab0058fb0, __in_chrg=<value optimized out>)
    at qpid/broker/Message.cpp:59
#6  0x00002b1cade2c63c in ~intrusive_ptr (this=0x2aaaac032840,
    __in_chrg=<value optimized out>)
    at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42

This issue was observed on RHEL 5.5 x86_64 OS/arch.


Version-Release number of selected component (if applicable):
qpid-cpp-*0.7.935473

How reproducible:
very hard

Steps to Reproduce:
1. run qpid_test_qpidd-perftest_performance and wait for crash
  launch broker
  loop the perftest client to sweep the parameters
  keep running until broker crashes
  
Actual results:
qpidd broker crashes.

Expected results:
qpidd broker should not crash.

Additional info:

[root@mrg-qe-02 qpid_test_qpidd-perftest_performance_bck]# cat dump_core.30166
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...done.
done.
[New Thread 30180]
[New Thread 30179]
[New Thread 30178]
[New Thread 30177]
[New Thread 30176]
[New Thread 30175]
[New Thread 30174]
[New Thread 30173]
[New Thread 30172]
[New Thread 30171]
Reading symbols from /usr/lib64/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.2
Reading symbols from /usr/lib64/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.2
Reading symbols from /usr/lib64/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_program_options.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_filesystem.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/acl.so
Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/xml.so
Reading symbols from /usr/lib64/libxerces-c.so.28...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxerces-c.so.28
Reading symbols from /usr/lib64/libxqilla.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxqilla.so.3
Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/cluster.so
Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/openais/libcpg.so.2
Reading symbols from /usr/lib64/libcman.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcman.so.2
Reading symbols from /usr/lib64/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.2
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/msgstore.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so
Reading symbols from /usr/lib64/libdb_cxx-4.3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libdb_cxx-4.3.so
Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/ssl.so
Reading symbols from /usr/lib64/qpid/daemon/watchdog.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/watchdog.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/watchdog.so
Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so
Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>,
    __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50
50          );
(gdb) rax            0xffffffff 4294967295
rbx            0x2aaab0080a10   46912586123792
rcx            0x2aaab0000038   46912585596984
rdx            0x2aaab0091970   46912586193264
rsi            0x2aaab008fa60   46912586185312
rdi            0x5858585858585858       6365935209750747224
rbp            0x5858585858585858       0x5858585858585858
rsp            0x7fff8596d048   0x7fff8596d048
r8             0x2aaab0003cb8   46912585612472
r9             0x2aaab00591c0   46912585961920
r10            0x5511bb0        89201584
r11            0x36a1c08740     234641983296
r12            0x2aaab0059040   46912585961536
r13            0x2aaab0058fb0   46912585961392
r14            0x2aaab0058fb0   46912585961392
r15            0x2aaab0059158   46912585961816
rip            0x2b1cac50800d   0x2b1cac50800d <boost::detail::weak_count::~weak_count()+13>
eflags         0x10202  [ IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            0        (raw 0x00000000000000000000)
st3            0        (raw 0x00000000000000000000)
st4            0        (raw 0x00000000000000000000)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            0        (raw 0x00000000000000000000)
fctrl          0x0      0
fstat          0x0      0
ftag           0x0      0
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
(gdb) Using memory regions provided by the target.
There are no memory regions defined.
(gdb) From                To                  Syms Read   Shared Object Library
0x00002b1cac4291a0  0x00002b1cac5ad0b8  Yes (*)     /usr/lib64/libqpidbroker.so.2
0x0000003c5630af10  0x0000003c5640bc38  Yes (*)     /usr/lib64/libqpidcommon.so.2
0x0000003674010aa0  0x000000367402dae8  Yes (*)     /usr/lib64/libboost_program_options.so.2
0x0000003674404810  0x000000367440cff8  Yes (*)     /usr/lib64/libboost_filesystem.so.2
0x00000036b5e01500  0x00000036b5e02918  Yes (*)     /lib64/libuuid.so.1
0x00000036a1800e10  0x00000036a1801a08  Yes (*)     /lib64/libdl.so.2
0x00000036a2402220  0x00000036a2405cc8  Yes (*)     /lib64/librt.so.1
0x00000036b66046e0  0x00000036b6613be8  Yes (*)     /usr/lib64/libsasl2.so.2
0x00000036b3a4f430  0x00000036b3ac3058  Yes (*)     /usr/lib64/libstdc++.so.6
0x00000036a1403e60  0x00000036a1443e38  Yes (*)     /lib64/libm.so.6
0x00000036af201e50  0x00000036af20b018  Yes (*)     /lib64/libgcc_s.so.1
0x00000036a101d780  0x00000036a1109ff8  Yes (*)     /lib64/libc.so.6
0x00000036a0c00a70  0x00000036a0c1671e  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00000036a1c051f0  0x00000036a1c10258  Yes (*)     /lib64/libpthread.so.0
0x00000036a54032a0  0x00000036a540e2d8  Yes (*)     /lib64/libresolv.so.2
0x00000036b1e009f0  0x00000036b1e06918  Yes (*)     /lib64/libcrypt.so.1
0x00002b1cac834bf0  0x00002b1cac853f98  Yes (*)     /usr/lib64/qpid/daemon/acl.so
0x00002b1caca667f0  0x00002b1caca70988  Yes (*)     /usr/lib64/qpid/daemon/xml.so
0x0000003673773070  0x00000036738f4758  Yes (*)     /usr/lib64/libxerces-c.so.28
0x00002b1cacdf9090  0x00002b1cacf84b28  Yes (*)     /usr/lib64/libxqilla.so.3
0x00002b1cad2fdda0  0x00002b1cad369528  Yes (*)     /usr/lib64/qpid/daemon/cluster.so
0x00002b1cad5a73d0  0x00002b1cad5a9338  Yes (*)     /usr/lib64/openais/libcpg.so.2
0x00002b1cad7ab110  0x00002b1cad7adb78  Yes (*)     /usr/lib64/libcman.so.2
0x0000003c5685f120  0x0000003c56922078  Yes (*)     /usr/lib64/libqpidclient.so.2
0x00002b1cad9b8880  0x00002b1cad9c5248  Yes (*)     /usr/lib64/qpid/client/sslconnector.so
0x00002b1cadbd6350  0x00002b1cadbed058  Yes (*)     /usr/lib64/libsslcommon.so.2
0x00000030b8c183b0  0x00000030b8cf6f08  Yes (*)     /usr/lib64/libnss3.so
0x00000030b98085e0  0x00000030b982b638  Yes (*)     /usr/lib64/libssl3.so
0x00000030b800cf30  0x00000030b802b738  Yes (*)     /usr/lib64/libnspr4.so
0x00000030b9008340  0x00000030b9012c38  Yes (*)     /usr/lib64/libnssutil3.so
0x00000030b8401370  0x00000030b8402978  Yes (*)     /usr/lib64/libplc4.so
0x00000030b8800e10  0x00000030b8801c08  Yes (*)     /usr/lib64/libplds4.so
0x00000036a2001fd0  0x00000036a200cac8  Yes (*)     /usr/lib64/libz.so.1
0x00002b1cade2ad10  0x00002b1cadeba5d8  Yes (*)     /usr/lib64/qpid/daemon/msgstore.so
0x00002b1cae1245d0  0x00002b1cae1dd288  Yes (*)     /usr/lib64/libdb_cxx-4.3.so
0x00002b1cae408510  0x00002b1cae4086d1  Yes (*)     /usr/lib64/libaio.so.1
0x00002b1cae612570  0x00002b1cae61a698  Yes (*)     /usr/lib64/qpid/daemon/ssl.so
0x00002b1cae8245e0  0x00002b1cae827c18  Yes (*)     /usr/lib64/qpid/daemon/watchdog.so
0x00002b1caea2f620  0x00002b1caea33a68  Yes (*)     /usr/lib64/qpid/daemon/replication_exchange.so
0x00002b1caec3cb70  0x00002b1caec42708  Yes (*)     /usr/lib64/qpid/daemon/replicating_listener.so
(*): Shared library is missing debugging information.
(gdb)   11 Thread 30171  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  10 Thread 30172  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  9 Thread 30173  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  8 Thread 30174  0x00000036a10c6070 in __write_nocancel ()
   from /lib64/libc.so.6
  7 Thread 30175  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  6 Thread 30176  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  5 Thread 30177  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  4 Thread 30178  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  3 Thread 30179  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  2 Thread 30180  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 30166  0x00002b1cac50800d in atomic_exchange_and_add (
    this=<value optimized out>, __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50
(gdb)
Thread 11 (Thread 30171):
#0  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000003c564074d8 in qpid::sys::Timer::run (this=0x529fcf0)
    at ../include/qpid/sys/posix/Condition.h:69
#2  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0x529fd24) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 10 (Thread 30172):
#0  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000003c564074d8 in qpid::sys::Timer::run (this=0x52a87f0)
    at ../include/qpid/sys/posix/Condition.h:69
#2  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0x52a8824) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 9 (Thread 30173):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 8 (Thread 30174):
#0  0x00000036a10c6070 in __write_nocancel () from /lib64/libc.so.6
#1  0x00000036a106a8d4 in __libc_message () from /lib64/libc.so.6
#2  0x00000036a107230f in _int_free () from /lib64/libc.so.6
#3  0x00000036a107276b in free () from /lib64/libc.so.6
#4  0x00002b1cade3f1d2 in deallocate (this=0x52a0710, queue=0x5376c10, txn=
    0x4346be00, messageId=..., message=..., newId=176)
    at /usr/include/c++/4.1.2/ext/new_allocator.h:94
#5  _M_deallocate (this=0x52a0710, queue=0x5376c10, txn=0x4346be00,
    messageId=..., message=..., newId=176)
    at /usr/include/c++/4.1.2/bits/stl_vector.h:133
#6  ~_Vector_base (this=0x52a0710, queue=0x5376c10, txn=0x4346be00,
    messageId=..., message=..., newId=176)
    at /usr/include/c++/4.1.2/bits/stl_vector.h:119
#7  ~vector (this=0x52a0710, queue=0x5376c10, txn=0x4346be00, messageId=...,
    message=..., newId=176) at /usr/include/c++/4.1.2/bits/stl_vector.h:272
#8  mrg::msgstore::MessageStoreImpl::store (this=0x52a0710, queue=0x5376c10,
    txn=0x4346be00, messageId=..., message=..., newId=176)
    at MessageStoreImpl.cpp:1493
#9  0x00002b1cade4d01b in mrg::msgstore::MessageStoreImpl::enqueue (
    this=0x52a0710, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
#10 0x00002b1cac5035cb in qpid::broker::MessageStoreModule::enqueue (
    this=<value optimized out>, ctxt=0x4346afd0, msg=..., queue=...)
    at qpid/broker/MessageStoreModule.cpp:124
#11 0x00002b1cac51260c in qpid::broker::Queue::enqueue (this=0x5376c10,
    ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>)
    at qpid/broker/Queue.cpp:753
#12 0x00002b1cac513dc6 in qpid::broker::Queue::deliver (this=0x5376c10,
    msg=...) at qpid/broker/Queue.cpp:164
#13 0x00002b1cac4b67e3 in qpid::broker::DeliverableMessage::deliverTo (
    this=0x4346d050, queue=...) at qpid/broker/DeliverableMessage.cpp:31
#14 0x00002b1cac4ce065 in qpid::broker::Exchange::doRoute (this=0x52a99b0,
    msg=..., b=...) at qpid/broker/Exchange.cpp:91
#15 0x00002b1cac56ea8c in qpid::broker::TopicExchange::route (this=0x52a99b0,
    msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321
#16 0x00002b1cac544bf3 in qpid::broker::SemanticState::route (
    this=<value optimized out>, msg=..., strategy=...)
    at qpid/broker/SemanticState.cpp:461
#17 0x00002b1cac545b4d in qpid::broker::SemanticState::handle (this=0x5333a78,
    msg=...) at qpid/broker/SemanticState.cpp:415
#18 0x00002b1cac56a90e in qpid::broker::SessionState::handleContent (
    this=0x53338a0, frame=..., id=<value optimized out>)
    at qpid/broker/SessionState.cpp:249
#19 0x00002b1cac56aeb0 in qpid::broker::SessionState::handleIn (
    this=0x53338a0, frame=...) at qpid/broker/SessionState.cpp:327
#20 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0x53336c0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#21 0x00002b1cac4a7ee2 in operator() (this=0x5332a30, frame=...)
    at qpid/framing/Handler.h:42
#22 qpid::broker::Connection::received (this=0x5332a30, frame=...)
    at qpid/broker/Connection.cpp:143
#23 0x00002b1cac487d84 in qpid::amqp_0_10::Connection::decode (this=0x5332500,
    buffer=<value optimized out>, size=<value optimized out>)
    at qpid/amqp_0_10/Connection.cpp:58
#24 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff (
    this=0x52edf80, buff=0x52edfe0) at qpid/sys/AsynchIOHandler.cpp:135
#25 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
    this=0x155, a0=..., a1=0x400)
    at /usr/include/boost/function/function_template.hpp:576
#26 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable (
    this=0x52f1610, h=...) at qpid/sys/posix/AsynchIO.cpp:418
#27 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x155, a0=...)
    at /usr/include/boost/function/function_template.hpp:576
#28 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent (
    this=0x52f1618, type=<value optimized out>)
    at qpid/sys/DispatchHandle.cpp:278
#29 0x0000003c56329fff in process (this=0x5283eb0) at qpid/sys/Poller.h:123
#30 qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:483
#31 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0x9d) at qpid/sys/posix/Thread.cpp:35
#32 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#33 0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 30175):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 30176):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 30177):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 30178):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 30179):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 30180):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0x5283eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 30166):
#0  0x00002b1cac50800d in atomic_exchange_and_add (this=<value optimized out>,
    __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:50
#1  weak_release (this=<value optimized out>, __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:157
#2  boost::detail::weak_count::~weak_count (this=<value optimized out>,
    __in_chrg=<value optimized out>)
    at /usr/include/boost/detail/shared_count.hpp:262
#3  0x00002b1cac50661f in ~list (this=0x2aaab0058fb0,
    __in_chrg=<value optimized out>) at /usr/include/boost/weak_ptr.hpp:27
#4  qpid::broker::PersistableMessage::~PersistableMessage (
    this=0x2aaab0058fb0, __in_chrg=<value optimized out>)
    at qpid/broker/PersistableMessage.cpp:34
#5  0x00002b1cac4fbaa6 in qpid::broker::Message::~Message (
    this=0x2aaab0058fb0, __in_chrg=<value optimized out>)
    at qpid/broker/Message.cpp:59
#6  0x00002b1cade2c63c in ~intrusive_ptr (this=0x2aaaac032840,
    __in_chrg=<value optimized out>)
    at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42
#7  mrg::msgstore::DataTokenImpl::~DataTokenImpl (this=0x2aaaac032840,
    __in_chrg=<value optimized out>) at DataTokenImpl.cpp:30
#8  0x00002b1cade2dd70 in release (this=<value optimized out>, dtokl=...)
    at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42
#9  mrg::msgstore::JournalImpl::wr_aio_cb (this=<value optimized out>,
    dtokl=...) at JournalImpl.cpp:584
#10 0x00002b1cadea04a7 in mrg::journal::wmgr::get_events (this=0x53bd498,
    state=UNUSED) at jrnl/wmgr.cpp:755
#11 0x00002b1cadea2d7f in mrg::journal::wmgr::write_flush (this=0x53bd498)
    at jrnl/wmgr.cpp:615
#12 0x00002b1cadea314d in mrg::journal::wmgr::flush_check (
    this=0x5858585858585858, res=@0x7fff8596dadc, cont=@0x7fff8596dae2,
    done=@0x7fff8596dae1) at jrnl/wmgr.cpp:530
#13 0x00002b1cadea4921 in mrg::journal::wmgr::enqueue (this=0x53bd498,
    data_buff=0x2aaab005ad40, tot_data_len=1115,
    this_data_len=<value optimized out>, dtokp=0x2aaab000be20, xid_ptr=0x0,
    xid_len=0, transient=false, external=false) at jrnl/wmgr.cpp:208
#14 0x00002b1cade7f3be in mrg::journal::jcntl::enqueue_data_record (
    this=0x53bd138, data_buff=0x2aaab005ad40, tot_data_len=1115,
    this_data_len=1115, dtokp=0x2aaab000be20, transient=<value optimized out>)
    at jrnl/jcntl.cpp:206
#15 0x00002b1cade34283 in mrg::msgstore::JournalImpl::enqueue_data_record (
    this=0x53bd130, data_buff=0x2aaab008fa60, tot_data_len=46912586193264,
    this_data_len=46912585596984, dtokp=0x2aaab0003cb8, transient=192)
    at JournalImpl.cpp:355
#16 0x00002b1cade3f1a7 in mrg::msgstore::MessageStoreImpl::store (
    this=0x52a0710, queue=0x53bc6a0, txn=0x7fff8596e210, messageId=...,
    message=..., newId=<value optimized out>) at MessageStoreImpl.cpp:1474
#17 0x00002b1cade4d01b in mrg::msgstore::MessageStoreImpl::enqueue (
    this=0x52a0710, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
#18 0x00002b1cac5035cb in qpid::broker::MessageStoreModule::enqueue (
    this=<value optimized out>, ctxt=0x2aaab008fa60, msg=..., queue=...)
    at qpid/broker/MessageStoreModule.cpp:124
#19 0x00002b1cac51260c in qpid::broker::Queue::enqueue (this=0x53bc6a0,
    ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>)
    at qpid/broker/Queue.cpp:753
#20 0x00002b1cac513dc6 in qpid::broker::Queue::deliver (this=0x53bc6a0,
    msg=...) at qpid/broker/Queue.cpp:164
#21 0x00002b1cac4b67e3 in qpid::broker::DeliverableMessage::deliverTo (
    this=0x7fff8596f460, queue=...) at qpid/broker/DeliverableMessage.cpp:31
#22 0x00002b1cac4ce065 in qpid::broker::Exchange::doRoute (this=0x52a99b0,
    msg=..., b=...) at qpid/broker/Exchange.cpp:91
#23 0x00002b1cac56ea8c in qpid::broker::TopicExchange::route (this=0x52a99b0,
    msg=..., routingKey=...) at qpid/broker/TopicExchange.cpp:321
#24 0x00002b1cac544bf3 in qpid::broker::SemanticState::route (
    this=<value optimized out>, msg=..., strategy=...)
    at qpid/broker/SemanticState.cpp:461
#25 0x00002b1cac545b4d in qpid::broker::SemanticState::handle (this=0x52f1178,
    msg=...) at qpid/broker/SemanticState.cpp:415
#26 0x00002b1cac56a90e in qpid::broker::SessionState::handleContent (
    this=0x52f0fa0, frame=..., id=<value optimized out>)
    at qpid/broker/SessionState.cpp:249
#27 0x00002b1cac56aeb0 in qpid::broker::SessionState::handleIn (
    this=0x52f0fa0, frame=...) at qpid/broker/SessionState.cpp:327
#28 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0x52ee210, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#29 0x00002b1cac4a7ee2 in operator() (this=0x52ecdb0, frame=...)
    at qpid/framing/Handler.h:42
#30 qpid::broker::Connection::received (this=0x52ecdb0, frame=...)
    at qpid/broker/Connection.cpp:143
#31 0x00002b1cac487d84 in qpid::amqp_0_10::Connection::decode (this=0x52ed770,
    buffer=<value optimized out>, size=<value optimized out>)
    at qpid/amqp_0_10/Connection.cpp:58
#32 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff (
    this=0x52ed710, buff=0x52abd70) at qpid/sys/AsynchIOHandler.cpp:135
#33 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
    this=0xffffffff, a0=..., a1=0x2aaab0091970)
    at /usr/include/boost/function/function_template.hpp:576
#34 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable (
    this=0x52ab8d0, h=...) at qpid/sys/posix/AsynchIO.cpp:418
#35 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xffffffff, a0=...)
    at /usr/include/boost/function/function_template.hpp:576
#36 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent (
    this=0x52ab8d8, type=<value optimized out>)
    at qpid/sys/DispatchHandle.cpp:278
#37 0x0000003c56329fff in process (this=0x5283eb0) at qpid/sys/Poller.h:123
#38 qpid::sys::Poller::run (this=0x5283eb0)
    at qpid/sys/epoll/EpollPoller.cpp:483
#39 0x00002b1cac4942f2 in qpid::broker::Broker::run (
    this=<value optimized out>) at qpid/broker/Broker.cpp:334
#40 0x0000000000406ae6 in QpiddBroker::execute (this=<value optimized out>,
    options=0x5283160) at posix/QpiddBroker.cpp:176
#41 0x00000000004055af in main (argc=23, argv=0x7fff85971cf8) at qpidd.cpp:80
(gdb) quit