Description of problem:

While running the test in order to reproduce bug 695716 a broker crash occurred that appears to be unrelated to BZ695716.

The bug seems to be valid only for RHEL6, no crash on the broker occurred while testing with RHEL5.6.

This was seen on RHEL6.1, architectures: x86_64, i686

Version-Release number of selected component (if applicable):
python-qpid-0.10-1.el6.noarch
python-qpid-qmf-0.10-6.el6.x86_64
qpid-cpp-client-0.10-3.el6.x86_64
qpid-cpp-client-devel-0.10-3.el6.x86_64
qpid-cpp-client-devel-docs-0.10-3.el6.noarch
qpid-cpp-client-rdma-0.10-3.el6.x86_64
qpid-cpp-client-ssl-0.10-3.el6.x86_64
qpid-cpp-debuginfo-0.10-3.el6.x86_64
qpid-cpp-server-0.10-3.el6.x86_64
qpid-cpp-server-cluster-0.10-3.el6.x86_64
qpid-cpp-server-devel-0.10-3.el6.x86_64
qpid-cpp-server-rdma-0.10-3.el6.x86_64
qpid-cpp-server-ssl-0.10-3.el6.x86_64
qpid-cpp-server-store-0.10-3.el6.x86_64
qpid-cpp-server-xml-0.10-3.el6.x86_64
qpid-java-client-0.10-4.el6.noarch
qpid-java-common-0.10-4.el6.noarch
qpid-java-example-0.10-4.el6.noarch
qpid-qmf-0.10-6.el6.x86_64
qpid-qmf-devel-0.10-6.el6.x86_64
qpid-tests-0.10-1.el6.noarch
qpid-tools-0.10-3.el6.noarch

How reproducible:
very hard ~0.00004%

Segmentation fault occurred independently on two VM servers (with RHEL6.1 installed):
RHEL6.1 x86_64, 0.00004% (loop number 27453), test was running over 7 hours
RHEL6.1 i686, 0.00002% (loop number 47794), test was running over 12 hours


Steps to Reproduce:
please follow bug 695716, comment 2
(the broker was started from command line with logging set to info+)
  
Actual results:
qpidd broker crashes.

Expected results:
qpidd broker should not crash.

Additional info:

Core was generated by `qpidd --log-enable=info+'.
Program terminated with signal 11, Segmentation fault.
#0  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
336		      if (__w->_M_color == _S_red) 
(gdb) info threads
  3 Thread 0x7f77b0a1e700 (LWP 6355)  std::_Rb_tree_insert_and_rebalance (__insert_left=true, __x=0x7f77ac249830, __p=0x1fa3258, __header=...) at ../../../../libstdc++-v3/src/tree.cc:185
  2 Thread 0x7f77b141f700 (LWP 6354)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
* 1 Thread 0x7f77b87407a0 (LWP 6353)  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
(gdb) thread apply all bt

Thread 3 (Thread 0x7f77b0a1e700 (LWP 6355)):
#0  std::_Rb_tree_insert_and_rebalance (__insert_left=true, __x=0x7f77ac249830, __p=0x1fa3258, __header=...) at ../../../../libstdc++-v3/src/tree.cc:185
#1  0x00007f77b81e2581 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_ (this=0x1fa3250, __x=<value optimized out>, __p=0x1fa3258, __v="\006\004\a\000LANG$cc0de65b-7f64-4055-9781-341631c9d0bc\ttx-test-5")
    at /usr/include/c++/4.4.4/bits/stl_tree.h:883
#2  0x00007f77b8222240 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_unique(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libqpidbroker.so.5.0.0
#3  0x00007f77b8221ce3 in insert (this=0x1fa3240, ctxt=<value optimized out>) at /usr/include/c++/4.4.4/bits/stl_set.h:411
#4  qpid::broker::NullMessageStore::prepare (this=0x1fa3240, ctxt=<value optimized out>) at qpid/broker/NullMessageStore.cpp:129
#5  0x00007f77b81f085e in qpid::broker::DtxWorkRecord::prepare (this=0x7f77ac2bd4a0) at qpid/broker/DtxWorkRecord.cpp:47
#6  0x00007f77b81eee08 in qpid::broker::DtxManager::prepare (this=0x1fa46d0, xid="\006\004\a\000LANG$cc0de65b-7f64-4055-9781-341631c9d0bc\ttx-test-5") at qpid/broker/DtxManager.cpp:60
#7  0x00007f77b8268ad9 in qpid::broker::SessionAdapter::DtxHandlerImpl::prepare (this=0x7f77ac4d7900, xid=<value optimized out>) at qpid/broker/SessionAdapter.cpp:615
#8  0x00007f77b7d43c3d in invoke<qpid::framing::AMQP_ServerOperations::DtxHandler> (this=<value optimized out>, body=<value optimized out>) at qpid/framing/DtxPrepareBody.h:64
#9  qpid::framing::AMQP_ServerOperations::DtxHandler::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at qpid/framing/ServerInvoker.cpp:614
#10 0x00007f77b7d45a11 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7f77b0a1c210, body=...) at qpid/framing/ServerInvoker.cpp:318
#11 0x00007f77b82700b8 in invoke<qpid::broker::SessionAdapter> (this=<value optimized out>, method=0x7f77ac4debb0, id=...) at qpid/framing/Invoker.h:67
#12 qpid::broker::SessionState::handleCommand (this=<value optimized out>, method=0x7f77ac4debb0, id=...) at qpid/broker/SessionState.cpp:208
#13 0x00007f77b82735a9 in qpid::broker::SessionState::handleIn (this=0x7f77ac4d7490, frame=...) at qpid/broker/SessionState.cpp:357
#14 0x00007f77b7db4b19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f77ac4d7c20, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#15 0x00007f77b81d0a92 in operator() (this=0x7f77ac4d63e0, frame=...) at qpid/framing/Handler.h:42
#16 qpid::broker::Connection::received (this=0x7f77ac4d63e0, frame=...) at qpid/broker/Connection.cpp:164
#17 0x00007f77b81a9a0d in qpid::amqp_0_10::Connection::decode (this=0x7f77ac4d5e60, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#18 0x00007f77b7de3e56 in qpid::sys::AsynchIOHandler::readbuff (this=0x7f77ac4d4690, buff=0x7f77ac4d4240) at qpid/sys/AsynchIOHandler.cpp:135
#19 0x00007f77b7d24b69 in operator() (this=0x7f77ac4d4b40, h=...) at /usr/include/boost/function/function_template.hpp:1013
#20 qpid::sys::posix::AsynchIO::readable (this=0x7f77ac4d4b40, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#21 0x00007f77b7de9633 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /usr/lib64/libqpidcommon.so.5.0.0
#22 0x00007f77b7de6836 in qpid::sys::DispatchHandle::processEvent (this=0x7f77ac4d4b48, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#23 0x00007f77b7d3063d in process (this=0x1fa21b0) at qpid/sys/Poller.h:131
#24 qpid::sys::Poller::run (this=0x1fa21b0) at qpid/sys/epoll/EpollPoller.cpp:519
#25 0x00007f77b7d27faa in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#26 0x00007f77b60057e1 in start_thread (arg=0x7f77b0a1e700) at pthread_create.c:301
#27 0x00007f77b63018ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f77b141f700 (LWP 6354)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
#1  0x00007f77b7dee412 in wait (this=0x1fa4200) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0x1fa4200) at ../include/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0x1fa4200) at qpid/sys/Timer.cpp:153
#4  0x00007f77b7d27faa in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x00007f77b60057e1 in start_thread (arg=0x7f77b141f700) at pthread_create.c:301
#6  0x00007f77b63018ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f77b87407a0 (LWP 6353)):
#0  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
#1  0x00007f77b81e4219 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::erase(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libqpidbroker.so.5.0.0
#2  0x00007f77b8221c33 in erase (this=0x1fa3240, ctxt=<value optimized out>) at /usr/include/c++/4.4.4/bits/stl_set.h:491
#3  qpid::broker::NullMessageStore::commit (this=0x1fa3240, ctxt=<value optimized out>) at qpid/broker/NullMessageStore.cpp:134
#4  0x00007f77b81f192e in qpid::broker::DtxWorkRecord::commit (this=0x23157c0, onePhase=false) at qpid/broker/DtxWorkRecord.cpp:79
#5  0x00007f77b81ed2f4 in qpid::broker::DtxManager::commit (this=0x1fa46d0, xid="\006\004\a\000LANG$792ad566-7e41-41fe-bc3b-8ec826cf0c1d\ttx-test-1", onePhase=false) at qpid/broker/DtxManager.cpp:71
#6  0x00007f77b82689fd in qpid::broker::SessionAdapter::DtxHandlerImpl::commit (this=0x2309ca0, xid=<value optimized out>, onePhase=false) at qpid/broker/SessionAdapter.cpp:626
#7  0x00007f77b7d43ef4 in invoke<qpid::framing::AMQP_ServerOperations::DtxHandler> (this=0x7fff13a6f160, body=...) at qpid/framing/DtxCommitBody.h:68
#8  qpid::framing::AMQP_ServerOperations::DtxHandler::Invoker::visit (this=0x7fff13a6f160, body=...) at qpid/framing/ServerInvoker.cpp:602
#9  0x00007f77b7d45da1 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7fff13a6f340, body=...) at qpid/framing/ServerInvoker.cpp:303
#10 0x00007f77b82700b8 in invoke<qpid::broker::SessionAdapter> (this=<value optimized out>, method=0x24b8dc0, id=...) at qpid/framing/Invoker.h:67
#11 qpid::broker::SessionState::handleCommand (this=<value optimized out>, method=0x24b8dc0, id=...) at qpid/broker/SessionState.cpp:208
#12 0x00007f77b82735a9 in qpid::broker::SessionState::handleIn (this=0x2309830, frame=...) at qpid/broker/SessionState.cpp:357
#13 0x00007f77b7db4b19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x230a080, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#14 0x00007f77b81d0a92 in operator() (this=0x20cbc50, frame=...) at qpid/framing/Handler.h:42
#15 qpid::broker::Connection::received (this=0x20cbc50, frame=...) at qpid/broker/Connection.cpp:164
#16 0x00007f77b81a9a0d in qpid::amqp_0_10::Connection::decode (this=0x20cb6d0, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#17 0x00007f77b7de3e56 in qpid::sys::AsynchIOHandler::readbuff (this=0x20aa950, buff=0x20aa8c0) at qpid/sys/AsynchIOHandler.cpp:135
#18 0x00007f77b7d24b69 in operator() (this=0x20aa9b0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#19 qpid::sys::posix::AsynchIO::readable (this=0x20aa9b0, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#20 0x00007f77b7de9633 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /usr/lib64/libqpidcommon.so.5.0.0
#21 0x00007f77b7de6836 in qpid::sys::DispatchHandle::processEvent (this=0x20aa9b8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#22 0x00007f77b7d3063d in process (this=0x1fa21b0) at qpid/sys/Poller.h:131
#23 qpid::sys::Poller::run (this=0x1fa21b0) at qpid/sys/epoll/EpollPoller.cpp:519
#24 0x00007f77b81c0392 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:385
#25 0x000000000040dcc2 in QpiddBroker::execute (this=<value optimized out>, options=0x1f9c260) at posix/QpiddBroker.cpp:187
#26 0x000000000040a1f2 in main (argc=2, argv=0x7fff13a71828) at qpidd.cpp:80