Bug 703466

Summary:	C++ broker crash - _Rb_tree_rebalance_for_erase
Product:	Red Hat Enterprise MRG	Reporter:	Petr Matousek <pematous>
Component:	qpid-cpp	Assignee:	messaging-bugs <messaging-bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:	ppecka <ppecka>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	Development	CC:	freznice, gsim, iboverma
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-12-07 17:43:19 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	703839

Description Petr Matousek 2011-05-10 12:39:29 UTC

Description of problem:

While running the test in order to reproduce bug 695716 a broker crash occurred that appears to be unrelated to BZ695716.

The bug seems to be valid only for RHEL6, no crash on the broker occurred while testing with RHEL5.6.

This was seen on RHEL6.1, architectures: x86_64, i686

Version-Release number of selected component (if applicable):
python-qpid-0.10-1.el6.noarch
python-qpid-qmf-0.10-6.el6.x86_64
qpid-cpp-client-0.10-3.el6.x86_64
qpid-cpp-client-devel-0.10-3.el6.x86_64
qpid-cpp-client-devel-docs-0.10-3.el6.noarch
qpid-cpp-client-rdma-0.10-3.el6.x86_64
qpid-cpp-client-ssl-0.10-3.el6.x86_64
qpid-cpp-debuginfo-0.10-3.el6.x86_64
qpid-cpp-server-0.10-3.el6.x86_64
qpid-cpp-server-cluster-0.10-3.el6.x86_64
qpid-cpp-server-devel-0.10-3.el6.x86_64
qpid-cpp-server-rdma-0.10-3.el6.x86_64
qpid-cpp-server-ssl-0.10-3.el6.x86_64
qpid-cpp-server-store-0.10-3.el6.x86_64
qpid-cpp-server-xml-0.10-3.el6.x86_64
qpid-java-client-0.10-4.el6.noarch
qpid-java-common-0.10-4.el6.noarch
qpid-java-example-0.10-4.el6.noarch
qpid-qmf-0.10-6.el6.x86_64
qpid-qmf-devel-0.10-6.el6.x86_64
qpid-tests-0.10-1.el6.noarch
qpid-tools-0.10-3.el6.noarch

How reproducible:
very hard ~0.00004%

Segmentation fault occurred independently on two VM servers (with RHEL6.1 installed):
RHEL6.1 x86_64, 0.00004% (loop number 27453), test was running over 7 hours
RHEL6.1 i686, 0.00002% (loop number 47794), test was running over 12 hours


Steps to Reproduce:
please follow bug 695716, comment 2
(the broker was started from command line with logging set to info+)
  
Actual results:
qpidd broker crashes.

Expected results:
qpidd broker should not crash.

Additional info:

Core was generated by `qpidd --log-enable=info+'.
Program terminated with signal 11, Segmentation fault.
#0  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
336		      if (__w->_M_color == _S_red) 
(gdb) info threads
  3 Thread 0x7f77b0a1e700 (LWP 6355)  std::_Rb_tree_insert_and_rebalance (__insert_left=true, __x=0x7f77ac249830, __p=0x1fa3258, __header=...) at ../../../../libstdc++-v3/src/tree.cc:185
  2 Thread 0x7f77b141f700 (LWP 6354)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
* 1 Thread 0x7f77b87407a0 (LWP 6353)  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
(gdb) thread apply all bt

Thread 3 (Thread 0x7f77b0a1e700 (LWP 6355)):
#0  std::_Rb_tree_insert_and_rebalance (__insert_left=true, __x=0x7f77ac249830, __p=0x1fa3258, __header=...) at ../../../../libstdc++-v3/src/tree.cc:185
#1  0x00007f77b81e2581 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_ (this=0x1fa3250, __x=<value optimized out>, __p=0x1fa3258, __v="\006\004\a\000LANG$cc0de65b-7f64-4055-9781-341631c9d0bc\ttx-test-5")
    at /usr/include/c++/4.4.4/bits/stl_tree.h:883
#2  0x00007f77b8222240 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_insert_unique(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libqpidbroker.so.5.0.0
#3  0x00007f77b8221ce3 in insert (this=0x1fa3240, ctxt=<value optimized out>) at /usr/include/c++/4.4.4/bits/stl_set.h:411
#4  qpid::broker::NullMessageStore::prepare (this=0x1fa3240, ctxt=<value optimized out>) at qpid/broker/NullMessageStore.cpp:129
#5  0x00007f77b81f085e in qpid::broker::DtxWorkRecord::prepare (this=0x7f77ac2bd4a0) at qpid/broker/DtxWorkRecord.cpp:47
#6  0x00007f77b81eee08 in qpid::broker::DtxManager::prepare (this=0x1fa46d0, xid="\006\004\a\000LANG$cc0de65b-7f64-4055-9781-341631c9d0bc\ttx-test-5") at qpid/broker/DtxManager.cpp:60
#7  0x00007f77b8268ad9 in qpid::broker::SessionAdapter::DtxHandlerImpl::prepare (this=0x7f77ac4d7900, xid=<value optimized out>) at qpid/broker/SessionAdapter.cpp:615
#8  0x00007f77b7d43c3d in invoke<qpid::framing::AMQP_ServerOperations::DtxHandler> (this=<value optimized out>, body=<value optimized out>) at qpid/framing/DtxPrepareBody.h:64
#9  qpid::framing::AMQP_ServerOperations::DtxHandler::Invoker::visit (this=<value optimized out>, body=<value optimized out>) at qpid/framing/ServerInvoker.cpp:614
#10 0x00007f77b7d45a11 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7f77b0a1c210, body=...) at qpid/framing/ServerInvoker.cpp:318
#11 0x00007f77b82700b8 in invoke<qpid::broker::SessionAdapter> (this=<value optimized out>, method=0x7f77ac4debb0, id=...) at qpid/framing/Invoker.h:67
#12 qpid::broker::SessionState::handleCommand (this=<value optimized out>, method=0x7f77ac4debb0, id=...) at qpid/broker/SessionState.cpp:208
#13 0x00007f77b82735a9 in qpid::broker::SessionState::handleIn (this=0x7f77ac4d7490, frame=...) at qpid/broker/SessionState.cpp:357
#14 0x00007f77b7db4b19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f77ac4d7c20, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#15 0x00007f77b81d0a92 in operator() (this=0x7f77ac4d63e0, frame=...) at qpid/framing/Handler.h:42
#16 qpid::broker::Connection::received (this=0x7f77ac4d63e0, frame=...) at qpid/broker/Connection.cpp:164
#17 0x00007f77b81a9a0d in qpid::amqp_0_10::Connection::decode (this=0x7f77ac4d5e60, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#18 0x00007f77b7de3e56 in qpid::sys::AsynchIOHandler::readbuff (this=0x7f77ac4d4690, buff=0x7f77ac4d4240) at qpid/sys/AsynchIOHandler.cpp:135
#19 0x00007f77b7d24b69 in operator() (this=0x7f77ac4d4b40, h=...) at /usr/include/boost/function/function_template.hpp:1013
#20 qpid::sys::posix::AsynchIO::readable (this=0x7f77ac4d4b40, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#21 0x00007f77b7de9633 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /usr/lib64/libqpidcommon.so.5.0.0
#22 0x00007f77b7de6836 in qpid::sys::DispatchHandle::processEvent (this=0x7f77ac4d4b48, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#23 0x00007f77b7d3063d in process (this=0x1fa21b0) at qpid/sys/Poller.h:131
#24 qpid::sys::Poller::run (this=0x1fa21b0) at qpid/sys/epoll/EpollPoller.cpp:519
#25 0x00007f77b7d27faa in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#26 0x00007f77b60057e1 in start_thread (arg=0x7f77b0a1e700) at pthread_create.c:301
#27 0x00007f77b63018ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f77b141f700 (LWP 6354)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
#1  0x00007f77b7dee412 in wait (this=0x1fa4200) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0x1fa4200) at ../include/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0x1fa4200) at qpid/sys/Timer.cpp:153
#4  0x00007f77b7d27faa in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x00007f77b60057e1 in start_thread (arg=0x7f77b141f700) at pthread_create.c:301
#6  0x00007f77b63018ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f77b87407a0 (LWP 6353)):
#0  std::_Rb_tree_rebalance_for_erase (__z=<value optimized out>, __header=...) at ../../../../libstdc++-v3/src/tree.cc:336
#1  0x00007f77b81e4219 in std::_Rb_tree<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::_Identity<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::erase(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libqpidbroker.so.5.0.0
#2  0x00007f77b8221c33 in erase (this=0x1fa3240, ctxt=<value optimized out>) at /usr/include/c++/4.4.4/bits/stl_set.h:491
#3  qpid::broker::NullMessageStore::commit (this=0x1fa3240, ctxt=<value optimized out>) at qpid/broker/NullMessageStore.cpp:134
#4  0x00007f77b81f192e in qpid::broker::DtxWorkRecord::commit (this=0x23157c0, onePhase=false) at qpid/broker/DtxWorkRecord.cpp:79
#5  0x00007f77b81ed2f4 in qpid::broker::DtxManager::commit (this=0x1fa46d0, xid="\006\004\a\000LANG$792ad566-7e41-41fe-bc3b-8ec826cf0c1d\ttx-test-1", onePhase=false) at qpid/broker/DtxManager.cpp:71
#6  0x00007f77b82689fd in qpid::broker::SessionAdapter::DtxHandlerImpl::commit (this=0x2309ca0, xid=<value optimized out>, onePhase=false) at qpid/broker/SessionAdapter.cpp:626
#7  0x00007f77b7d43ef4 in invoke<qpid::framing::AMQP_ServerOperations::DtxHandler> (this=0x7fff13a6f160, body=...) at qpid/framing/DtxCommitBody.h:68
#8  qpid::framing::AMQP_ServerOperations::DtxHandler::Invoker::visit (this=0x7fff13a6f160, body=...) at qpid/framing/ServerInvoker.cpp:602
#9  0x00007f77b7d45da1 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7fff13a6f340, body=...) at qpid/framing/ServerInvoker.cpp:303
#10 0x00007f77b82700b8 in invoke<qpid::broker::SessionAdapter> (this=<value optimized out>, method=0x24b8dc0, id=...) at qpid/framing/Invoker.h:67
#11 qpid::broker::SessionState::handleCommand (this=<value optimized out>, method=0x24b8dc0, id=...) at qpid/broker/SessionState.cpp:208
#12 0x00007f77b82735a9 in qpid::broker::SessionState::handleIn (this=0x2309830, frame=...) at qpid/broker/SessionState.cpp:357
#13 0x00007f77b7db4b19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x230a080, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#14 0x00007f77b81d0a92 in operator() (this=0x20cbc50, frame=...) at qpid/framing/Handler.h:42
#15 qpid::broker::Connection::received (this=0x20cbc50, frame=...) at qpid/broker/Connection.cpp:164
#16 0x00007f77b81a9a0d in qpid::amqp_0_10::Connection::decode (this=0x20cb6d0, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#17 0x00007f77b7de3e56 in qpid::sys::AsynchIOHandler::readbuff (this=0x20aa950, buff=0x20aa8c0) at qpid/sys/AsynchIOHandler.cpp:135
#18 0x00007f77b7d24b69 in operator() (this=0x20aa9b0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#19 qpid::sys::posix::AsynchIO::readable (this=0x20aa9b0, h=...) at qpid/sys/posix/AsynchIO.cpp:428
#20 0x00007f77b7de9633 in boost::function1<void, qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const () from /usr/lib64/libqpidcommon.so.5.0.0
#21 0x00007f77b7de6836 in qpid::sys::DispatchHandle::processEvent (this=0x20aa9b8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#22 0x00007f77b7d3063d in process (this=0x1fa21b0) at qpid/sys/Poller.h:131
#23 qpid::sys::Poller::run (this=0x1fa21b0) at qpid/sys/epoll/EpollPoller.cpp:519
#24 0x00007f77b81c0392 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:385
#25 0x000000000040dcc2 in QpiddBroker::execute (this=<value optimized out>, options=0x1f9c260) at posix/QpiddBroker.cpp:187
#26 0x000000000040a1f2 in main (argc=2, argv=0x7fff13a71828) at qpidd.cpp:80

Comment 1 Gordon Sim 2011-05-10 12:57:40 UTC

That looks to be a similar issue to bug 695716, very likely a result of the same root defect. 

You listed above that you are testing on qpid-cpp-server-0.10-3.el6.x86_64, whereas bug 695716 is marked as fixed in qpid-cpp-mrg-0.10-4. Is that an error?

Comment 2 Petr Matousek 2011-05-10 14:29:46 UTC

No, it is not an error. So the issue was fixed in qpid-cpp-mrg-0.10-4 for RHEL4/5, but not yet available in any RHEL6 package.

-> Waiting for new packages in order to verify this bug and bug 695716

Comment 3 Gordon Sim 2011-05-10 14:56:40 UTC

I'd close this and clone bug 695716 for RHEL6 (or use this as the rhel6 clone if you prefer). That lets the issue be verified form MRG2.0 independently from the RHEL6 package updates (which are controlled separately).

Comment 8 Frantisek Reznicek 2011-05-11 14:49:36 UTC

This issue has most probably same rootcause as bug 703839 and will be tested together with bug 703839 as soon as RHEL6 packages with fix are available.

Comment 9 Petr Matousek 2011-06-02 13:52:20 UTC

This issue will be tested soon on qpid-cpp-mrg-0.10-5.el6, removing blocker.

Comment 10 ppecka 2011-06-08 10:10:30 UTC

retesting on rhel6 -5 packages are over 70000 runs with no crash so far

Comment 11 ppecka 2011-06-09 11:58:16 UTC

Verified on RHEL6 both i686 / x86_64
(over 250000 runs)

# rpm -qa | grep qpid
qpid-cpp-client-devel-0.10-5.el6.i686
qpid-cpp-server-cluster-0.10-5.el6.i686
rh-tests-distribution-MRG-Messaging-qpid_common-1.6-57.noarch
qpid-java-common-0.10-6.el6.noarch
qpid-cpp-server-0.10-5.el6.i686
qpid-cpp-client-rdma-0.10-5.el6.i686
qpid-qmf-devel-0.10-10.el6.i686
qpid-cpp-server-rdma-0.10-5.el6.i686
qpid-java-jca-0.10-6.el6.noarch
qpid-cpp-client-0.10-5.el6.i686
ruby-qpid-qmf-0.10-10.el6.i686
qpid-tests-0.10-1.el6.noarch
qpid-cpp-server-ssl-0.10-5.el6.i686
qpid-cpp-client-devel-docs-0.10-5.el6.noarch
python-qpid-qmf-0.10-10.el6.i686
qpid-java-client-0.10-6.el6.noarch
qpid-tools-0.10-4.el6.noarch
qpid-cpp-server-xml-0.10-5.el6.i686
python-qpid-0.10-1.el6.noarch
qpid-qmf-0.10-10.el6.i686
qpid-cpp-client-ssl-0.10-5.el6.i686
qpid-cpp-server-devel-0.10-5.el6.i686
ruby-qpid-0.7.946106-2.el6.i686
qpid-java-example-0.10-6.el6.noarch
rh-qpid-cpp-tests-0.10-5.el6.i686

--> VERIFIED