Bug 554980

Summary:	[qpidd+store] broker rarely aborts when stressed by perftest
Product:	Red Hat Enterprise MRG	Reporter:	Frantisek Reznicek <freznice>
Component:	qpid-cpp	Assignee:	Andrew Stitcher <astitcher>
Status:	CLOSED ERRATA	QA Contact:	Frantisek Reznicek <freznice>
Severity:	high	Docs Contact:
Priority:	high
Version:	1.2	CC:	esammons, gsim
Target Milestone:	1.3
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Under rare conditions a broker with the persistence storage module could crash.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-10-14 16:04:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Frantisek Reznicek 2010-01-13 10:09:31 UTC

Description of problem:
There is running long term qpidd+store stress test using perftest with swept parameters, during that test (qpid_test_qpidd-perftest_performance) which was primarily focused on bug 484048 verification and test itself debugging I detected following qpidd aborts all as I believe related to message store and/or DispatchHandle.

There are two cores generated + three another added to bug 484048 which I believe might be relevant to that bz.

May I ask you for backtrace review and suggest whether create separate BZ and/or alpha] is relevant to known issue, please?


alpha] RHEL 5.4 i386, abort

root@mrg-qe-11:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# gdb `which qpidd` /root/MRG/Messaging/qpid_test_qpidd-perftest_performance/core.506
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.

warning: .dynamic section for "/lib/libuuid.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/usr/lib/libsasl2.so.2" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done.
done.
...
done.
Loaded symbols for /usr/lib/qpid/daemon/rdma.so
Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
Program terminated with signal 6, Aborted.
[New process 520]
[New process 521]
[New process 519]
[New process 518]
[New process 517]
[New process 516]
[New process 515]
[New process 514]
[New process 513]
[New process 512]
[New process 511]
[New process 506]
#0  0x00ddd410 in __kernel_vsyscall ()
(gdb) info threads
  12 process 506  0x00ddd410 in __kernel_vsyscall ()
  11 process 511  0x00ddd410 in __kernel_vsyscall ()
  10 process 512  0x00ddd410 in __kernel_vsyscall ()
  9 process 513  0x00cebcae in pthread_mutex_unlock () from /lib/libpthread.so.0
  8 process 514  0x00ddd410 in __kernel_vsyscall ()
  7 process 515  0x00ddd410 in __kernel_vsyscall ()
  6 process 516  0x00ddd410 in __kernel_vsyscall ()
  5 process 517  0x00ddd410 in __kernel_vsyscall ()
  4 process 518  0x00ddd410 in __kernel_vsyscall ()
  3 process 519  0x00ddd410 in __kernel_vsyscall ()
  2 process 521  0x00ddd410 in __kernel_vsyscall ()
* 1 process 520  0x00ddd410 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 12 (process 506):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x002f7f61 in qpid::broker::Broker::run (this=0x8475ec0) at qpid/broker/Broker.cpp:324
#6  0x0804dd44 in QpiddBroker::execute (this=0xbfb7cc85, options=0x8473760) at posix/QpiddBroker.cpp:166
#7  0x0804c6c7 in main (argc=23, argv=0xbfb7cd34) at qpidd.cpp:77

Thread 11 (process 511):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00cecd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00553414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x003cdc5e in qpid::broker::Timer::run (this=0x84760c8) at qpid/sys/posix/Condition.h:69
#4  0x0071f871 in runRunnable (p=0x84760c8) at qpid/sys/posix/Thread.cpp:35
#5  0x00ce873b in start_thread () from /lib/libpthread.so.0
#6  0x00546cfe in clone () from /lib/libc.so.6

Thread 10 (process 512):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00cecd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00553414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x003cdc5e in qpid::broker::Timer::run (this=0x8476148) at qpid/sys/posix/Condition.h:69
#4  0x0071f871 in runRunnable (p=0x8476148) at qpid/sys/posix/Thread.cpp:35
#5  0x00ce873b in start_thread () from /lib/libpthread.so.0
#6  0x00546cfe in clone () from /lib/libc.so.6

Thread 9 (process 513):
#0  0x00cebcae in pthread_mutex_unlock () from /lib/libpthread.so.0
#1  0x00553566 in pthread_mutex_unlock () from /lib/libc.so.6
#2  0x002a5bc2 in qpid::sys::Mutex::unlock (this=0xb092bd30) at qpid/sys/posix/Mutex.h:120
#3  0x00374383 in qpid::broker::Message::allEnqueuesComplete (this=0xb092bcc8) at qpid/sys/Mutex.h:34
---Type <return> to continue, or q <return> to quit---
#4  0x0031ff17 in qpid::broker::PersistableMessage::enqueueComplete (this=0xb092bcc8)
    at qpid/broker/PersistableMessage.cpp:87
#5  0x076fb890 in mrg::msgstore::JournalImpl::wr_aio_cb () from /usr/lib/qpid/daemon/msgstore.so
#6  0x07787ba3 in mrg::journal::wmgr::get_events () from /usr/lib/qpid/daemon/msgstore.so
#7  0x0775c09e in mrg::journal::jcntl::get_wr_events () from /usr/lib/qpid/daemon/msgstore.so
#8  0x076fe17d in mrg::msgstore::JournalImpl::getEventsFire () from /usr/lib/qpid/daemon/msgstore.so
#9  0x076fe641 in mrg::msgstore::GetEventsFireEvent::fire () from /usr/lib/qpid/daemon/msgstore.so
#10 0x003cd9c2 in qpid::broker::Timer::run (this=0x8468848) at qpid/broker/Timer.cpp:79
#11 0x0071f871 in runRunnable (p=0x8468848) at qpid/sys/posix/Thread.cpp:35
#12 0x00ce873b in start_thread () from /lib/libpthread.so.0
#13 0x00546cfe in clone () from /lib/libc.so.6

Thread 8 (process 514):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 7 (process 515):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00cef509 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0x00ceabbf in _L_lock_885 () from /lib/libpthread.so.0
#3  0x00ceaa86 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x00553536 in pthread_mutex_lock () from /lib/libc.so.6
#5  0x0775b463 in mrg::journal::jcntl::enqueue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#6  0x076fdf6c in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#7  0x0771191a in mrg::msgstore::MessageStoreImpl::store () from /usr/lib/qpid/daemon/msgstore.so
#8  0x0772aa52 in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib/qpid/daemon/msgstore.so
#9  0x0037af8d in qpid::broker::MessageStoreModule::enqueue (this=0x8467948, ctxt=0x0, msg=@0xb5644a40, queue=@0x84bc978)
    at qpid/broker/MessageStoreModule.cpp:124
---Type <return> to continue, or q <return> to quit---
#10 0x003140e1 in qpid::broker::Queue::enqueue (this=0x84bc978, ctxt=0x0, msg={p_ = 0xb5644d10},
    suppressPolicyCheck=false) at qpid/broker/Queue.cpp:717
#11 0x00318908 in qpid::broker::Queue::deliver (this=0x84bc978, msg=@0xb56451d4) at qpid/broker/Queue.cpp:171
#12 0x00334dce in qpid::broker::DeliverableMessage::deliverTo (this=0xb56451cc, queue=@0x84bd8f4)
    at qpid/broker/DeliverableMessage.cpp:31
#13 0x00303a1d in qpid::broker::Exchange::doRoute (this=0x84689f4, msg=@0xb56451cc, b=
        {px = 0xb5644df0, pn = {pi_ = 0x766845}}) at qpid/broker/Exchange.cpp:88
#14 0x00338e30 in qpid::broker::DirectExchange::route (this=0x84689f4, msg=@0xb56451cc, routingKey=@0xb5645118)
    at qpid/broker/DirectExchange.cpp:141
#15 0x003a05b7 in qpid::broker::SemanticState::route (this=0x85132b8, msg={p_ = 0xb56451e8}, strategy=@0xb56451cc)
    at qpid/broker/SemanticState.cpp:386
#16 0x003a187a in qpid::broker::SemanticState::handle (this=0x85132b8, msg={p_ = 0xb5645328})
    at qpid/broker/SemanticState.cpp:349
#17 0x003c5e73 in qpid::broker::SessionState::handleContent (this=0x8513190, frame=@0xb5645aa8, id=@0xb564540c)
    at qpid/broker/SessionState.cpp:248
#18 0x003c6a03 in qpid::broker::SessionState::handleIn (this=0x8513190, frame=@0xb5645aa8)
    at qpid/broker/SessionState.cpp:326
#19 0x003c6b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x8513290, t=@0xb5645aa8) at qpid/framing/Handler.h:67
#20 0x0074b266 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x84d1fd8, f=@0xb5645aa8)
    at qpid/amqp_0_10/SessionHandler.cpp:93
#21 0x003c6b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x84d1fe0, t=@0xb5645aa8) at qpid/framing/Handler.h:67
#22 0x00326610 in qpid::broker::Connection::received (this=0x84d20c8, frame=@0xb5645aa8) at qpid/framing/Handler.h:42
#23 0x002e8d0f in qpid::amqp_0_10::Connection::decode (this=0x84d2380, buffer=0x84d2b60 "\a\003", size=65536)
    at qpid/amqp_0_10/Connection.cpp:55
#24 0x0039ad94 in qpid::broker::SecureConnection::decode (this=0x84ba458, buffer=0x84d2b60 "\a\003", size=128)
    at qpid/broker/SecureConnection.cpp:42
#25 0x00779be6 in qpid::sys::AsynchIOHandler::readbuff (this=0x84ba408, buff=0x846aa38)
    at qpid/sys/AsynchIOHandler.cpp:113
#26 0x003f46f4 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (
---Type <return> to continue, or q <return> to quit---
    function_obj_ptr={obj_ptr = 0x846b258, const_obj_ptr = 0x846b258, func_ptr = 0x846b258, data = "X"}, a0=@0x846b138,
    a1=0x846aa38) at /usr/include/boost/bind/mem_fn_template.hpp:252
#27 0x0071b82e in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (this=0x846b1b4, a0=@0x846b138, a1=0x846aa38)
    at /usr/include/boost/function/function_template.hpp:576
#28 0x0071971e in qpid::sys::posix::AsynchIO::readable (this=0x846b138, h=@0x846b13c) at qpid/sys/posix/AsynchIO.cpp:448
#29 0x0071a29d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0x846b2a0, const_obj_ptr = 0x846b2a0, func_ptr = 0x846b2a0, data = "�"}, a0=@0x846b13c)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#30 0x0077df14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x846b144, a0=@0x846b13c) at /usr/include/boost/function/function_template.hpp:576
#31 0x0077d5f9 in qpid::sys::DispatchHandle::processEvent (this=0x846b13c, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#32 0x0072a934 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/Poller.h:122
#33 0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#34 0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#35 0x00ce873b in start_thread () from /lib/libpthread.so.0
#36 0x00546cfe in clone () from /lib/libc.so.6

Thread 6 (process 516):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 5 (process 517):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 4 (process 518):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 3 (process 519):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 2 (process 521):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x00547376 in epoll_wait () from /lib/libc.so.6
#2  0x007298e9 in qpid::sys::Poller::wait (this=0x8475788, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
---Type <return> to continue, or q <return> to quit---
#3  0x0072a916 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#5  0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#6  0x00ce873b in start_thread () from /lib/libpthread.so.0
#7  0x00546cfe in clone () from /lib/libc.so.6

Thread 1 (process 520):
#0  0x00ddd410 in __kernel_vsyscall ()
#1  0x0049ddf0 in raise () from /lib/libc.so.6
#2  0x0049f701 in abort () from /lib/libc.so.6
#3  0x001f14d0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x001eef35 in ?? () from /usr/lib/libstdc++.so.6
#5  0x001eef72 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x001ef0aa in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x0077d566 in qpid::sys::DispatchHandle::processEvent (this=0x853453c, type=qpid::sys::Poller::READABLE)
    at qpid/sys/posix/Mutex.h:120
#8  0x0072a934 in qpid::sys::Poller::run (this=0x8475788) at qpid/sys/Poller.h:122
#9  0x0077a8d4 in qpid::sys::Dispatcher::run (this=0xbfb7c998) at qpid/sys/Dispatcher.cpp:37
#10 0x0071f871 in runRunnable (p=0xbfb7c998) at qpid/sys/posix/Thread.cpp:35
#11 0x00ce873b in start_thread () from /lib/libpthread.so.0
#12 0x00546cfe in clone () from /lib/libc.so.6
(gdb) quit
root@mrg-qe-11:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# rpm -qa | grep -E '(qpidd|opena)'
qpidd-rdma-0.5.752581-34.el5
qpidd-acl-0.5.752581-34.el5
openais-devel-0.80.6-8.el5_4.1
qpidd-cluster-0.5.752581-34.el5
qpidd-xml-0.5.752581-34.el5
openais-debuginfo-0.80.6-8.el5_4.1
qpidd-devel-0.5.752581-34.el5
qpidd-0.5.752581-34.el5
openais-0.80.6-8.el5_4.1
qpidd-ssl-0.5.752581-34.el5


beta] RHEL 5.4 i386 abort in mrg::journal::wmgr::get_events
root@mrg-qe-12:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# gdb `which qpidd` /root/MRG/Messaging/qpid_test_qpidd-perftest_performance/core.8200
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.

warning: .dynamic section for "/usr/lib/libnssutil3.so" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done.
done.
...
Loaded symbols for /usr/lib/qpid/daemon/rdma.so
Reading symbols from /usr/lib/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib/qpid/daemon/ssl.so
Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
Program terminated with signal 6, Aborted.
[New process 8215]
[New process 8214]
[New process 8213]
[New process 8212]
[New process 8211]
[New process 8210]
[New process 8209]
[New process 8208]
[New process 8207]
[New process 8206]
[New process 8205]
[New process 8200]
#0  0x00898410 in __kernel_vsyscall ()
(gdb) info threads
  12 process 8200  0x00898410 in __kernel_vsyscall ()
  11 process 8205  0x00898410 in __kernel_vsyscall ()
  10 process 8206  0x00898410 in __kernel_vsyscall ()
  9 process 8207  0x00bf8805 in mrg::journal::wmgr::get_events () from /usr/lib/qpid/daemon/msgstore.so
  8 process 8208  0x00898410 in __kernel_vsyscall ()
  7 process 8209  0x00898410 in __kernel_vsyscall ()
  6 process 8210  0x00898410 in __kernel_vsyscall ()
  5 process 8211  0x00898410 in __kernel_vsyscall ()
  4 process 8212  0x00898410 in __kernel_vsyscall ()
  3 process 8213  0x00898410 in __kernel_vsyscall ()
  2 process 8214  0x00898410 in __kernel_vsyscall ()
* 1 process 8215  0x00898410 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 12 (process 8200):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x003c7f61 in qpid::broker::Broker::run (this=0x9378a30) at qpid/broker/Broker.cpp:324
#6  0x0804dd44 in QpiddBroker::execute (this=0xbfd3b4a5, options=0x9375d88) at posix/QpiddBroker.cpp:166
#7  0x0804c6c7 in main (argc=23, argv=0xbfd3b554) at qpidd.cpp:77

Thread 11 (process 8205):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00251d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00adb414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x0049dc5e in qpid::broker::Timer::run (this=0x9378c38) at qpid/sys/posix/Condition.h:69
#4  0x00666871 in runRunnable (p=0x9378c38) at qpid/sys/posix/Thread.cpp:35
#5  0x0024d73b in start_thread () from /lib/libpthread.so.0
#6  0x00acecfe in clone () from /lib/libc.so.6

Thread 10 (process 8206):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00251d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00adb414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x0049dc5e in qpid::broker::Timer::run (this=0x9378cb8) at qpid/sys/posix/Condition.h:69
#4  0x00666871 in runRunnable (p=0x9378cb8) at qpid/sys/posix/Thread.cpp:35
#5  0x0024d73b in start_thread () from /lib/libpthread.so.0
#6  0x00acecfe in clone () from /lib/libc.so.6

Thread 9 (process 8207):
#0  0x00bf8805 in mrg::journal::wmgr::get_events () from /usr/lib/qpid/daemon/msgstore.so
#1  0x00bcd09e in mrg::journal::jcntl::get_wr_events () from /usr/lib/qpid/daemon/msgstore.so
#2  0x00b6f17d in mrg::msgstore::JournalImpl::getEventsFire () from /usr/lib/qpid/daemon/msgstore.so
#3  0x00b6f641 in mrg::msgstore::GetEventsFireEvent::fire () from /usr/lib/qpid/daemon/msgstore.so
---Type <return> to continue, or q <return> to quit---
#4  0x0049d9c2 in qpid::broker::Timer::run (this=0x936b000) at qpid/broker/Timer.cpp:79
#5  0x00666871 in runRunnable (p=0x936b000) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 8 (process 8208):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 7 (process 8209):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00254509 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0x0024fbbf in _L_lock_885 () from /lib/libpthread.so.0
#3  0x0024fa86 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x00adb536 in pthread_mutex_lock () from /lib/libc.so.6
#5  0x00bcbbc3 in mrg::journal::jcntl::dequeue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#6  0x00b6e491 in mrg::msgstore::JournalImpl::dequeue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#7  0x00b820be in mrg::msgstore::MessageStoreImpl::async_dequeue () from /usr/lib/qpid/daemon/msgstore.so
#8  0x00b9adda in mrg::msgstore::MessageStoreImpl::dequeue () from /usr/lib/qpid/daemon/msgstore.so
#9  0x0044ae7d in qpid::broker::MessageStoreModule::dequeue (this=0x936a980, ctxt=0x0, msg=@0xb561e6f4, queue=@0x93bf988)
    at qpid/broker/MessageStoreModule.cpp:131
#10 0x003e212d in qpid::broker::Queue::dequeue (this=0x93bf988, ctxt=0x0, msg=@0xb561e7a8) at qpid/broker/Queue.cpp:748
#11 0x0046bcb2 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0xb0d44148, msg=@0xb561e7a8)
    at qpid/broker/SemanticState.cpp:281
#12 0x003e3e03 in qpid::broker::Queue::dispatch (this=0x93bf988, c={px = 0xb561eaf8, pn = {pi_ = 0x0}})
    at qpid/broker/Queue.cpp:418
#13 0x00474426 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0xb0d44148)
    at qpid/broker/SemanticState.cpp:601
---Type <return> to continue, or q <return> to quit---
#14 0x006bed39 in qpid::sys::AggregateOutput::doOutput (this=0xb0d43d58) at qpid/sys/AggregateOutput.cpp:50
#15 0x00476f72 in qpid::broker::SemanticState::doOutput (this=0xb0d43ca8) at qpid/broker/SemanticState.h:213
#16 0x006bed39 in qpid::sys::AggregateOutput::doOutput (this=0xb0d42eb8) at qpid/sys/AggregateOutput.cpp:50
#17 0x003fa42d in qpid::broker::Connection::doOutput (this=0xb0d42e80) at qpid/broker/Connection.cpp:284
#18 0x003b89fe in qpid::amqp_0_10::Connection::canEncode (this=0xb0d023f0) at qpid/amqp_0_10/Connection.cpp:61
#19 0x0046adcb in qpid::broker::SecureConnection::canEncode (this=0xb0d01e88) at qpid/broker/SecureConnection.cpp:58
#20 0x006bfe59 in qpid::sys::AsynchIOHandler::idle (this=0xb0d01d98) at qpid/sys/AsynchIOHandler.cpp:202
#21 0x004c472d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&>, boost::_bi::list2<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1> > >, void, qpid::sys::AsynchIO&>::invoke (function_obj_ptr=
      {obj_ptr = 0xb0d02628, const_obj_ptr = 0xb0d02628, func_ptr = 0xb0d02628, data = "("}, a0=@0xb0d02508)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#22 0x006641d7 in boost::function1<void, qpid::sys::AsynchIO&, std::allocator<boost::function_base> >::operator() (
    this=0xb0d025c0, a0=@0xb0d02508) at /usr/include/boost/function/function_template.hpp:576
#23 0x0065fe33 in qpid::sys::posix::AsynchIO::writeable (this=0xb0d02508, h=@0xb0d0250c)
    at qpid/sys/posix/AsynchIO.cpp:562
#24 0x0066129d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0xb0d01f50, const_obj_ptr = 0xb0d01f50, func_ptr = 0xb0d01f50, data = "P"}, a0=@0xb0d0250c)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#25 0x006c4f14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0xb0d02520, a0=@0xb0d0250c) at /usr/include/boost/function/function_template.hpp:576
#26 0x006c45e2 in qpid::sys::DispatchHandle::processEvent (this=0xb0d0250c, type=qpid::sys::Poller::WRITABLE)
    at qpid/sys/DispatchHandle.cpp:439
#27 0x00671934 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/Poller.h:122
#28 0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#29 0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#30 0x0024d73b in start_thread () from /lib/libpthread.so.0
#31 0x00acecfe in clone () from /lib/libc.so.6

Thread 6 (process 8210):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
---Type <return> to continue, or q <return> to quit---
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 5 (process 8211):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 4 (process 8212):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 3 (process 8213):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00acf376 in epoll_wait () from /lib/libc.so.6
#2  0x006708e9 in qpid::sys::Poller::wait (this=0x9377dd8, timeout={nanosecs = 9223372036854775807})
    at qpid/sys/epoll/EpollPoller.cpp:439
#3  0x00671916 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/epoll/EpollPoller.cpp:405
---Type <return> to continue, or q <return> to quit---
#4  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#5  0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#6  0x0024d73b in start_thread () from /lib/libpthread.so.0
#7  0x00acecfe in clone () from /lib/libc.so.6

Thread 2 (process 8214):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00254509 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0x0024fbbf in _L_lock_885 () from /lib/libpthread.so.0
#3  0x0024fa86 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x00adb536 in pthread_mutex_lock () from /lib/libc.so.6
#5  0x00bcc463 in mrg::journal::jcntl::enqueue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#6  0x00b6ef6c in mrg::msgstore::JournalImpl::enqueue_data_record () from /usr/lib/qpid/daemon/msgstore.so
#7  0x00b8291a in mrg::msgstore::MessageStoreImpl::store () from /usr/lib/qpid/daemon/msgstore.so
#8  0x00b9ba52 in mrg::msgstore::MessageStoreImpl::enqueue () from /usr/lib/qpid/daemon/msgstore.so
#9  0x0044af8d in qpid::broker::MessageStoreModule::enqueue (this=0x936a980, ctxt=0x0, msg=@0xb2418a40, queue=@0x93bf988)
    at qpid/broker/MessageStoreModule.cpp:124
#10 0x003e40e1 in qpid::broker::Queue::enqueue (this=0x93bf988, ctxt=0x0, msg={p_ = 0xb2418d10},
    suppressPolicyCheck=false) at qpid/broker/Queue.cpp:717
#11 0x003e8908 in qpid::broker::Queue::deliver (this=0x93bf988, msg=@0xb24191d4) at qpid/broker/Queue.cpp:171
#12 0x00404dce in qpid::broker::DeliverableMessage::deliverTo (this=0xb24191cc, queue=@0x93c0924)
    at qpid/broker/DeliverableMessage.cpp:31
#13 0x003d3a1d in qpid::broker::Exchange::doRoute (this=0x936b18c, msg=@0xb24191cc, b=
        {px = 0xb2418df0, pn = {pi_ = 0x6ad845}}) at qpid/broker/Exchange.cpp:88
#14 0x00408e30 in qpid::broker::DirectExchange::route (this=0x936b18c, msg=@0xb24191cc, routingKey=@0xb2419118)
    at qpid/broker/DirectExchange.cpp:141
#15 0x004705b7 in qpid::broker::SemanticState::route (this=0x93d4ff0, msg={p_ = 0xb24191e8}, strategy=@0xb24191cc)
    at qpid/broker/SemanticState.cpp:386
#16 0x0047187a in qpid::broker::SemanticState::handle (this=0x93d4ff0, msg={p_ = 0xb2419328})
    at qpid/broker/SemanticState.cpp:349
#17 0x00495e73 in qpid::broker::SessionState::handleContent (this=0x93d4ec8, frame=@0xb2419aa8, id=@0xb241940c)
    at qpid/broker/SessionState.cpp:248
#18 0x00496a03 in qpid::broker::SessionState::handleIn (this=0x93d4ec8, frame=@0xb2419aa8)
    at qpid/broker/SessionState.cpp:326
#19 0x00496b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFram---Type <return> to continue, or q <return> to quit---
e&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x93d4fc8, t=@0xb2419aa8) at qpid/framing/Handler.h:67
#20 0x00692266 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x936cd48, f=@0xb2419aa8)
    at qpid/amqp_0_10/SessionHandler.cpp:93
#21 0x00496b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x936cd50, t=@0xb2419aa8) at qpid/framing/Handler.h:67
#22 0x003f6610 in qpid::broker::Connection::received (this=0x93bf788, frame=@0xb2419aa8) at qpid/framing/Handler.h:42
#23 0x003b8d0f in qpid::amqp_0_10::Connection::decode (this=0x93c0858, buffer=0x93ad370 "\a\003\004\f", size=65536)
    at qpid/amqp_0_10/Connection.cpp:55
#24 0x0046ad94 in qpid::broker::SecureConnection::decode (this=0x93c29c0, buffer=0x93ad370 "\a\003\004\f", size=128)
    at qpid/broker/SecureConnection.cpp:42
#25 0x006c0be6 in qpid::sys::AsynchIOHandler::readbuff (this=0x936cea0, buff=0x93c2668)
    at qpid/sys/AsynchIOHandler.cpp:113
#26 0x004c46f4 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (
    function_obj_ptr={obj_ptr = 0x936d170, const_obj_ptr = 0x936d170, func_ptr = 0x936d170, data = "p"}, a0=@0x936d050,
    a1=0x93c2668) at /usr/include/boost/bind/mem_fn_template.hpp:252
#27 0x0066282e in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (this=0x936d0cc, a0=@0x936d050, a1=0x93c2668)
    at /usr/include/boost/function/function_template.hpp:576
#28 0x0066071e in qpid::sys::posix::AsynchIO::readable (this=0x936d050, h=@0x936d054) at qpid/sys/posix/AsynchIO.cpp:448
#29 0x0066129d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr=
      {obj_ptr = 0x936bee8, const_obj_ptr = 0x936bee8, func_ptr = 0x936bee8, data = "�"}, a0=@0x936d054)
    at /usr/include/boost/bind/mem_fn_template.hpp:149
#30 0x006c4f14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (
    this=0x936d05c, a0=@0x936d054) at /usr/include/boost/function/function_template.hpp:576
#31 0x006c45f9 in qpid::sys::DispatchHandle::processEvent (this=0x936d054, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:432
#32 0x00671934 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/Poller.h:122
#33 0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#34 0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
---Type <return> to continue, or q <return> to quit---
#35 0x0024d73b in start_thread () from /lib/libpthread.so.0
#36 0x00acecfe in clone () from /lib/libc.so.6

Thread 1 (process 8215):
#0  0x00898410 in __kernel_vsyscall ()
#1  0x00a25df0 in raise () from /lib/libc.so.6
#2  0x00a27701 in abort () from /lib/libc.so.6
#3  0x001f94d0 in __gnu_cxx::__verbose_terminate_handler () from /usr/lib/libstdc++.so.6
#4  0x001f6f35 in ?? () from /usr/lib/libstdc++.so.6
#5  0x001f6f72 in std::terminate () from /usr/lib/libstdc++.so.6
#6  0x001f70aa in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x006c4566 in qpid::sys::DispatchHandle::processEvent (this=0x93d52fc, type=qpid::sys::Poller::READABLE)
    at qpid/sys/posix/Mutex.h:120
#8  0x00671934 in qpid::sys::Poller::run (this=0x9377dd8) at qpid/sys/Poller.h:122
#9  0x006c18d4 in qpid::sys::Dispatcher::run (this=0xbfd3b1b8) at qpid/sys/Dispatcher.cpp:37
#10 0x00666871 in runRunnable (p=0xbfd3b1b8) at qpid/sys/posix/Thread.cpp:35
#11 0x0024d73b in start_thread () from /lib/libpthread.so.0
#12 0x00acecfe in clone () from /lib/libc.so.6
(gdb) quit
root@mrg-qe-12:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# rpm -qa | grep -E '(qpidd|opena)'
openais-debuginfo-0.80.6-8.el5_4.1
qpidd-ssl-0.5.752581-34.el5
openais-0.80.6-8.el5_4.1
qpidd-acl-0.5.752581-34.el5
qpidd-cluster-0.5.752581-34.el5
qpidd-xml-0.5.752581-34.el5
qpidd-0.5.752581-34.el5
qpidd-devel-0.5.752581-34.el5
qpidd-rdma-0.5.752581-34.el5



Version-Release number of selected component (if applicable):
openais-debuginfo-0.80.6-8.el5_4.1
qpidd-ssl-0.5.752581-34.el5
openais-0.80.6-8.el5_4.1
qpidd-acl-0.5.752581-34.el5
qpidd-cluster-0.5.752581-34.el5
qpidd-xml-0.5.752581-34.el5
qpidd-0.5.752581-34.el5
qpidd-devel-0.5.752581-34.el5
qpidd-rdma-0.5.752581-34.el5


How reproducible:
very hard ~ 1% (corner cases)

Steps to Reproduce:
1. run qpid_test_qpidd-perftest_performance (which is running perftest against standalone qpidd with and w/o msgstore.so
2. wait for crash/abort
  
Actual results:
qpidd+store occasionly aborts

Expected results:
qpidd+store should not abort

Additional info:

Comment 1 Gordon Sim 2010-01-13 10:51:35 UTC

Andrew, this looks to me like the issue with DispatchHandles getting processed while being deleted(?). If you agree and we believe this is fixed on trunk lets mark modified for 1.3.

Comment 2 Frantisek Reznicek 2010-01-15 10:21:57 UTC

https://bugzilla.redhat.com/show_bug.cgi?id=555716#c1 shows details how the qpidd/perftest parameters are swept.

Comment 3 Andrew Stitcher 2010-01-15 16:43:28 UTC

I agree with Gordon - it looks like this is a bug that was already fixed for 1.3

Comment 4 Frantisek Reznicek 2010-06-14 08:32:42 UTC

The issue has been fixed (no aborts / crashes detected), tested in four extended week runs on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qpid-0.7.946106-1.el5
python-saslwrapper-0.1.934605-2.el5
qpid-cpp-client-0.7.946106-2.el5
qpid-cpp-client-devel-0.7.946106-2.el5
qpid-cpp-client-devel-docs-0.7.946106-2.el5
qpid-cpp-client-ssl-0.7.946106-2.el5
qpid-cpp-mrg-debuginfo-0.7.946106-2.el5
qpid-cpp-server-0.7.946106-2.el5
qpid-cpp-server-cluster-0.7.946106-2.el5
qpid-cpp-server-devel-0.7.946106-2.el5
qpid-cpp-server-ssl-0.7.946106-2.el5
qpid-cpp-server-store-0.7.946106-2.el5
qpid-cpp-server-xml-0.7.946106-2.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-4.el5
ruby-qpid-0.7.946106-2.el5
ruby-saslwrapper-0.1.934605-2.el5
saslwrapper-0.1.934605-2.el5
saslwrapper-devel-0.1.934605-2.el5

-> VERIFIED

Comment 5 Andrew Stitcher 2010-10-11 19:43:49 UTC

This bug seems to have been fixed as part of some other work and there is no information here about that other bug fix. I'm afraid there isn't enough information here to figure out real a release note.

Comment 6 Andrew Stitcher 2010-10-11 19:43:49 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Under rare conditions a broker with the persistence storage module could crash.

Comment 8 errata-xmlrpc 2010-10-14 16:04:26 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html