Created attachment 323348 [details] core file Description of problem: qpidd crashing while running perftest kernel RHEL5.2 GA Intel X86_64 #0 0x000000369d830155 in raise () from /lib64/libc.so.6 #1 0x000000369d831bf0 in abort () from /lib64/libc.so.6 #2 0x000000369d86a38b in __libc_message () from /lib64/libc.so.6 #3 0x000000369d871634 in _int_free () from /lib64/libc.so.6 #4 0x000000369d874c5c in free () from /lib64/libc.so.6 #5 0x00000036ab69dc1a in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string () from /usr/lib64/libstdc++.so.6 #6 0x00002b74b57c669b in qpid::Url::~Url () from /usr/lib64/libqpidbroker.so.0 #7 0x00002b74b57bcf1d in qpid::broker::Broker::getKnownBrokersImpl () from /usr/lib64/libqpidbroker.so.0 #8 0x00002b74b57c1b59 in boost::detail::function::function_obj_invoker0<boost::_bi::bind_t<std::vector<qpid::Url, std::allocator<qpid::Url> >, boost::_mfi::mf0<std::vector<qpid::Url, std::allocator<qpid::Url> >, qpid::broker::Broker>, boost::_bi::list1<boost::_bi::value<qpid::broker::Broker*> > >, std::vector<qpid::Url, std::allocator<qpid::Url> > >::invoke () from /usr/lib64/libqpidbroker.so.0 #9 0x00002b74b57eb6f7 in boost::function0<std::vector<qpid::Url, std::allocator<qpid::Url> >, std::allocator<void> >::operator() () from /usr/lib64/libqpidbroker.so.0 #10 0x00002b74b57ea057 in qpid::broker::ConnectionHandler::Handler::open () from /usr/lib64/libqpidbroker.so.0 #11 0x00002b74b5bd09a0 in qpid::framing::AMQP_ServerOperations::ConnectionHandler::Invoker::visit () from /usr/lib64/libqpidcommon.so.0 #12 0x00002b74b57eb8b4 in qpid::framing::invoke<qpid::framing::AMQP_ServerOperations::ConnectionHandler> () from /usr/lib64/libqpidbroker.so.0 #13 0x00002b74b57ea99f in qpid::broker::ConnectionHandler::handle () from /usr/lib64/libqpidbroker.so.0 #14 0x00002b74b57e4920 in qpid::broker::Connection::received () from /usr/lib64/libqpidbroker.so.0 #15 0x00002b74b57b7c36 in qpid::amqp_0_10::Connection::decode () from /usr/lib64/libqpidbroker.so.0 #16 0x00002b74b5c94513 in qpid::sys::AsynchIOHandler::readbuff () from /usr/lib64/libqpidcommon.so.0 #17 0x00002b74b5c43bba in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.0 #18 0x00002b74b5c40625 in qpid::sys::posix::AsynchIO::readable () from /usr/lib64/libqpidcommon.so.0 #19 0x00002b74b5c97219 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() () from /usr/lib64/libqpidcommon.so.0 #20 0x00002b74b5c955a4 in qpid::sys::DispatchHandle::processEvent () from /usr/lib64/libqpidcommon.so.0 #21 0x00002b74b5c9505e in qpid::sys::Dispatcher::run () from /usr/lib64/libqpidcommon.so.0 #22 0x00002b74b5c4690a in qpid::sys::AbsTime::AbsTime () from /usr/lib64/libqpidcommon.so.0 #23 0x000000369e4062f7 in start_thread () from /lib64/libpthread.so.0 #24 0x000000369d8d1b6d in clone () from /lib64/libc.so.6
Created attachment 323444 [details] test to open lots of connections concurrently Also in https://svn.devel.redhat.com/repos/mrg-team/trunk/people/gordon/connection_test
The key to this is having connections opened concurrently. If using perftest to do that you need to run it as separate process for each subscriber etc. Attached is a simple test case that just starts multiple threads opening and closing connections in a loop. I found it to reliably reproduce this and BZ468822. I also believe that the root cause is the same as BZ468821 though I didn't observe that exact stack trace.
*** Bug 468822 has been marked as a duplicate of this bug. ***
Fixed by r713714.
*** Bug 468821 has been marked as a duplicate of this bug. ***
Manual testing & qpid_test_broker_perftest_ecte_bz468428 test part B validate that issue has been fixed. Validated on RHEL 4.7 / 5.2, i386 / x86_64 using packages: qpidd-0.3.719671-1.el5, rhm-0.3.2804-4.el5 ->VERIFIED
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-0035.html