Description of problem: There is running long term qpidd+store stress test using perftest with swept parameters, also qpidd broker is run with swept parameters. In particular test submode when perftests with swept parameters are running against broker serially (max. one instance at the time), broker is not restarted. I luckily found broker crashing with segfault (one case only atm): root@mrg-qe-11:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# file core.26177 core.26177: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qpidd' root@mrg-qe-11:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# gdb `which qpidd` core.26177 GNU gdb Fedora (6.8-37.el5) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... warning: Can't read pathname for load map: Input/output error. warning: .dynamic section for "/lib/libuuid.so.1" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libssl3.so" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations warning: .dynamic section for "/usr/lib/libnssutil3.so" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib/libqpidbroker.so.0...Reading symbols from /usr/lib/debug/usr/lib/libqpidbroker.so.0.1.0.debug...done. ... done. Loaded symbols for /usr/lib/qpid/daemon/rdma.so Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'. Program terminated with signal 11, Segmentation fault. [New process 26193] [New process 26192] [New process 26191] [New process 26190] [New process 26189] [New process 26188] [New process 26187] [New process 26186] [New process 26185] [New process 26184] [New process 26183] [New process 26182] [New process 26177] #0 0x87cc9b38 in ?? () (gdb) info threads 13 process 26177 0x005c3410 in __kernel_vsyscall () 12 process 26182 0x005c3410 in __kernel_vsyscall () 11 process 26183 0x005c3410 in __kernel_vsyscall () 10 process 26184 0x005c3410 in __kernel_vsyscall () 9 process 26185 0x005c3410 in __kernel_vsyscall () 8 process 26186 0x005c3410 in __kernel_vsyscall () 7 process 26187 0x005c3410 in __kernel_vsyscall () 6 process 26188 0x005c3410 in __kernel_vsyscall () 5 process 26189 0x005c3410 in __kernel_vsyscall () 4 process 26190 0x005c3410 in __kernel_vsyscall () 3 process 26191 0x005c3410 in __kernel_vsyscall () 2 process 26192 0x005c3410 in __kernel_vsyscall () * 1 process 26193 0x87cc9b38 in ?? () (gdb) thread apply all bt Thread 13 (process 26177): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x006a1f61 in qpid::broker::Broker::run (this=0x8846ec0) at qpid/broker/Broker.cpp:324 #6 0x0804dd44 in QpiddBroker::execute (this=0xbf82e8b5, options=0x8844760) at posix/QpiddBroker.cpp:166 #7 0x0804c6c7 in main (argc=23, argv=0xbf82e964) at qpidd.cpp:77 Thread 12 (process 26182): #0 0x005c3410 in __kernel_vsyscall () #1 0x00150d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x009d6414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x00777c5e in qpid::broker::Timer::run (this=0xb7fb20b0) at qpid/sys/posix/Condition.h:69 #4 0x002d0871 in runRunnable (p=0xb7fb20b0) at qpid/sys/posix/Thread.cpp:35 #5 0x0014c73b in start_thread () from /lib/libpthread.so.0 #6 0x009c9cfe in clone () from /lib/libc.so.6 Thread 11 (process 26183): #0 0x005c3410 in __kernel_vsyscall () #1 0x00150d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x009d6414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x00777c5e in qpid::broker::Timer::run (this=0x88470c8) at qpid/sys/posix/Condition.h:69 #4 0x002d0871 in runRunnable (p=0x88470c8) at qpid/sys/posix/Thread.cpp:35 #5 0x0014c73b in start_thread () from /lib/libpthread.so.0 #6 0x009c9cfe in clone () from /lib/libc.so.6 Thread 10 (process 26184): #0 0x005c3410 in __kernel_vsyscall () #1 0x00150d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x009d6414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x00777c5e in qpid::broker::Timer::run (this=0x8847148) at qpid/sys/posix/Condition.h:69 ---Type <return> to continue, or q <return> to quit--- #4 0x002d0871 in runRunnable (p=0x8847148) at qpid/sys/posix/Thread.cpp:35 #5 0x0014c73b in start_thread () from /lib/libpthread.so.0 #6 0x009c9cfe in clone () from /lib/libc.so.6 Thread 9 (process 26185): #0 0x005c3410 in __kernel_vsyscall () #1 0x00150d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x009d6414 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x00777c5e in qpid::broker::Timer::run (this=0x883a3a8) at qpid/sys/posix/Condition.h:69 #4 0x002d0871 in runRunnable (p=0x883a3a8) at qpid/sys/posix/Thread.cpp:35 #5 0x0014c73b in start_thread () from /lib/libpthread.so.0 #6 0x009c9cfe in clone () from /lib/libc.so.6 Thread 8 (process 26186): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #6 0x0014c73b in start_thread () from /lib/libpthread.so.0 #7 0x009c9cfe in clone () from /lib/libc.so.6 Thread 7 (process 26187): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #6 0x0014c73b in start_thread () from /lib/libpthread.so.0 #7 0x009c9cfe in clone () from /lib/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 6 (process 26188): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ba95b in write () from /lib/libc.so.6 #2 0x0242f595 in mrg::journal::fcntl::clean_file () from /usr/lib/qpid/daemon/msgstore.so #3 0x02430524 in mrg::journal::fcntl::create_jfile () from /usr/lib/qpid/daemon/msgstore.so #4 0x024305f5 in mrg::journal::fcntl::initialize () from /usr/lib/qpid/daemon/msgstore.so #5 0x02430721 in mrg::journal::fcntl::fcntl () from /usr/lib/qpid/daemon/msgstore.so #6 0x02437d72 in mrg::journal::jcntl::new_fcntl () from /usr/lib/qpid/daemon/msgstore.so #7 0x0244c1ac in mrg::journal::lpmgr::append () from /usr/lib/qpid/daemon/msgstore.so #8 0x0244c881 in mrg::journal::lpmgr::initialize () from /usr/lib/qpid/daemon/msgstore.so #9 0x02437bd5 in mrg::journal::jcntl::initialize () from /usr/lib/qpid/daemon/msgstore.so #10 0x023dac0e in mrg::msgstore::JournalImpl::initialize () from /usr/lib/qpid/daemon/msgstore.so #11 0x0240305a in mrg::msgstore::MessageStoreImpl::create () from /usr/lib/qpid/daemon/msgstore.so #12 0x00724856 in qpid::broker::MessageStoreModule::create (this=0x88394a8, queue=@0xa531b2c8, args=@0xa7d3c89c) at qpid/broker/MessageStoreModule.cpp:51 #13 0x006b8ba9 in qpid::broker::Queue::create (this=0xa531b2c8, _settings=@0xa7d3c89c) at qpid/broker/Queue.cpp:793 #14 0x00767900 in qpid::broker::SessionAdapter::QueueHandlerImpl::declare (this=0xa531b170, name=@0xa7d3c894, alternateExchange=@0xa7d3c898, passive=false, durable=true, exclusive=true, autoDelete=true, arguments=@0xa7d3c89c) at qpid/broker/SessionAdapter.cpp:367 #15 0x00266b9f in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=0xb42ac250, body=@0xa7d3c888) at gen/qpid/framing/QueueDeclareBody.h:95 #16 0x0029fc2b in qpid::framing::QueueDeclareBody::accept (this=0xa7d3c888, v=@0xb42ac250) at gen/qpid/framing/QueueDeclareBody.h:99 #17 0x0026c2b6 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0xb42ac2ac, body=@0xa7d3c888) at gen/qpid/framing/ServerInvoker.cpp:368 #18 0x0029fc2b in qpid::framing::QueueDeclareBody::accept (this=0xa7d3c888, v=@0xb42ac2ac) at gen/qpid/framing/QueueDeclareBody.h:99 #19 0x00771227 in qpid::framing::invoke<qpid::broker::SessionAdapter> (target=@0xa531b158, body=@0xa7d3c888) at qpid/framing/Invoker.h:67 #20 0x007702b2 in qpid::broker::SessionState::handleCommand (this=0xa531af58, method=0xa7d3c888, id=@0xb42ac40c) at qpid/broker/SessionState.cpp:194 #21 0x00770a2e in qpid::broker::SessionState::handleIn (this=0xa531af58, frame=@0xb42acaa8) at qpid/broker/SessionState.cpp:328 #22 0x00770b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::frami---Type <return> to continue, or q <return> to quit--- ng::AMQFrame&))>::handle (this=0xa531b058, t=@0xb42acaa8) at qpid/framing/Handler.h:67 #23 0x002fc266 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xa531abd0, f=@0xb42acaa8) at qpid/amqp_0_10/SessionHandler.cpp:93 #24 0x00770b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xa531abd8, t=@0xb42acaa8) at qpid/framing/Handler.h:67 #25 0x006d0610 in qpid::broker::Connection::received (this=0xa531a810, frame=@0xb42acaa8) at qpid/framing/Handler.h:42 #26 0x00692d0f in qpid::amqp_0_10::Connection::decode (this=0xa5319cb0, buffer=0x8e870050 "\017", size=198) at qpid/amqp_0_10/Connection.cpp:55 #27 0x00744d94 in qpid::broker::SecureConnection::decode (this=0xa1db42c8, buffer=0x8e870050 "\017", size=1857024512) at qpid/broker/SecureConnection.cpp:42 #28 0x0032abe6 in qpid::sys::AsynchIOHandler::readbuff (this=0xa7d57a30, buff=0xa5327008) at qpid/sys/AsynchIOHandler.cpp:113 #29 0x0079e6f4 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke ( function_obj_ptr={obj_ptr = 0xa5326dd8, const_obj_ptr = 0xa5326dd8, func_ptr = 0xa5326dd8, data = "�"}, a0=@0xa5326cb8, a1=0xa5327008) at /usr/include/boost/bind/mem_fn_template.hpp:252 #30 0x002cc82e in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (this=0xa5326d34, a0=@0xa5326cb8, a1=0xa5327008) at /usr/include/boost/function/function_template.hpp:576 #31 0x002ca71e in qpid::sys::posix::AsynchIO::readable (this=0xa5326cb8, h=@0xa5326cbc) at qpid/sys/posix/AsynchIO.cpp:448 #32 0x002cb29d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr= {obj_ptr = 0xa5326e50, const_obj_ptr = 0xa5326e50, func_ptr = 0xa5326e50, data = "P"}, a0=@0xa5326cbc) at /usr/include/boost/bind/mem_fn_template.hpp:149 #33 0x0032ef14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ( this=0xa5326cc4, a0=@0xa5326cbc) at /usr/include/boost/function/function_template.hpp:576 #34 0x0032e5f9 in qpid::sys::DispatchHandle::processEvent (this=0xa5326cbc, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:432 #35 0x002db934 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/Poller.h:122 #36 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #37 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #38 0x0014c73b in start_thread () from /lib/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #39 0x009c9cfe in clone () from /lib/libc.so.6 Thread 5 (process 26189): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #6 0x0014c73b in start_thread () from /lib/libpthread.so.0 #7 0x009c9cfe in clone () from /lib/libc.so.6 Thread 4 (process 26190): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #6 0x0014c73b in start_thread () from /lib/libpthread.so.0 #7 0x009c9cfe in clone () from /lib/libc.so.6 Thread 3 (process 26191): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ca376 in epoll_wait () from /lib/libc.so.6 #2 0x002da8e9 in qpid::sys::Poller::wait (this=0x8846788, timeout={nanosecs = 9223372036854775807}) at qpid/sys/epoll/EpollPoller.cpp:439 #3 0x002db916 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/epoll/EpollPoller.cpp:405 #4 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #5 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #6 0x0014c73b in start_thread () from /lib/libpthread.so.0 #7 0x009c9cfe in clone () from /lib/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 2 (process 26192): #0 0x005c3410 in __kernel_vsyscall () #1 0x009ba95b in write () from /lib/libc.so.6 #2 0x0242f595 in mrg::journal::fcntl::clean_file () from /usr/lib/qpid/daemon/msgstore.so #3 0x02430524 in mrg::journal::fcntl::create_jfile () from /usr/lib/qpid/daemon/msgstore.so #4 0x024305f5 in mrg::journal::fcntl::initialize () from /usr/lib/qpid/daemon/msgstore.so #5 0x02430721 in mrg::journal::fcntl::fcntl () from /usr/lib/qpid/daemon/msgstore.so #6 0x02437d72 in mrg::journal::jcntl::new_fcntl () from /usr/lib/qpid/daemon/msgstore.so #7 0x0244c1ac in mrg::journal::lpmgr::append () from /usr/lib/qpid/daemon/msgstore.so #8 0x0244c881 in mrg::journal::lpmgr::initialize () from /usr/lib/qpid/daemon/msgstore.so #9 0x02437bd5 in mrg::journal::jcntl::initialize () from /usr/lib/qpid/daemon/msgstore.so #10 0x023dac0e in mrg::msgstore::JournalImpl::initialize () from /usr/lib/qpid/daemon/msgstore.so #11 0x0240305a in mrg::msgstore::MessageStoreImpl::create () from /usr/lib/qpid/daemon/msgstore.so #12 0x00724856 in qpid::broker::MessageStoreModule::create (this=0x88394a8, queue=@0x9f06d2a8, args=@0xa45a6c44) at qpid/broker/MessageStoreModule.cpp:51 #13 0x006b8ba9 in qpid::broker::Queue::create (this=0x9f06d2a8, _settings=@0xa45a6c44) at qpid/broker/Queue.cpp:793 #14 0x00767900 in qpid::broker::SessionAdapter::QueueHandlerImpl::declare (this=0xa333c798, name=@0xa45a6c3c, alternateExchange=@0xa45a6c40, passive=false, durable=true, exclusive=true, autoDelete=true, arguments=@0xa45a6c44) at qpid/broker/SessionAdapter.cpp:367 #15 0x00266b9f in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=0xb1aa8250, body=@0xa45a6c30) at gen/qpid/framing/QueueDeclareBody.h:95 #16 0x0029fc2b in qpid::framing::QueueDeclareBody::accept (this=0xa45a6c30, v=@0xb1aa8250) at gen/qpid/framing/QueueDeclareBody.h:99 #17 0x0026c2b6 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0xb1aa82ac, body=@0xa45a6c30) at gen/qpid/framing/ServerInvoker.cpp:368 #18 0x0029fc2b in qpid::framing::QueueDeclareBody::accept (this=0xa45a6c30, v=@0xb1aa82ac) at gen/qpid/framing/QueueDeclareBody.h:99 #19 0x00771227 in qpid::framing::invoke<qpid::broker::SessionAdapter> (target=@0xa333c780, body=@0xa45a6c30) at qpid/framing/Invoker.h:67 #20 0x007702b2 in qpid::broker::SessionState::handleCommand (this=0xa333c580, method=0xa45a6c30, id=@0xb1aa840c) at qpid/broker/SessionState.cpp:194 #21 0x00770a2e in qpid::broker::SessionState::handleIn (this=0xa333c580, frame=@0xb1aa8aa8) at qpid/broker/SessionState.cpp:328 #22 0x00770b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::frami---Type <return> to continue, or q <return> to quit--- ng::AMQFrame&))>::handle (this=0xa333c680, t=@0xb1aa8aa8) at qpid/framing/Handler.h:67 #23 0x002fc266 in qpid::amqp_0_10::SessionHandler::handleIn (this=0xb032ec58, f=@0xb1aa8aa8) at qpid/amqp_0_10/SessionHandler.cpp:93 #24 0x00770b0b in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0xb032ec60, t=@0xb1aa8aa8) at qpid/framing/Handler.h:67 #25 0x006d0610 in qpid::broker::Connection::received (this=0x7ff43a58, frame=@0xb1aa8aa8) at qpid/framing/Handler.h:42 #26 0x00692d0f in qpid::amqp_0_10::Connection::decode (this=0x866875c8, buffer=0x8e8b0070 "\017", size=198) at qpid/amqp_0_10/Connection.cpp:55 #27 0x00744d94 in qpid::broker::SecureConnection::decode (this=0x9cc41270, buffer=0x8e8b0070 "\017", size=239605248) at qpid/broker/SecureConnection.cpp:42 #28 0x0032abe6 in qpid::sys::AsynchIOHandler::readbuff (this=0xa7d8dbf0, buff=0xa6c21e50) at qpid/sys/AsynchIOHandler.cpp:113 #29 0x0079e6f4 in boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::_mfi::mf2<bool, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke ( function_obj_ptr={obj_ptr = 0xa531aaf0, const_obj_ptr = 0xa531aaf0, func_ptr = 0xa531aaf0, data = "�"}, a0=@0xa531a9d0, a1=0xa6c21e50) at /usr/include/boost/bind/mem_fn_template.hpp:252 #30 0x002cc82e in boost::function2<bool, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (this=0xa531aa4c, a0=@0xa531a9d0, a1=0xa6c21e50) at /usr/include/boost/function/function_template.hpp:576 #31 0x002ca71e in qpid::sys::posix::AsynchIO::readable (this=0xa531a9d0, h=@0xa531a9d4) at qpid/sys/posix/AsynchIO.cpp:448 #32 0x002cb29d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr= {obj_ptr = 0xa531ab68, const_obj_ptr = 0xa531ab68, func_ptr = 0xa531ab68, data = "h"}, a0=@0xa531a9d4) at /usr/include/boost/bind/mem_fn_template.hpp:149 #33 0x0032ef14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ( this=0xa531a9dc, a0=@0xa531a9d4) at /usr/include/boost/function/function_template.hpp:576 #34 0x0032e5f9 in qpid::sys::DispatchHandle::processEvent (this=0xa531a9d4, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:432 #35 0x002db934 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/Poller.h:122 #36 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #37 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #38 0x0014c73b in start_thread () from /lib/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #39 0x009c9cfe in clone () from /lib/libc.so.6 Thread 1 (process 26193): #0 0x87cc9b38 in ?? () #1 0x006d16d0 in qpid::broker::Connection::closed (this=0x90c106d0) at qpid/broker/Connection.cpp:253 #2 0x006914e4 in qpid::amqp_0_10::Connection::closed (this=0x90c10638) at qpid/amqp_0_10/Connection.cpp:117 #3 0x00744d54 in qpid::broker::SecureConnection::closed (this=0x90c10228) at qpid/broker/SecureConnection.cpp:63 #4 0x00329ac7 in ~AsynchIOHandler (this=0xb01a33f8) at qpid/sys/AsynchIOHandler.cpp:55 #5 0x0032a49f in qpid::sys::AsynchIOHandler::closedSocket (this=0xb01a33f8, s=@0xb0136208) at qpid/sys/AsynchIOHandler.cpp:182 #6 0x0079e774 in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::Socket const&>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::Socket const&>::invoke ( function_obj_ptr={obj_ptr = 0xa2256e68, const_obj_ptr = 0xa2256e68, func_ptr = 0xa2256e68, data = "h"}, a0=@0xb01a3a50, a1=@0xb0136208) at /usr/include/boost/bind/mem_fn_template.hpp:252 #7 0x002ccbae in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::Socket const&, std::allocator<boost::function_base> >::operator() (this=0xb01a3af0, a0=@0xb01a3a50, a1=@0xb0136208) at /usr/include/boost/function/function_template.hpp:576 #8 0x002c55d3 in qpid::sys::posix::AsynchIO::close (this=0xb01a3a50, h=@0xb01a3a54) at qpid/sys/posix/AsynchIO.cpp:599 #9 0x002c9ee2 in qpid::sys::posix::AsynchIO::writeable (this=0xb01a3a50, h=@0xb01a3a54) at qpid/sys/posix/AsynchIO.cpp:556 #10 0x002cb29d in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (function_obj_ptr= {obj_ptr = 0xb01a3478, const_obj_ptr = 0xb01a3478, func_ptr = 0xb01a3478, data = "x"}, a0=@0xb01a3a54) at /usr/include/boost/bind/mem_fn_template.hpp:149 #11 0x0032ef14 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() ( this=0xb01a3a68, a0=@0xb01a3a54) at /usr/include/boost/function/function_template.hpp:576 #12 0x0032e5e2 in qpid::sys::DispatchHandle::processEvent (this=0xb01a3a54, type=qpid::sys::Poller::WRITABLE) at qpid/sys/DispatchHandle.cpp:439 #13 0x002db934 in qpid::sys::Poller::run (this=0x8846788) at qpid/sys/Poller.h:122 #14 0x0032b8d4 in qpid::sys::Dispatcher::run (this=0xbf82e5c8) at qpid/sys/Dispatcher.cpp:37 #15 0x002d0871 in runRunnable (p=0xbf82e5c8) at qpid/sys/posix/Thread.cpp:35 #16 0x0014c73b in start_thread () from /lib/libpthread.so.0 #17 0x009c9cfe in clone () from /lib/libc.so.6 (gdb) quit Version-Release number of selected component (if applicable): root@mrg-qe-11:~/MRG/Messaging/qpid_test_qpidd-perftest_performance# rpm -qa | grep -E '(qpidd|opena)' | sort openais-0.80.6-8.el5_4.1 openais-debuginfo-0.80.6-8.el5_4.1 openais-devel-0.80.6-8.el5_4.1 qpidd-0.5.752581-34.el5 qpidd-acl-0.5.752581-34.el5 qpidd-cluster-0.5.752581-34.el5 qpidd-devel-0.5.752581-34.el5 qpidd-rdma-0.5.752581-34.el5 qpidd-ssl-0.5.752581-34.el5 qpidd-xml-0.5.752581-34.el5 [03:44:56] Machine and user info: [03:44:56] os: Red Hat Enterprise Linux Server release 5.4 (Tikanga) [03:44:56] who: root pts/0 2010-01-08 06:59 (dhcp-lab-111.englab.brq.redhat.com) [03:44:56] uname: Linux mrg-qe-11.lab.eng.brq.redhat.com 2.6.18-164.9.1.el5PAE #1 SMP Wed Dec 9 03:46:34 EST 2009 i686 athlon i386 GNU/Linux [03:44:56] uptime: 03:44:56 up 4 days, 19:27, 1 user, load average: 0.00, 0.00, 0.22 [03:44:56] whoami: root (USER:root, LOGNAME:root, id:root bin daemon sys adm disk wheel ais, HOME:/root) [03:44:56] date: 2010-01-11 03:44:56 1263199496 [03:44:56] pwd: /root/MRG/Messaging/qpid_test_qpidd-perftest_performance [03:44:56] df: /dev/mapper/VolGroup00-LogVol00 127G 19G 102G 16% / [03:44:56] get_cpu_info():CPU information: processor : 0 1 2 3 4 5 6 7 vendor_id : AuthenticAMD model name : Quad-Core AMD Opteron(tm) Processor 2376 cpu MHz : 2294.322 cpu cores : 4 bogomips : 4588.64 4588.50 4588.55 4588.54 4588.46 4588.52 4588.59 4588.48 [03:44:56] Memory info: total used free shared buffers cached Mem: 8312340 6057280 2255060 0 341368 5402728 -/+ buffers/cache: 313184 7999156 Swap: 5668856 0 5668856 How reproducible: extremely hard (one case per weeks) Steps to Reproduce: 1. run qpid_test_qpidd-perftest_performance (which is running perftest against standalone qpidd with and w/o msgstore.so 2. wait for crash Actual results: qpidd+store occasionly crashes Expected results: qpidd+store should not crash Additional info:
Additional info to reproducer, following snippet shows how are the qpidd|perftest parameters sweept (in qpid_test_qpidd-perftest_performance test): linked: http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_test_qpidd-perftest_performance/runtest.sh?rev=HEAD for ((i_loop=0; i_loop<${NR_OF_LOOPS}; i_loop++)); do # test start-up settings case $((${i_loop}%3)) in 0) # w/o management w store qpidd_test_params_run="${QPIDD_TEST_PARAMS} \ ${QPIDD_TEST_PARAMS_COMMON_JRNL} --mgmt-enable 0" ;; 1) # w management w store qpidd_test_params_run="${QPIDD_TEST_PARAMS} \ ${QPIDD_TEST_PARAMS_COMMON_JRNL} --mgmt-enable 1" ;; 2) # w/o management w/o store qpidd_test_params_run="${QPIDD_TEST_PARAMS} --mgmt-enable 0" # rename the msgstore if [ -e ${msgstore_fp} ]; then mv ${msgstore_fp} ${msgstore_fp}_ else lognl "WARNING: store module not found - skipping ${msgstore_fp} " \ "rename ${i_loop_p1}/${NR_OF_LOOPS}" fi ;; esac ... pt_mode_list="shared fanout topic" pt_qt_list="1 2" pt_durable_list="yes no" pt_npubs_list="1 2 3" pt_nsubs_list="1 2 3" pt_msg_count_list="200000 400000" pt_msg_size_list="128 1024" pt_tx_list="0 1 2" pt_tx_list="0" pt_ac_list="yes no" pt_ac_list="no" pt_iterations=1 pt_common="--iterations ${pt_iterations} --summary --unique-data yes" pt_common="${pt_common} --log-enable info+" # perftest mode loop for i_pt_mode in ${pt_mode_list}; do pt_mode="--mode ${i_pt_mode}" # modify qt in perftest fanout mode if [ "${i_pt_mode}" == "fanout" ]; then pt_qt_list="1" fi # qt switch loop for i_pt_qt in ${pt_qt_list} ; do pt_qt="--qt ${i_pt_qt}" # perftest durable loop for i_pt_durable in ${pt_durable_list}; do pt_durable="--durable ${i_pt_durable}" # perftest npubs loop for i_pt_npubs in ${pt_npubs_list}; do pt_npubs="--npubs ${i_pt_npubs}" # perftest nsubs loop for i_pt_nsubs in ${pt_nsubs_list}; do pt_nsubs="--nsubs ${i_pt_nsubs}" # perftest msg count loop for i_pt_msg_count in ${pt_msg_count_list}; do pt_msg_count="--count ${i_pt_msg_count}" # perftest msg size loop for i_pt_msg_size in ${pt_msg_size_list}; do pt_msg_size="--size ${i_pt_msg_size}" # perftest tx loop for i_pt_tx in ${pt_tx_list}; do pt_tx="--tx ${i_pt_tx}" # perftest async-commit loop for i_pt_ac in ${pt_ac_list}; do pt_ac="--async-commit ${i_pt_ac}" # randomly select the qpidd port - conditioned if [ "${i_loop}" -lt \ "${i_loop_thr_for_qpidd_keep_running}" ]; then mrg_gen_my_rand_in_range 40001 43590 QPIDD_PORT=${MY_RAND} fi # collect perftest parameters pt_params="${pt_common} -p ${QPIDD_PORT}" pt_params="${pt_params} ${pt_mode} ${pt_qt} ${pt_durable}" pt_params="${pt_params} ${pt_npubs} ${pt_nsubs}" pt_params="${pt_params} ${pt_msg_count} ${pt_msg_size}" pt_params="${pt_params} ${pt_tx} ${pt_ac}" ... ( /usr/bin/time -f "%e" -o ${TIME_TRANSCRIPT} \ perftest ${pt_params} >> ${PERFTEST_TRANSCRIPT} 2>&1; \ echo $? > ${TEMP_FILE} ) & ... done done done done done done done done done done
In the 1.2 source code it looks most likely that the qpid::broker::Connection timeoutTimer member is 0 causing a 0 dereference and a SIGSEGV. I think there's a reasonable chance that changes in the Connection class has fixed the bug on the trunk code line. So retesting for this bug would be very helpful.
The issue has been fixed (no aborts / crashes detected), tested in four extended week runs on RHEL 4.8 / 5.5 i386 / x86_64 on packages: python-qpid-0.7.946106-1.el5 python-saslwrapper-0.1.934605-2.el5 qpid-cpp-client-0.7.946106-2.el5 qpid-cpp-client-devel-0.7.946106-2.el5 qpid-cpp-client-devel-docs-0.7.946106-2.el5 qpid-cpp-client-ssl-0.7.946106-2.el5 qpid-cpp-mrg-debuginfo-0.7.946106-2.el5 qpid-cpp-server-0.7.946106-2.el5 qpid-cpp-server-cluster-0.7.946106-2.el5 qpid-cpp-server-devel-0.7.946106-2.el5 qpid-cpp-server-ssl-0.7.946106-2.el5 qpid-cpp-server-store-0.7.946106-2.el5 qpid-cpp-server-xml-0.7.946106-2.el5 qpid-java-client-0.7.946106-3.el5 qpid-java-common-0.7.946106-3.el5 qpid-tests-0.7.946106-1.el5 qpid-tools-0.7.946106-4.el5 ruby-qpid-0.7.946106-2.el5 ruby-saslwrapper-0.1.934605-2.el5 saslwrapper-0.1.934605-2.el5 saslwrapper-devel-0.1.934605-2.el5 -> VERIFIED
This bug seems to have been fixed as part of some other work and there is no information here about that other bug fix. I'm afraid there isn't enough information here to figure out a real release note.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Under rare conditions a broker with the persistence storage module could crash with a SIGSEGV signal.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0773.html