Description of problem: While running stress test on the latest qpid-cpp build, qpidd crash occurred on RHEL6.3 x86_64: Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50 50 unsigned int type = PTHREAD_MUTEX_TYPE (mutex) [root@tyan-gt24-04 qpid_ptest_stress]# ll -h ./core.9929 -rw-------. 1 root root 638M Aug 22 05:57 ./core.9929 [root@tyan-gt24-04 qpid_ptest_stress]# gdb /usr/sbin/qpidd ./core.9929 ....... Core was generated by `qpidd --auth=no -p 5672 --data-dir=/mnt/tests/distribution/MRG/Messaging/qpid_p'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50 50 unsigned int type = PTHREAD_MUTEX_TYPE (mutex); (gdb) info threads 10 Thread 0x7f096bfff700 (LWP 9943) release (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143 9 Thread 0x7f0979d70700 (LWP 9941) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 8 Thread 0x7f09805f57a0 (LWP 9929) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 7 Thread 0x7f097b172700 (LWP 9939) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 6 Thread 0x7f096abfd700 (LWP 9945) 0x00007f09803b3bf7 in get_fcntlp (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/lpmgr.hpp:276 5 Thread 0x7f097bd2a700 (LWP 9930) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216 4 Thread 0x7f096a1fc700 (LWP 9946) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 3 Thread 0x7f096b5fe700 (LWP 9944) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 2 Thread 0x7f097936f700 (LWP 9942) 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 * 1 Thread 0x7f097a771700 (LWP 9940) __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50 (gdb) thread apply all bt Thread 10 (Thread 0x7f096bfff700 (LWP 9943)): #0 release (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143 #1 boost::detail::shared_count::~shared_count (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217 #2 0x00000030675760c0 in ~shared_ptr (this=0xb86a10, msg=..., routingKey=<value optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169 #3 qpid::broker::DirectExchange::route (this=0xb86a10, msg=..., routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:164 #4 0x00000030675fd5b2 in qpid::broker::SemanticState::route (this=0x2c53fa8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:495 #5 0x00000030675fdfa1 in qpid::broker::SemanticState::handle (this=0x2c53fa8, msg=...) at qpid/broker/SemanticState.cpp:449 #6 0x000000306761b253 in qpid::broker::SessionState::handleContent (this=0x2c53dd0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:266 #7 0x000000306761b8d7 in qpid::broker::SessionState::handleIn (this=0x2c53dd0, frame=...) at qpid/broker/SessionState.cpp:362 #8 0x00000030641d5f19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x1af03f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #9 0x000000306756ebf3 in operator() (this=0x1aafe68, frame=...) at qpid/framing/Handler.h:42 #10 qpid::broker::ConnectionHandler::handle (this=0x1aafe68, frame=...) at qpid/broker/ConnectionHandler.cpp:87 #11 0x0000003067565cc8 in qpid::broker::Connection::received (this=0x1aafc80, frame=...) at qpid/broker/Connection.cpp:174 #12 0x000000306753c96d in qpid::amqp_0_10::Connection::decode (this=0x1ecd880, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #13 0x0000003064205d7d in qpid::sys::AsynchIOHandler::readbuff (this=0x7f09591cf230, buff=0x7f09592f0340) at qpid/sys/AsynchIOHandler.cpp:164 #14 0x000000306413fc43 in operator() (this=0x7f09581480d0, h=...) at /usr/include/boost/function/function_template.hpp:1013 #15 qpid::sys::posix::AsynchIO::readable (this=0x7f09581480d0, h=...) at qpid/sys/posix/AsynchIO.cpp:446 #16 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #17 0x0000003064208dd6 in qpid::sys::DispatchHandle::processEvent (this=0x7f09581480d8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #18 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131 #19 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524 #20 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #21 0x0000003063007851 in start_thread (arg=0x7f096bfff700) at pthread_create.c:301 #22 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 9 (Thread 0x7f0979d70700 (LWP 9941)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003063007851 in start_thread (arg=0x7f0979d70700) at pthread_create.c:301 #5 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 8 (Thread 0x7f09805f57a0 (LWP 9929)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x0000003067552e02 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:405 #4 0x000000000040dcb2 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:195 #5 0x000000000040a32a in run_broker (argc=8, argv=0x7fff305a7898, hidden=<value optimized out>) at qpidd.cpp:83 #6 0x0000003062c1ecdd in __libc_start_main (main=0x40d6c0 <main(int, char**)>, argc=8, ubp_av=0x7fff305a7898, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff305a7888) at libc-start.c:226 #7 0x0000000000409f79 in _start () ---Type <return> to continue, or q <return> to quit--- Thread 7 (Thread 0x7f097b172700 (LWP 9939)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003063007851 in start_thread (arg=0x7f097b172700) at pthread_create.c:301 #5 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 6 (Thread 0x7f096abfd700 (LWP 9945)): #0 0x00007f09803b3bf7 in get_fcntlp (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/lpmgr.hpp:276 #1 decr_enqcnt (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/rfc.hpp:166 #2 mrg::journal::wmgr::dequeue (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/wmgr.cpp:317 #3 0x00007f098038afea in mrg::journal::jcntl::dequeue_data_record (this=0x7f0974a9e2f8, dtokp=0x7f0955e53250, txn_coml_commit=false) at jrnl/jcntl.cpp:308 #4 0x00007f098034a4e3 in mrg::msgstore::JournalImpl::dequeue_data_record (this=0x7f0974a9e2f0, dtokp=<value optimized out>, txn_coml_commit=<value optimized out>) at JournalImpl.cpp:413 #5 0x00007f098035aa01 in mrg::msgstore::MessageStoreImpl::async_dequeue (this=0xb7ef10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1413 #6 0x00007f098036829c in mrg::msgstore::MessageStoreImpl::dequeue (this=0xb7ef10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1388 #7 0x00000030675bc29b in qpid::broker::MessageStoreModule::dequeue (this=<value optimized out>, ctxt=<value optimized out>, msg=<value optimized out>, queue=<value optimized out>) at qpid/broker/MessageStoreModule.cpp:132 #8 0x00000030675c7407 in qpid::broker::Queue::dequeue (this=0x7f09748dd220, ctxt=0x0, msg=...) at qpid/broker/Queue.cpp:850 #9 0x00000030675fb320 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x7f0959300690, msg=...) at qpid/broker/SemanticState.cpp:347 #10 0x00000030675cc4d8 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at qpid/broker/Queue.cpp:394 #11 0x00000030675fd1f5 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x7f0959300690) at qpid/broker/SemanticState.cpp:741 #12 0x0000003064202ac2 in qpid::sys::AggregateOutput::doOutput (this=0x7f095925b738) at qpid/sys/AggregateOutput.cpp:59 #13 0x0000003067567079 in qpid::broker::Connection::doOutput (this=0x7f095925b6e0) at qpid/broker/Connection.cpp:390 #14 0x000000306753d4d2 in qpid::amqp_0_10::Connection::canEncode (this=0x7f09593d1930) at qpid/amqp_0_10/Connection.cpp:67 #15 0x000000306420593b in qpid::sys::AsynchIOHandler::idle (this=0x1ec4620) at qpid/sys/AsynchIOHandler.cpp:250 #16 0x00000030641400dc in operator() (this=0x1ff0710, h=...) at /usr/include/boost/function/function_template.hpp:1013 #17 qpid::sys::posix::AsynchIO::writeable (this=0x1ff0710, h=...) at qpid/sys/posix/AsynchIO.cpp:564 #18 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #19 0x0000003064208a9e in qpid::sys::DispatchHandle::processEvent (this=0x1ff0718, type=qpid::sys::Poller::WRITABLE) at qpid/sys/DispatchHandle.cpp:287 #20 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131 #21 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524 #22 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #23 0x0000003063007851 in start_thread (arg=0x7f096abfd700) at pthread_create.c:301 #24 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 5 (Thread 0x7f097bd2a700 (LWP 9930)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216 #1 0x0000003064210d32 in wait (this=0xb6bfc0) at ../include/qpid/sys/posix/Condition.h:69 #2 wait (this=0xb6bfc0) at ../include/qpid/sys/Monitor.h:45 #3 qpid::sys::Timer::run (this=0xb6bfc0) at qpid/sys/Timer.cpp:159 #4 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #5 0x0000003063007851 in start_thread (arg=0x7f097bd2a700) at pthread_create.c:301 ---Type <return> to continue, or q <return> to quit--- #6 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 4 (Thread 0x7f096a1fc700 (LWP 9946)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003063007851 in start_thread (arg=0x7f096a1fc700) at pthread_create.c:301 #5 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 3 (Thread 0x7f096b5fe700 (LWP 9944)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003063007851 in start_thread (arg=0x7f096b5fe700) at pthread_create.c:301 #5 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 2 (Thread 0x7f097936f700 (LWP 9942)): #0 0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568 #2 0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520 #3 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003063007851 in start_thread (arg=0x7f097936f700) at pthread_create.c:301 #5 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Thread 1 (Thread 0x7f097a771700 (LWP 9940)): #0 __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50 #1 0x00000030675416ca in qpid::sys::Mutex::lock (this=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:116 #2 0x00000030675b3080 in ScopedLock (this=0x0) at ../include/qpid/sys/Mutex.h:33 #3 qpid::broker::Message::getApplicationHeaders (this=0x0) at qpid/broker/Message.cpp:99 #4 0x00000030675ba12c in qpid::broker::MessageMap::getKey (this=0x7f09742e02c0, message=<value optimized out>) at qpid/broker/MessageMap.cpp:32 #5 0x00000030675ba4b8 in qpid::broker::MessageMap::erase (this=0x7f09742e02c0, i=...) at qpid/broker/MessageMap.cpp:160 #6 0x00000030675ba5de in qpid::broker::MessageMap::removeIf (this=0x7f09742e02c0, p=...) at qpid/broker/MessageMap.cpp:153 #7 0x00000030675a032c in qpid::broker::LegacyLVQ::removeIf (this=0x7f09742e02c0, p=<value optimized out>) at qpid/broker/LegacyLVQ.cpp:97 #8 0x00000030675ca946 in qpid::broker::Queue::purge (this=0x7f09741bce40, purge_request=<value optimized out>, dest=..., filter=<value optimized out>) at qpid/broker/Queue.cpp:630 #9 0x000000306760db26 in qpid::broker::SessionAdapter::QueueHandlerImpl::purge (this=0x1fea9c0, queue="TEST.SENDER.LV.5") at qpid/broker/SessionAdapter.cpp:342 #10 0x00000030641b0423 in invoke<qpid::framing::AMQP_ServerOperations::QueueHandler> (this=0x7f097a76efc0, body=<value optimized out>) at qpid/framing/QueuePurgeBody.h:63 #11 qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=0x7f097a76efc0, body=<value optimized out>) at qpid/framing/ServerInvoker.cpp:662 #12 0x00000030641b30d1 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7f097a76f1a0, body=...) at qpid/framing/ServerInvoker.cpp:378 #13 0x00000030676179b8 in invoke<qpid::broker::SessionAdapter> (this=0x1fea660, method=0x7f096c21c660, id=...) at qpid/framing/Invoker.h:67 #14 qpid::broker::SessionState::handleCommand (this=0x1fea660, method=0x7f096c21c660, id=...) at qpid/broker/SessionState.cpp:210 #15 0x000000306761b9c1 in qpid::broker::SessionState::handleIn (this=0x1fea660, frame=...) at qpid/broker/SessionState.cpp:364 #16 0x00000030641d5f19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x29d38f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #17 0x000000306756ebf3 in operator() (this=0x2857378, frame=...) at qpid/framing/Handler.h:42 ---Type <return> to continue, or q <return> to quit--- #18 qpid::broker::ConnectionHandler::handle (this=0x2857378, frame=...) at qpid/broker/ConnectionHandler.cpp:87 #19 0x0000003067565cc8 in qpid::broker::Connection::received (this=0x2857190, frame=...) at qpid/broker/Connection.cpp:174 #20 0x000000306753c96d in qpid::amqp_0_10::Connection::decode (this=0x1fefa60, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #21 0x0000003064205d7d in qpid::sys::AsynchIOHandler::readbuff (this=0x1a529b0, buff=0xc172b0) at qpid/sys/AsynchIOHandler.cpp:164 #22 0x000000306413fc43 in operator() (this=0x1ef8310, h=...) at /usr/include/boost/function/function_template.hpp:1013 #23 qpid::sys::posix::AsynchIO::readable (this=0x1ef8310, h=...) at qpid/sys/posix/AsynchIO.cpp:446 #24 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #25 0x0000003064208dd6 in qpid::sys::DispatchHandle::processEvent (this=0x1ef8318, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #26 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131 #27 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524 #28 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #29 0x0000003063007851 in start_thread (arg=0x7f097a771700) at pthread_create.c:301 #30 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) Version-Release number of selected component (if applicable): qpid-cpp-server-0.14-21.el6_3.x86_64
Closing as a duplicate. Will reopen if this turns out to be an issue distinct from bug 849654. *** This bug has been marked as a duplicate of bug 849654 ***