Bug 851087 - qpidd crash __pthread_mutex_lock (mutex=0x100) under stress test
Summary: qpidd crash __pthread_mutex_lock (mutex=0x100) under stress test
Keywords:
Status: CLOSED DUPLICATE of bug 849654
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 2.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 2.2
: ---
Assignee: Andrew Stitcher
QA Contact: Leonid Zhaldybin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-23 08:38 UTC by Leonid Zhaldybin
Modified: 2014-11-09 22:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-27 14:13:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Leonid Zhaldybin 2012-08-23 08:38:15 UTC
Description of problem:

While running stress test on the latest qpid-cpp build, qpidd crash occurred on RHEL6.3 x86_64:
Program terminated with signal 11, Segmentation fault.
#0  __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50
50        unsigned int type = PTHREAD_MUTEX_TYPE (mutex)

[root@tyan-gt24-04 qpid_ptest_stress]# ll -h ./core.9929 
-rw-------. 1 root root 638M Aug 22 05:57 ./core.9929
[root@tyan-gt24-04 qpid_ptest_stress]# gdb /usr/sbin/qpidd ./core.9929
.......
Core was generated by `qpidd --auth=no -p 5672 --data-dir=/mnt/tests/distribution/MRG/Messaging/qpid_p'.
Program terminated with signal 11, Segmentation fault.
#0  __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50
50        unsigned int type = PTHREAD_MUTEX_TYPE (mutex);
(gdb) info threads
  10 Thread 0x7f096bfff700 (LWP 9943)  release (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143
  9 Thread 0x7f0979d70700 (LWP 9941)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  8 Thread 0x7f09805f57a0 (LWP 9929)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  7 Thread 0x7f097b172700 (LWP 9939)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  6 Thread 0x7f096abfd700 (LWP 9945)  0x00007f09803b3bf7 in get_fcntlp (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>)
    at jrnl/lpmgr.hpp:276
  5 Thread 0x7f097bd2a700 (LWP 9930)  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
  4 Thread 0x7f096a1fc700 (LWP 9946)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  3 Thread 0x7f096b5fe700 (LWP 9944)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
  2 Thread 0x7f097936f700 (LWP 9942)  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
* 1 Thread 0x7f097a771700 (LWP 9940)  __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50
(gdb) thread apply all bt

Thread 10 (Thread 0x7f096bfff700 (LWP 9943)):
#0  release (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:143
#1  boost::detail::shared_count::~shared_count (this=0x7f096bffc558, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#2  0x00000030675760c0 in ~shared_ptr (this=0xb86a10, msg=..., routingKey=<value optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
#3  qpid::broker::DirectExchange::route (this=0xb86a10, msg=..., routingKey=<value optimized out>) at qpid/broker/DirectExchange.cpp:164
#4  0x00000030675fd5b2 in qpid::broker::SemanticState::route (this=0x2c53fa8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:495
#5  0x00000030675fdfa1 in qpid::broker::SemanticState::handle (this=0x2c53fa8, msg=...) at qpid/broker/SemanticState.cpp:449
#6  0x000000306761b253 in qpid::broker::SessionState::handleContent (this=0x2c53dd0, frame=..., id=<value optimized out>) at qpid/broker/SessionState.cpp:266
#7  0x000000306761b8d7 in qpid::broker::SessionState::handleIn (this=0x2c53dd0, frame=...) at qpid/broker/SessionState.cpp:362
#8  0x00000030641d5f19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x1af03f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#9  0x000000306756ebf3 in operator() (this=0x1aafe68, frame=...) at qpid/framing/Handler.h:42
#10 qpid::broker::ConnectionHandler::handle (this=0x1aafe68, frame=...) at qpid/broker/ConnectionHandler.cpp:87
#11 0x0000003067565cc8 in qpid::broker::Connection::received (this=0x1aafc80, frame=...) at qpid/broker/Connection.cpp:174
#12 0x000000306753c96d in qpid::amqp_0_10::Connection::decode (this=0x1ecd880, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#13 0x0000003064205d7d in qpid::sys::AsynchIOHandler::readbuff (this=0x7f09591cf230, buff=0x7f09592f0340) at qpid/sys/AsynchIOHandler.cpp:164
#14 0x000000306413fc43 in operator() (this=0x7f09581480d0, h=...) at /usr/include/boost/function/function_template.hpp:1013
#15 qpid::sys::posix::AsynchIO::readable (this=0x7f09581480d0, h=...) at qpid/sys/posix/AsynchIO.cpp:446
#16 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#17 0x0000003064208dd6 in qpid::sys::DispatchHandle::processEvent (this=0x7f09581480d8, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#18 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131
#19 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524
#20 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#21 0x0000003063007851 in start_thread (arg=0x7f096bfff700) at pthread_create.c:301
#22 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7f0979d70700 (LWP 9941)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003063007851 in start_thread (arg=0x7f0979d70700) at pthread_create.c:301
#5  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 8 (Thread 0x7f09805f57a0 (LWP 9929)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x0000003067552e02 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:405
#4  0x000000000040dcb2 in QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:195
#5  0x000000000040a32a in run_broker (argc=8, argv=0x7fff305a7898, hidden=<value optimized out>) at qpidd.cpp:83
#6  0x0000003062c1ecdd in __libc_start_main (main=0x40d6c0 <main(int, char**)>, argc=8, ubp_av=0x7fff305a7898, init=<value optimized out>, fini=<value optimized out>, 
    rtld_fini=<value optimized out>, stack_end=0x7fff305a7888) at libc-start.c:226
#7  0x0000000000409f79 in _start ()
---Type <return> to continue, or q <return> to quit---

Thread 7 (Thread 0x7f097b172700 (LWP 9939)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003063007851 in start_thread (arg=0x7f097b172700) at pthread_create.c:301
#5  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 6 (Thread 0x7f096abfd700 (LWP 9945)):
#0  0x00007f09803b3bf7 in get_fcntlp (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/lpmgr.hpp:276
#1  decr_enqcnt (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/rfc.hpp:166
#2  mrg::journal::wmgr::dequeue (this=0x7f0974a9e680, dtokp=0x7f0955e53250, xid_ptr=0x0, xid_len=0, txn_coml_commit=<value optimized out>) at jrnl/wmgr.cpp:317
#3  0x00007f098038afea in mrg::journal::jcntl::dequeue_data_record (this=0x7f0974a9e2f8, dtokp=0x7f0955e53250, txn_coml_commit=false) at jrnl/jcntl.cpp:308
#4  0x00007f098034a4e3 in mrg::msgstore::JournalImpl::dequeue_data_record (this=0x7f0974a9e2f0, dtokp=<value optimized out>, txn_coml_commit=<value optimized out>) at JournalImpl.cpp:413
#5  0x00007f098035aa01 in mrg::msgstore::MessageStoreImpl::async_dequeue (this=0xb7ef10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1413
#6  0x00007f098036829c in mrg::msgstore::MessageStoreImpl::dequeue (this=0xb7ef10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1388
#7  0x00000030675bc29b in qpid::broker::MessageStoreModule::dequeue (this=<value optimized out>, ctxt=<value optimized out>, msg=<value optimized out>, queue=<value optimized out>)
    at qpid/broker/MessageStoreModule.cpp:132
#8  0x00000030675c7407 in qpid::broker::Queue::dequeue (this=0x7f09748dd220, ctxt=0x0, msg=...) at qpid/broker/Queue.cpp:850
#9  0x00000030675fb320 in qpid::broker::SemanticState::ConsumerImpl::deliver (this=0x7f0959300690, msg=...) at qpid/broker/SemanticState.cpp:347
#10 0x00000030675cc4d8 in qpid::broker::Queue::dispatch (this=<value optimized out>, c=...) at qpid/broker/Queue.cpp:394
#11 0x00000030675fd1f5 in qpid::broker::SemanticState::ConsumerImpl::doOutput (this=0x7f0959300690) at qpid/broker/SemanticState.cpp:741
#12 0x0000003064202ac2 in qpid::sys::AggregateOutput::doOutput (this=0x7f095925b738) at qpid/sys/AggregateOutput.cpp:59
#13 0x0000003067567079 in qpid::broker::Connection::doOutput (this=0x7f095925b6e0) at qpid/broker/Connection.cpp:390
#14 0x000000306753d4d2 in qpid::amqp_0_10::Connection::canEncode (this=0x7f09593d1930) at qpid/amqp_0_10/Connection.cpp:67
#15 0x000000306420593b in qpid::sys::AsynchIOHandler::idle (this=0x1ec4620) at qpid/sys/AsynchIOHandler.cpp:250
#16 0x00000030641400dc in operator() (this=0x1ff0710, h=...) at /usr/include/boost/function/function_template.hpp:1013
#17 qpid::sys::posix::AsynchIO::writeable (this=0x1ff0710, h=...) at qpid/sys/posix/AsynchIO.cpp:564
#18 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#19 0x0000003064208a9e in qpid::sys::DispatchHandle::processEvent (this=0x1ff0718, type=qpid::sys::Poller::WRITABLE) at qpid/sys/DispatchHandle.cpp:287
#20 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131
#21 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524
#22 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#23 0x0000003063007851 in start_thread (arg=0x7f096abfd700) at pthread_create.c:301
#24 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 5 (Thread 0x7f097bd2a700 (LWP 9930)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
#1  0x0000003064210d32 in wait (this=0xb6bfc0) at ../include/qpid/sys/posix/Condition.h:69
#2  wait (this=0xb6bfc0) at ../include/qpid/sys/Monitor.h:45
#3  qpid::sys::Timer::run (this=0xb6bfc0) at qpid/sys/Timer.cpp:159
#4  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x0000003063007851 in start_thread (arg=0x7f097bd2a700) at pthread_create.c:301
---Type <return> to continue, or q <return> to quit---
#6  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 4 (Thread 0x7f096a1fc700 (LWP 9946)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003063007851 in start_thread (arg=0x7f096a1fc700) at pthread_create.c:301
#5  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 3 (Thread 0x7f096b5fe700 (LWP 9944)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003063007851 in start_thread (arg=0x7f096b5fe700) at pthread_create.c:301
#5  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f097936f700 (LWP 9942)):
#0  0x0000003062ce7c73 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1  0x000000306414b91a in qpid::sys::Poller::wait (this=0xab0d20, timeout=...) at qpid/sys/epoll/EpollPoller.cpp:568
#2  0x000000306414be77 in qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:520
#3  0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003063007851 in start_thread (arg=0x7f097936f700) at pthread_create.c:301
#5  0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f097a771700 (LWP 9940)):
#0  __pthread_mutex_lock (mutex=0x100) at pthread_mutex_lock.c:50
#1  0x00000030675416ca in qpid::sys::Mutex::lock (this=<value optimized out>) at ../include/qpid/sys/posix/Mutex.h:116
#2  0x00000030675b3080 in ScopedLock (this=0x0) at ../include/qpid/sys/Mutex.h:33
#3  qpid::broker::Message::getApplicationHeaders (this=0x0) at qpid/broker/Message.cpp:99
#4  0x00000030675ba12c in qpid::broker::MessageMap::getKey (this=0x7f09742e02c0, message=<value optimized out>) at qpid/broker/MessageMap.cpp:32
#5  0x00000030675ba4b8 in qpid::broker::MessageMap::erase (this=0x7f09742e02c0, i=...) at qpid/broker/MessageMap.cpp:160
#6  0x00000030675ba5de in qpid::broker::MessageMap::removeIf (this=0x7f09742e02c0, p=...) at qpid/broker/MessageMap.cpp:153
#7  0x00000030675a032c in qpid::broker::LegacyLVQ::removeIf (this=0x7f09742e02c0, p=<value optimized out>) at qpid/broker/LegacyLVQ.cpp:97
#8  0x00000030675ca946 in qpid::broker::Queue::purge (this=0x7f09741bce40, purge_request=<value optimized out>, dest=..., filter=<value optimized out>) at qpid/broker/Queue.cpp:630
#9  0x000000306760db26 in qpid::broker::SessionAdapter::QueueHandlerImpl::purge (this=0x1fea9c0, queue="TEST.SENDER.LV.5") at qpid/broker/SessionAdapter.cpp:342
#10 0x00000030641b0423 in invoke<qpid::framing::AMQP_ServerOperations::QueueHandler> (this=0x7f097a76efc0, body=<value optimized out>) at qpid/framing/QueuePurgeBody.h:63
#11 qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=0x7f097a76efc0, body=<value optimized out>) at qpid/framing/ServerInvoker.cpp:662
#12 0x00000030641b30d1 in qpid::framing::AMQP_ServerOperations::Invoker::visit (this=0x7f097a76f1a0, body=...) at qpid/framing/ServerInvoker.cpp:378
#13 0x00000030676179b8 in invoke<qpid::broker::SessionAdapter> (this=0x1fea660, method=0x7f096c21c660, id=...) at qpid/framing/Invoker.h:67
#14 qpid::broker::SessionState::handleCommand (this=0x1fea660, method=0x7f096c21c660, id=...) at qpid/broker/SessionState.cpp:210
#15 0x000000306761b9c1 in qpid::broker::SessionState::handleIn (this=0x1fea660, frame=...) at qpid/broker/SessionState.cpp:364
#16 0x00000030641d5f19 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x29d38f0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#17 0x000000306756ebf3 in operator() (this=0x2857378, frame=...) at qpid/framing/Handler.h:42
---Type <return> to continue, or q <return> to quit---
#18 qpid::broker::ConnectionHandler::handle (this=0x2857378, frame=...) at qpid/broker/ConnectionHandler.cpp:87
#19 0x0000003067565cc8 in qpid::broker::Connection::received (this=0x2857190, frame=...) at qpid/broker/Connection.cpp:174
#20 0x000000306753c96d in qpid::amqp_0_10::Connection::decode (this=0x1fefa60, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#21 0x0000003064205d7d in qpid::sys::AsynchIOHandler::readbuff (this=0x1a529b0, buff=0xc172b0) at qpid/sys/AsynchIOHandler.cpp:164
#22 0x000000306413fc43 in operator() (this=0x1ef8310, h=...) at /usr/include/boost/function/function_template.hpp:1013
#23 qpid::sys::posix::AsynchIO::readable (this=0x1ef8310, h=...) at qpid/sys/posix/AsynchIO.cpp:446
#24 0x000000306420bbd3 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>)
    at /usr/include/boost/function/function_template.hpp:1013
#25 0x0000003064208dd6 in qpid::sys::DispatchHandle::processEvent (this=0x1ef8318, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280
#26 0x000000306414be9d in process (this=0xab0d20) at qpid/sys/Poller.h:131
#27 qpid::sys::Poller::run (this=0xab0d20) at qpid/sys/epoll/EpollPoller.cpp:524
#28 0x000000306414405a in qpid::sys::(anonymous namespace)::runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#29 0x0000003063007851 in start_thread (arg=0x7f097a771700) at pthread_create.c:301
#30 0x0000003062ce767d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) 


Version-Release number of selected component (if applicable):
qpid-cpp-server-0.14-21.el6_3.x86_64

Comment 4 Justin Ross 2012-08-27 14:13:47 UTC
Closing as a duplicate.  Will reopen if this turns out to be an issue distinct from bug 849654.

*** This bug has been marked as a duplicate of bug 849654 ***


Note You need to log in before you can comment on or make changes to this bug.