Bug 801310 - qpidd crashes while qpid-perftest --mode fanout / topic in qpid::broker::Message::encode() -> map_if<> -> operator()
qpidd crashes while qpid-perftest --mode fanout / topic in qpid::broker::Mess...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp (Show other bugs)
Development
Unspecified Unspecified
high Severity urgent
: 2.1.2
: ---
Assigned To: Ken Giusti
Frantisek Reznicek
:
Depends On:
Blocks: 791249
  Show dependency treegraph
 
Reported: 2012-03-08 04:24 EST by Frantisek Reznicek
Modified: 2015-11-15 20:14 EST (History)
4 users (show)

See Also:
Fixed In Version: qpid-cpp-0.14-12
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-07 12:40:50 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Potential fix - needs testing. (1.76 KB, patch)
2012-03-09 15:00 EST, Ken Giusti
no flags Details | Diff

  None (edit)
Description Frantisek Reznicek 2012-03-08 04:24:20 EST
Description of problem:

qpidd crashes while qpid-perftest --mode fanout or topic in qpid::broker::Message::encode() -> map_if<> -> operator():

  (gdb)   6 Thread 0x2b6379e02040 (LWP 16744)  0x00000036088d4d98 in epoll_wait ()
     from /lib64/libc.so.6
    5 Thread 16745  0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/libpthread.so.0
    4 Thread 16746  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
    3 Thread 16748  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
    2 Thread 16749  0x00000036088c6c2b in write () from /lib64/libc.so.6
  * 1 Thread 0x42e13940 (LWP 16747)  0x00002aaab40158b0 in ?? ()
  ...
  Thread 1 (Thread 0x42e13940 (LWP 16747)):
  #0  0x00002aaab40158b0 in ?? ()
  #1  0x000000360bbaa358 in operator() (this=<value optimized out>, buffer=...)
      at qpid/framing/TypeFilter.h:38
  #2  map_if<qpid::framing::EncodeBody, qpid::framing::TypeFilter<3u> > (
      this=<value optimized out>, buffer=...) at qpid/framing/FrameSet.h:110
  #3  qpid::broker::Message::encode (this=<value optimized out>, buffer=...)
      at qpid/broker/Message.cpp:143
  #4  0x00002b637c262583 in mrg::msgstore::MessageStoreImpl::msgEncode (
      this=<value optimized out>, 
      buff=std::vector of length 160, capacity 160 = {...}, message=...)
      at MessageStoreImpl.cpp:1321
  #5  0x00002b637c262bf7 in mrg::msgstore::MessageStoreImpl::store (

qpid-perftest is ran with multiple publishers and subscribers in durable mode.
This defect is similar to bug 791249 and is likely caused by bug 791249's fix.


Version-Release number of selected component (if applicable):
 qpid-cpp*-0.14-9.el5 or qpid-cpp*-0.14-10.el5
 qpid-java-*-0.14-3.el5
 qpid-qmf-*0.14-3.el5
 qpid-tests-0.14-1.el5
 qpid-tools-0.14-1.el5


How reproducible:
80%
detected on updated rhel5.7 i386 / x86_64
detected on updated rhel5.8 i386 / x86_64
not detected on rhel6.2

Steps to Reproduce:
see bug 791249 steps

  
Actual results:
qpidd crashes while qpid-perftest is running.

Expected results:
qpidd should not crash.

Additional info:

Bug 791249 fix moved the original crash to other place:

-rw------- 1 root root 77475840 Mar  7 14:33 /root/.qpidd/core.16744
/root/.qpidd/core.16744: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'qpidd'
  GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-42.el5)
  ...
  Core was generated by `/usr/sbin/qpidd --auth no --daemon --port 0 --log-enable info+ --log-to-file qp'.
  Program terminated with signal 11, Segmentation fault.
  (gdb) Stack level 0, frame at 0x42e0e7e0:
   rip = 0x2aaab40158b0; saved rip 0x360bbaa358
   called by frame at 0x42e0e810
   Arglist at 0x42e0e7d0, args: 
   Locals at 0x42e0e7d0, Previous frame's sp is 0x42e0e7e0
   Saved registers:
    rip at 0x42e0e7d8
  (*): Shared library is missing debugging information.
  (gdb)   6 Thread 0x2b6379e02040 (LWP 16744)  0x00000036088d4d98 in epoll_wait ()
     from /lib64/libc.so.6
    5 Thread 16745  0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/libpthread.so.0
    4 Thread 16746  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
    3 Thread 16748  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
    2 Thread 16749  0x00000036088c6c2b in write () from /lib64/libc.so.6
  * 1 Thread 0x42e13940 (LWP 16747)  0x00002aaab40158b0 in ?? ()
  Thread 6 (Thread 0x2b6379e02040 (LWP 16744)):
  #0  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
  #1  0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, 
      timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568
  #2  0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0)
      at qpid/sys/epoll/EpollPoller.cpp:520
  #3  0x000000360bb31d46 in qpid::broker::Broker::run (
      this=<value optimized out>) at qpid/broker/Broker.cpp:400
  #4  0x0000000000409b8c in QpiddDaemon::child (this=0x141fcbd0)
      at posix/QpiddBroker.cpp:144
  #5  0x000000360bb5f60e in qpid::broker::Daemon::fork (this=0x7fff7f47ddb0)
      at qpid/broker/Daemon.cpp:91
  #6  0x0000000000407085 in QpiddBroker::execute (this=<value optimized out>, 
      options=<value optimized out>) at posix/QpiddBroker.cpp:182
  #7  0x0000000000405822 in run_broker (argc=16, argv=0x7fff7f47e3a8, 
      hidden=<value optimized out>) at qpidd.cpp:83
  #8  0x000000360881d994 in __libc_start_main () from /lib64/libc.so.6
  #9  0x0000000000405279 in _start ()
  Thread 5 (Thread 16745):
  #0  0x000000360900b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
     from /lib64/libpthread.so.0
  #1  0x000000360b61253a in qpid::sys::Timer::run (this=0xffffffffffffff92)
      at ../include/qpid/sys/posix/Condition.h:69
  #2  0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x141e9c24) at qpid/sys/posix/Thread.cpp:35
  #3  0x000000360900677d in start_thread () from /lib64/libpthread.so.0
  #4  0x00000036088d49ad in clone () from /lib64/libc.so.6
  Thread 4 (Thread 16746):
  #0  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
  #1  0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, 
      timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568
  #2  0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0)
      at qpid/sys/epoll/EpollPoller.cpp:520
  #3  0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x6) at qpid/sys/posix/Thread.cpp:35
  #4  0x000000360900677d in start_thread () from /lib64/libpthread.so.0
  #5  0x00000036088d49ad in clone () from /lib64/libc.so.6
  Thread 3 (Thread 16748):
  #0  0x00000036088d4d98 in epoll_wait () from /lib64/libc.so.6
  #1  0x000000360b534431 in qpid::sys::Poller::wait (this=0x1417bca0, 
      timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:568
  #2  0x000000360b534ea7 in qpid::sys::Poller::run (this=0x1417bca0)
      at qpid/sys/epoll/EpollPoller.cpp:520
  #3  0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x6) at qpid/sys/posix/Thread.cpp:35
  #4  0x000000360900677d in start_thread () from /lib64/libpthread.so.0
  #5  0x00000036088d49ad in clone () from /lib64/libc.so.6
  Thread 2 (Thread 16749):
  #0  0x00000036088c6c2b in write () from /lib64/libc.so.6
  #1  0x000000360b51c27f in qpid::sys::Socket::write (
      this=<value optimized out>, buf=0x142998f0, count=205)
      at qpid/sys/posix/Socket.cpp:232
  #2  0x000000360b52446b in qpid::sys::posix::AsynchIO::writeable (
      this=0x14297700, h=...) at qpid/sys/posix/AsynchIO.cpp:516
  #3  0x000000360b60b17a in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0xcd, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #4  0x000000360b60a801 in qpid::sys::DispatchHandle::processEvent (
      this=0x14297708, type=WRITABLE) at qpid/sys/DispatchHandle.cpp:283
  #5  0x000000360b534ed4 in process (this=0x1417bca0) at qpid/sys/Poller.h:131
  #6  qpid::sys::Poller::run (this=0x1417bca0)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #7  0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x17) at qpid/sys/posix/Thread.cpp:35
  #8  0x000000360900677d in start_thread () from /lib64/libpthread.so.0
  #9  0x00000036088d49ad in clone () from /lib64/libc.so.6
  Thread 1 (Thread 0x42e13940 (LWP 16747)):
  #0  0x00002aaab40158b0 in ?? ()
  #1  0x000000360bbaa358 in operator() (this=<value optimized out>, buffer=...)
      at qpid/framing/TypeFilter.h:38
  #2  map_if<qpid::framing::EncodeBody, qpid::framing::TypeFilter<3u> > (
      this=<value optimized out>, buffer=...) at qpid/framing/FrameSet.h:110
  #3  qpid::broker::Message::encode (this=<value optimized out>, buffer=...)
      at qpid/broker/Message.cpp:143
  #4  0x00002b637c262583 in mrg::msgstore::MessageStoreImpl::msgEncode (
      this=<value optimized out>, 
      buff=std::vector of length 160, capacity 160 = {...}, message=...)
      at MessageStoreImpl.cpp:1321
  #5  0x00002b637c262bf7 in mrg::msgstore::MessageStoreImpl::store (
      this=0x2aaab40158c0, queue=0x2aaaac5b52e0, txn=0x42e0edf0, message=...)
      at MessageStoreImpl.cpp:1331
  #6  0x00002b637c27b7db in mrg::msgstore::MessageStoreImpl::enqueue (
      this=0x141fd240, ctxt=0x0, msg=..., queue=...)
      at MessageStoreImpl.cpp:1303
  #7  0x000000360bbb783b in qpid::broker::MessageStoreModule::enqueue (
      this=<value optimized out>, ctxt=0x0, msg=..., queue=...)
      at qpid/broker/MessageStoreModule.cpp:125
  #8  0x000000360bbc9f80 in qpid::broker::Queue::enqueue (this=0x2aaaac5b52e0, 
      ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>)
      at qpid/broker/Queue.cpp:811
  #9  0x000000360bbcb908 in qpid::broker::Queue::deliver (this=0x2aaaac5b52e0, 
      msg=...) at qpid/broker/Queue.cpp:171
  #10 0x000000360bb620c2 in qpid::broker::DeliverableMessage::deliverTo (
      this=0x42e10870, queue=...) at qpid/broker/DeliverableMessage.cpp:33
  #11 0x000000360bb7fbc2 in qpid::broker::Exchange::doRoute (this=0x14205a88, 
      msg=..., b=...) at qpid/broker/Exchange.cpp:119
  #12 0x000000360bc3ba18 in qpid::broker::TopicExchange::route (
      this=0x14205a88, msg=..., routingKey="qpid-perftest0")
      at qpid/broker/TopicExchange.cpp:375
  #13 0x000000360bc06a1a in qpid::broker::SemanticState::route (
      this=<value optimized out>, msg=..., strategy=...)
      at qpid/broker/SemanticState.cpp:495
  #14 0x000000360bc0744d in qpid::broker::SemanticState::handle (
      this=0x14254b88, msg=...) at qpid/broker/SemanticState.cpp:449
  #15 0x000000360bc31448 in qpid::broker::SessionState::handleContent (
      this=0x142549b0, frame=..., id=<value optimized out>)
      at qpid/broker/SessionState.cpp:266
  #16 0x000000360bc31bd0 in qpid::broker::SessionState::handleIn (
      this=0x142549b0, frame=...) at qpid/broker/SessionState.cpp:362
  #17 0x000000360b5d2d65 in qpid::amqp_0_10::SessionHandler::handleIn (
      this=0x1424db30, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
  #18 0x000000360bb5cee1 in operator() (this=0x1424ae60, frame=...)
      at qpid/framing/Handler.h:42
  #19 qpid::broker::ConnectionHandler::handle (this=0x1424ae60, frame=...)
      at qpid/broker/ConnectionHandler.cpp:87
  #20 0x000000360bb520a8 in qpid::broker::Connection::received (
      this=0x1424ac80, frame=...) at qpid/broker/Connection.cpp:159
  #21 0x000000360bb22364 in qpid::amqp_0_10::Connection::decode (
      this=0x14255550, buffer=<value optimized out>, size=<value optimized out>)
      at qpid/amqp_0_10/Connection.cpp:58
  #22 0x000000360b605662 in qpid::sys::AsynchIOHandler::readbuff (
      this=0x14254710, buff=0x1424cc60) at qpid/sys/AsynchIOHandler.cpp:135
  #23 0x000000360b529d4a in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
      this=0x2aaab402b320, a0=..., a1=0x2aaab000fd10)
      at /usr/include/boost/function/function_template.hpp:576
  #24 0x000000360b527af0 in qpid::sys::posix::AsynchIO::readable (
      this=0x14209a70, h=...) at qpid/sys/posix/AsynchIO.cpp:446
  #25 0x000000360b60b17a in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2aaab402b320, a0=...)
      at /usr/include/boost/function/function_template.hpp:576
  #26 0x000000360b60a87f in qpid::sys::DispatchHandle::processEvent (
      this=0x14209a78, type=READABLE) at qpid/sys/DispatchHandle.cpp:280
  #27 0x000000360b534ed4 in process (this=0x1417bca0) at qpid/sys/Poller.h:131
  #28 qpid::sys::Poller::run (this=0x1417bca0)
      at qpid/sys/epoll/EpollPoller.cpp:524
  #29 0x000000360b52bfaa in qpid::sys::(anonymous namespace)::runRunnable (
      p=0x2aaab40158c0) at qpid/sys/posix/Thread.cpp:35
  #30 0x000000360900677d in start_thread () from /lib64/libpthread.so.0
  #31 0x00000036088d49ad in clone () from /lib64/libc.so.6
  (gdb) quit
Comment 2 Gordon Sim 2012-03-08 05:30:03 EST
Looks like dup of 791249
Comment 3 Ken Giusti 2012-03-08 15:20:00 EST
The 791249 crash was caused by accessing a message header outside of the lock.  This problem seems identical, but the access was to the message body.

Looks like we have to lock the message body as well - will try to repo to be sure.

-K
Comment 4 Ken Giusti 2012-03-09 15:00:46 EST
Created attachment 568990 [details]
Potential fix - needs testing.
Comment 8 Frantisek Reznicek 2012-03-20 04:39:36 EDT
Retested on rhel5.7/5.8/6.2 i/x on packages:
 qpid-cpp-*0.14-12.el5 + qpid-qmf-*0.14-3.el5
 qpid-cpp-*0.14-12.el6 + qpid-qmf-devel-0.14-5.el6

Issue is reliably fixed, no other crashes detected.

Waiting for installable set & retest
Comment 9 Ken Giusti 2012-03-20 08:59:12 EDT
Reopened upstream jira:

https://issues.apache.org/jira/browse/QPID-3877
Comment 12 Ken Giusti 2012-03-20 14:38:58 EDT
Submitted fix to upstream trunk.  Two patches applied, in order:


http://svn.apache.org/viewvc?view=rev&rev=1296230
http://svn.apache.org/viewvc?view=rev&rev=1303068
Comment 13 Frantisek Reznicek 2012-03-27 06:55:14 EDT
Retested on rhel5.7/5.8/6.2 i/x on packages:
 qpid-cpp-*0.14-14.el5 + qpid-qmf-*0.14-4.el5
 qpid-cpp-*0.14-12.el6 + qpid-qmf-*0.14-6.el6

Issue is reliably fixed, no other crashes detected.

Upstream patch to qpid-0.16 (comment 12) looks ok, contains bug 791249 + bug 801310 patches.


-> VERIFIED

Note You need to log in before you can comment on or make changes to this bug.