Bug 472219

Summary: Qpidd segv in qpid::broker::Queue::popAndDequeue or qpid::broker::Message::getRequiredCredit
Product: Red Hat Enterprise MRG Reporter: Andrew Stitcher <astitcher>
Component: qpid-cppAssignee: messaging-bugs <messaging-bugs>
Status: CLOSED WORKSFORME QA Contact: Kim van der Riet <kim.vdriet>
Severity: low Docs Contact:
Priority: low    
Version: betaCC: gsim
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-02-17 12:15:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andrew Stitcher 2008-11-19 13:51:47 UTC
Description of problem:

running qpidd and perftest on the same box with a loopback connection:

Start qpidd:
qpidd --log-enable=info+ -p 12345 --load-module src/.libs/rdma.so

Run perftest:
QPID_LOAD_MODULE=src/.libs/rdmaconnector.so src/tests/perftest -b 192.168.10.36 -p 12345 -P rdma --count 1000000 or
QPID_LOAD_MODULE=src/.libs/rdmaconnector.so src/tests/perftest -p 12345 --count 1000000
[192.168.10.36 is infiniband IP address for machine]

Now interrupt the run after a few seconds.

now run perftest again.
QPID_LOAD_MODULE=src/.libs/rdmaconnector.so src/tests/perftest -b 192.168.10.36 -p 12345 -P rdma --count 1000000 or
QPID_LOAD_MODULE=src/.libs/rdmaconnector.so src/tests/perftest -p 12345 --count 1000000

qpidd will segv with one of these stack traces:

#0  0x00002b08fe2b0e76 in qpid::broker::Queue::popAndDequeue (
    this=<value optimized out>) at ../../qpid/cpp/src/qpid/RefCounted.h:42
#1  0x00002b08fe2b4779 in qpid::broker::Queue::purge (this=0x178b6090, 
    purge_request=0) at ../../qpid/cpp/src/qpid/broker/Queue.cpp:467
#2  0x00002b08fe3217b4 in qpid::broker::SessionAdapter::QueueHandlerImpl::purge
    (this=<value optimized out>, queue=@0x2aaab07b9b70)
    at ../../qpid/cpp/src/qpid/broker/SessionAdapter.cpp:395
#3  0x00002b08fe722d42 in qpid::framing::AMQP_ServerOperations::QueueHandler::Invoker::visit (this=0x7fffac8e3670, body=<value optimized out>)
    at gen/qpid/framing/QueuePurgeBody.h:62
#4  0x00002b08fe728927 in qpid::framing::AMQP_ServerOperations::Invoker::visit
    (this=0x7fffac8e36f0, body=@0x2aaab07b9b60)
    at gen/qpid/framing/ServerInvoker.cpp:372
#5  0x00002b08fe32b35e in qpid::framing::invoke<qpid::broker::SessionAdapter> (
    target=<value optimized out>, body=@0x0)
    at ../../qpid/cpp/src/qpid/framing/Invoker.h:67
#6  0x00002b08fe329c88 in qpid::broker::SessionState::handleCommand (
    this=0x2aaaac001450, method=0x2aaab07b9b60, id=@0x7fffac8e3940)
    at ../../qpid/cpp/src/qpid/broker/SessionState.cpp:164
#7  0x00002b08fe32a268 in qpid::broker::SessionState::handleIn (
    this=0x2aaaac001450, frame=@0x7fffac8e4500)
    at ../../qpid/cpp/src/qpid/broker/SessionState.cpp:227
#8  0x00002b08fe7763e9 in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0x2aaab005db10, f=@0x7fffac8e4500)
    at ../../qpid/cpp/src/qpid/amqp_0_10/SessionHandler.cpp:92
#9  0x00002b08fe2befad in qpid::broker::Connection::received (
    this=0x2aaab00eca40, frame=@0x7fffac8e4500)
    at ../../qpid/cpp/src/qpid/framing/Handler.h:42
#10 0x00002b08fe292a9f in qpid::amqp_0_10::Connection::decode (
    this=0x2aaab0023f10, buffer=<value optimized out>, 
    size=<value optimized out>)
    at ../../qpid/cpp/src/qpid/amqp_0_10/Connection.cpp:55
#11 0x00002b08fea5f6eb in qpid::sys::RdmaIOHandler::readbuff (this=0x1fa997c0, 
    buff=0x2b08fea31cc0) at ../../qpid/cpp/src/qpid/sys/RdmaIOPlugin.cpp:158
#12 0x00002b08fec7984a in boost::function2<void, Rdma::AsynchIO&, Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x2b00fe5a3b50, 
    a0=@0xffffffff, a1=0x0)
    at /usr/include/boost/function/function_template.hpp:576
#13 0x00002b08fec75afb in Rdma::AsynchIO::processCompletions (this=0xffffffff)
    at ../../qpid/cpp/src/qpid/sys/rdma/RdmaIO.cpp:359
#14 0x00002b08fec76dc8 in Rdma::AsynchIO::dataEvent (this=0x18505760)
    at ../../qpid/cpp/src/qpid/sys/rdma/RdmaIO.cpp:273
#15 0x00002b08fe7a072f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2b00fe5a3b50, 
    a0=@0xffffffff) at /usr/include/boost/function/function_template.hpp:576
#16 0x00002b08fe79efaf in qpid::sys::DispatchHandle::processEvent (
    this=0x1912c430, type=<value optimized out>)
    at ../../qpid/cpp/src/qpid/sys/DispatchHandle.cpp:353
#17 0x00002b08fe79eb88 in qpid::sys::Dispatcher::run (this=0x7fffac8e4d10)
    at ../../qpid/cpp/src/qpid/sys/Poller.h:105
#18 0x00002b08fe294d5d in qpid::broker::Broker::run (
    this=<value optimized out>)
---Type <return> to continue, or q <return> to quit---
    at ../../qpid/cpp/src/qpid/broker/Broker.cpp:286
#19 0x0000000000406fe9 in QpiddBroker::execute (this=<value optimized out>, 
    options=0x1708e760) at ../../qpid/cpp/src/posix/QpiddBroker.cpp:161
#20 0x000000000040528c in main (argc=6, argv=0x7fffac8e5218)
    at ../../qpid/cpp/src/qpidd.cpp:76

OR

#0  qpid::broker::Message::getRequiredCredit (this=<value optimized out>)
    at ../../qpid/cpp/src/qpid/framing/TypeFilter.h:45
#1  0x00002ac927c307b9 in DeliveryRecord (this=0x7fff82f7e970, 
    _msg=<value optimized out>, _queue=@0x14b41c90, _tag=@0x14b41c88, 
    _acquired=<value optimized out>, accepted=false, _windowing=false)
    at ../../qpid/cpp/src/qpid/broker/DeliveryRecord.cpp:47
#2  0x00002ac927c7bf2a in qpid::broker::SemanticState::ConsumerImpl::deliver (
    this=0x14b41c30, msg=@0x7fff82f7e9e0)
    at ../../qpid/cpp/src/qpid/broker/SemanticState.cpp:269
#3  0x00002ac927c1baf0 in qpid::broker::Queue::dispatch (
    this=<value optimized out>, c={px = 0x7fff82f7ea40, pn = {pi_ = 0x1}})
    at ../../qpid/cpp/src/qpid/broker/Queue.cpp:348
#4  0x00002ac927c727dc in qpid::broker::SemanticState::ConsumerImpl::doOutput (
    this=<value optimized out>)
    at ../../qpid/cpp/src/qpid/broker/SemanticState.cpp:586
#5  0x00002ac92810200c in qpid::sys::AggregateOutput::doOutput (
    this=0x14b7c060) at ../../qpid/cpp/src/qpid/sys/AggregateOutput.cpp:49
#6  0x00002ac927c7cdc0 in qpid::broker::SemanticState::doOutput (
    this=<value optimized out>)
    at ../../qpid/cpp/src/qpid/broker/SemanticState.h:206
#7  0x00002ac92810200c in qpid::sys::AggregateOutput::doOutput (
    this=0x14b24210) at ../../qpid/cpp/src/qpid/sys/AggregateOutput.cpp:49
#8  0x00002ac927c25cfd in qpid::broker::Connection::doOutput (this=0x14b241a0)
    at ../../qpid/cpp/src/qpid/broker/Connection.cpp:217
#9  0x00002ac927bf7f01 in qpid::amqp_0_10::Connection::encode (
    this=0x14b5a400, buffer=<value optimized out>, size=<value optimized out>)
    at ../../qpid/cpp/src/qpid/amqp_0_10/Connection.cpp:86
#10 0x00002ac9283c0cfb in qpid::sys::RdmaIOHandler::idle (this=0x14b5a8f0)
    at ../../qpid/cpp/src/qpid/sys/RdmaIOPlugin.cpp:122
#11 0x00002ac9285df73a in boost::function1<void, Rdma::AsynchIO&, std::allocator<boost::function_base> >::operator() (this=0x2aaade4834a0, a0=@0x1)
    at /usr/include/boost/function/function_template.hpp:576
#12 0x00002ac9285d9434 in Rdma::AsynchIO::doWriteCallback (this=0x14b0af10)
    at ../../qpid/cpp/src/qpid/sys/rdma/RdmaIO.cpp:411
#13 0x00002ac9285dcdef in Rdma::AsynchIO::dataEvent (this=0x14b0af10)
    at ../../qpid/cpp/src/qpid/sys/rdma/RdmaIO.cpp:283
#14 0x00002ac92810672f in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x2aaade4834a0, a0=@0x1)
    at /usr/include/boost/function/function_template.hpp:576
#15 0x00002ac928104faf in qpid::sys::DispatchHandle::processEvent (
    this=0x14c1dfa0, type=<value optimized out>)
    at ../../qpid/cpp/src/qpid/sys/DispatchHandle.cpp:353
#16 0x00002ac928104b88 in qpid::sys::Dispatcher::run (this=0x7fff82f7f3b0)
    at ../../qpid/cpp/src/qpid/sys/Poller.h:105
#17 0x00002ac927bfad5d in qpid::broker::Broker::run (
    this=<value optimized out>)
    at ../../qpid/cpp/src/qpid/broker/Broker.cpp:286
#18 0x0000000000406fe9 in QpiddBroker::execute (this=<value optimized out>, 
    options=0x12aa9760) at ../../qpid/cpp/src/posix/QpiddBroker.cpp:161
#19 0x000000000040528c in main (argc=6, argv=0x7fff82f7f8b8)
    at ../../qpid/cpp/src/qpidd.cpp:76

Version-Release number of selected component (if applicable):

present in svn r713020 of qpidd

How reproducible:

more than 50% of the time

Steps to Reproduce:

As above
  
Actual results:

segv

Expected results:

run to completion

Additional info:

Comment 1 Gordon Sim 2008-12-02 14:30:48 UTC
Can't reproduce; suspected that this may be related to the machine. Reducing priority and will keep an eye out for it. (The test is quite a common one anyway so if there is an issue it should show up).

Comment 2 Gordon Sim 2009-02-17 12:15:34 UTC
Closing this as it hasn't cropped up on any other machine. Can reopen or create new BZ for new cases.