Bug 1121615

Summary: An attempt to enqueue&dequeue a message with "too high priority" to a ring queue causes broker segfault
Product: Red Hat Enterprise MRG Reporter: Pavel Moravec <pmoravec>
Component: qpid-cppAssignee: Alan Conway <aconway>
Status: CLOSED ERRATA QA Contact: Jitka Kocnova <jkocnova>
Severity: high Docs Contact:
Priority: high    
Version: 2.5CC: aconway, esammons, jkocnova, jross, lzhaldyb, mcressma, zkraus
Target Milestone: 2.5.5   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qpid-cpp-0.18-27 Doc Type: Bug Fix
Doc Text:
It was discovered that if a message in a ring queue was given an invalid priority (greater than the maximum defined for that queue), an attempt to enqueue and dequeue it caused a segfault. Invalid priorities are now treated as equivalent to the maximum priority. The system no longer segfaults when processing messages with an invalid priority.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-22 16:34:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1140368, 1140369    

Description Pavel Moravec 2014-07-21 11:43:07 UTC
Description of problem:
Testing bz860691, an attempt to send a message with "too high" priority to a ring queue causes broker segfault.


Version-Release number of selected component (if applicable):
0.18-25 (also 0.18-26)
_not_ seen in 0.22-* versions


How reproducible:
100%


Steps to Reproduce:
qpid-config add queue q --limit-policy=ring --argument=x-qpid-priorities=10
qpid-send -a q -m 1 --priority=10


Actual results:
Broker segfaults


Expected results:
No broker segfault


Additional info:
Backtrace:
#0  begin (this=0x7f7a7401b220, m=<value optimized out>) at /usr/include/boost/multi_index/sequenced_index.hpp:203
#1  qpid::broker::RingQueuePolicy::enqueued (this=0x7f7a7401b220, m=<value optimized out>) at qpid/broker/QueuePolicy.cpp:231
#2  0x00007f7a85f7a9f2 in qpid::broker::Queue::push (this=0x7f7a74019db0, msg=..., isRecovery=false) at qpid/broker/Queue.cpp:908
#3  0x00007f7a85f7be8f in qpid::broker::Queue::deliver (this=0x7f7a74019db0, msg=...) at qpid/broker/Queue.cpp:232
#4  0x00007f7a85f181bd in qpid::broker::DeliverableMessage::deliverTo (this=0x7fffbf21cf30, queue=<value optimized out>)
    at qpid/broker/DeliverableMessage.cpp:33
#5  0x00007f7a85f34611 in qpid::broker::Exchange::doRoute (this=0x2087a30, msg=..., b=...) at qpid/broker/Exchange.cpp:123
#6  0x00007f7a85f1b9ba in qpid::broker::DirectExchange::route (this=0x20879d0, msg=...) at qpid/broker/DirectExchange.cpp:168
#7  0x00007f7a85fb1349 in qpid::broker::SemanticState::route (this=0x7f7a70171fc8, msg=..., strategy=...)
    at qpid/broker/SemanticState.cpp:502
#8  0x00007f7a85fb1ce1 in qpid::broker::SemanticState::handle (this=0x7f7a70171fc8, msg=...) at qpid/broker/SemanticState.cpp:453
#9  0x00007f7a85fce7cb in qpid::broker::SessionState::handleContent (this=0x7f7a70171df0, frame=<value optimized out>, 
    id=<value optimized out>) at qpid/broker/SessionState.cpp:224
#10 0x00007f7a85fcf3f7 in qpid::broker::SessionState::handleIn (this=0x7f7a70171df0, frame=...) at qpid/broker/SessionState.cpp:283
#11 0x00007f7a85a93ee1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f7a70171c00, f=...)
    at qpid/amqp_0_10/SessionHandler.cpp:93
#12 0x00007f7a85f12d83 in operator() (this=0x7f7a70003180, frame=...) at qpid/framing/Handler.h:42
#13 qpid::broker::ConnectionHandler::handle (this=0x7f7a70003180, frame=...) at qpid/broker/ConnectionHandler.cpp:90
#14 0x00007f7a85f0b788 in qpid::broker::Connection::received (this=0x7f7a70002f70, frame=...) at qpid/broker/Connection.cpp:166
#15 0x00007f7a85ed61fd in qpid::amqp_0_10::Connection::decode (this=0x7f7a70003d50, buffer=<value optimized out>, 
    size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58
#16 0x00007f7a85ac460d in qpid::sys::AsynchIOHandler::readbuff (this=0x7f7a74005b60, buff=0x7f7a74000f80)
    at qpid/sys/AsynchIOHandler.cpp:170
#17 0x00007f7a859e60b1 in operator() (this=0x7f7a74004d20, h=...) at /usr/include/boost/function/function_template.hpp:1013
#18 qpid::sys::posix::AsynchIO::readable (this=0x7f7a74004d20, h=...) at qpid/sys/posix/AsynchIO.cpp:462
#19 0x00007f7a85aca523 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, 
    a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
#20 0x00007f7a85ac7726 in qpid::sys::DispatchHandle::processEvent (this=0x7f7a74004d28, type=qpid::sys::Poller::READABLE)
    at qpid/sys/DispatchHandle.cpp:280
#21 0x00007f7a859f30cd in process (this=0x2078d90) at qpid/sys/Poller.h:131
#22 qpid::sys::Poller::run (this=0x2078d90) at qpid/sys/epoll/EpollPoller.cpp:524
#23 0x00007f7a85eea7b2 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:432
---Type <return> to continue, or q <return> to quit---
#24 0x000000000040dea1 in qpid::broker::QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>)
    at posix/QpiddBroker.cpp:206
#25 0x000000000040aa34 in qpid::broker::run_broker (argc=1, argv=0x7fffbf21f8f8, hidden=<value optimized out>) at qpidd.cpp:106
#26 0x00000032c441ed1d in __libc_start_main () from /lib64/libc.so.6
#27 0x000000000040a2a9 in _start ()

Comment 1 Pavel Moravec 2014-07-21 12:39:42 UTC
Updating..

Original segfault is _not_ present in 0.18-26, exactly due to the change introduced in -26:

-void RingQueuePolicy::enqueued(const QueuedMessage& m)
+void RingQueuePolicy::enqueued(const QueuedMessage& m, bool inUpdate)
 {
     uint priority = m.payload->getPriority();
     assert(priority < queue.size());
-    // Need to insert in correct location based on position
-    typedef OrderedMessages::nth_index<0>::type MessageSeq;
-    MessageSeq& seq = queue[priority].get<0>();
-    MessageSeq::iterator i = lower_bound(seq.begin(), seq.end(), m);
-    seq.insert(i, m);
+    if (inUpdate) {
+        // If this is an acquired message being enqueued during a cluster update
+        // then we need to ensure it goes in order by position.
+        typedef OrderedMessages::nth_index<0>::type MessageSeq;
+        MessageSeq& seq = queue[priority].get<0>();
+        MessageSeq::iterator i = lower_bound(seq.begin(), seq.end(), m);
+        seq.insert(i, m);
+    }
+    else {
+        // If this is a normal enqueue then we know it goes to the back of the
+        // queue, so avoid performance overhead of sorting.
+        queue[priority].push_back(m);
+    }
 }



BUT 0.18-26 suffers by another segfault with reproducer (again, sending priority=10 is the key point):

qpid-config add queue q -b train2 --limit-policy=ring --max-queue-count=100 --flow-stop-count=0 --flow-stop-size=0 --argument=x-qpid-priorities=10

for i in $(seq 1 10); do
        qpid-send -a q -m 1000 -b train2 --content-size=128 --priority=$i &
done
qpid-receive -a q -m 10000 -b train2 --print-content=no -f &


and backtrace:
#0  find<qpid::framing::SequenceNumber, boost::hash<qpid::framing::SequenceNumber>, std::equal_to<qpid::framing::SequenceNumber> > (
    this=<value optimized out>, m=<value optimized out>, q=..., remove=false) at /usr/include/boost/multi_index/hashed_index.hpp:443
#1  find<qpid::framing::SequenceNumber> (this=<value optimized out>, m=<value optimized out>, q=..., remove=false)
    at /usr/include/boost/multi_index/hashed_index.hpp:429
#2  qpid::broker::RingQueuePolicy::find (this=<value optimized out>, m=<value optimized out>, q=..., remove=false)
    at qpid/broker/QueuePolicy.cpp:330
#3  0x00007f74486e1ecd in qpid::broker::Queue::isEnqueued (this=0x7f742c05d480, msg=...) at qpid/broker/Queue.cpp:1867
#4  0x00007f74486ecef7 in qpid::broker::Queue::dequeue (this=0x7f742c05d480, ctxt=0x0, msg=..., removed=false)
    at qpid/broker/Queue.cpp:1089
#5  0x00007f744868f1df in qpid::broker::DeliveryRecord::accept (this=0x7f74380d36b0, ctxt=0x0) at qpid/broker/DeliveryRecord.cpp:120
#6  0x00007f7448731687 in operator() (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/mem_fn_template.hpp:181
#7  operator()<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list1<qpid::broker::DeliveryRecord&> > (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/bind.hpp:296
#8  operator()<qpid::broker::DeliveryRecord> (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/bind_template.hpp:32
#9  operator() (__first=..., __last=..., __pred=...) at qpid/broker/SemanticState.cpp:786
#10 std::__find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., __last=..., 
    __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:222
#11 0x00007f7448731c99 in find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., 
    __last=..., __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:4248
#12 std::remove_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., __last=..., 
    __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:1153
#13 0x00007f744872981d in qpid::broker::SemanticState::accepted (this=0x7f7434005f38, commands=...)
    at qpid/broker/SemanticState.cpp:823
#14 0x00007f74481ad4d3 in invoke<qpid::framing::AMQP_ServerOperations::MessageHandler> (this=<value optimized out>, 
    body=<value optimized out>) at qpid/framing/MessageAcceptBody.h:64
..

Comment 2 Alan Conway 2014-07-21 15:50:04 UTC
Fixed 

http://git.app.eng.bos.redhat.com/git/rh-qpid.git/commit/?h=0.18-mrg-aconway-bz1121615

Treat priorities > 9 as priority 9.

Comment 3 Alan Conway 2014-07-21 16:22:41 UTC
Confirmed that this is not a problem on 0.22-mrg or trunk.

Comment 4 Alan Conway 2014-07-21 17:52:26 UTC
Rebased to include: 
0632fb6 Bug 860691,QPID-4287: Poor performance when a priority queue with a ring queue policy has a large backlog

New branch:

http://git.app.eng.bos.redhat.com/git/rh-qpid.git/log/?h=0.18-mrg-aconway-bz1121615-2

Comment 7 Pavel Moravec 2014-07-22 06:42:44 UTC
Stress-tested 0.18-27 by sending & consuming messages with various (wrong) priority, no issue found.

No impact to performance (wrt. orig bz 860691).

Comment 9 Jitka Kocnova 2014-09-26 14:16:22 UTC
This issue is fixed.

Verified OS: RHEL 6 x86_64, i386; RHEL 5 x86_64, i386

Packages:

qpid-cpp-client-devel-docs-0.18-25.el6 
qpid-cpp-client-devel-0.18-32.el6 
qpid-cpp-client-ssl-0.18-32.el6 
qpid-cpp-client-0.18-32.el6 
qpid-cpp-debuginfo-0.18-32.el6 
qpid-cpp-server-devel-0.18-32.el6   
qpid-cpp-server-ssl-0.18-32.el6 
qpid-cpp-server-store-0.18-32.el6 
qpid-cpp-server-xml-0.18-32.el6
qpid-cpp-server-0.18-32.el6  
qpid-tools-0.18-10.el6_4


-> VERIFIED

Comment 11 errata-xmlrpc 2014-10-22 16:34:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-1682.html