Description of problem: Testing bz860691, an attempt to send a message with "too high" priority to a ring queue causes broker segfault. Version-Release number of selected component (if applicable): 0.18-25 (also 0.18-26) _not_ seen in 0.22-* versions How reproducible: 100% Steps to Reproduce: qpid-config add queue q --limit-policy=ring --argument=x-qpid-priorities=10 qpid-send -a q -m 1 --priority=10 Actual results: Broker segfaults Expected results: No broker segfault Additional info: Backtrace: #0 begin (this=0x7f7a7401b220, m=<value optimized out>) at /usr/include/boost/multi_index/sequenced_index.hpp:203 #1 qpid::broker::RingQueuePolicy::enqueued (this=0x7f7a7401b220, m=<value optimized out>) at qpid/broker/QueuePolicy.cpp:231 #2 0x00007f7a85f7a9f2 in qpid::broker::Queue::push (this=0x7f7a74019db0, msg=..., isRecovery=false) at qpid/broker/Queue.cpp:908 #3 0x00007f7a85f7be8f in qpid::broker::Queue::deliver (this=0x7f7a74019db0, msg=...) at qpid/broker/Queue.cpp:232 #4 0x00007f7a85f181bd in qpid::broker::DeliverableMessage::deliverTo (this=0x7fffbf21cf30, queue=<value optimized out>) at qpid/broker/DeliverableMessage.cpp:33 #5 0x00007f7a85f34611 in qpid::broker::Exchange::doRoute (this=0x2087a30, msg=..., b=...) at qpid/broker/Exchange.cpp:123 #6 0x00007f7a85f1b9ba in qpid::broker::DirectExchange::route (this=0x20879d0, msg=...) at qpid/broker/DirectExchange.cpp:168 #7 0x00007f7a85fb1349 in qpid::broker::SemanticState::route (this=0x7f7a70171fc8, msg=..., strategy=...) at qpid/broker/SemanticState.cpp:502 #8 0x00007f7a85fb1ce1 in qpid::broker::SemanticState::handle (this=0x7f7a70171fc8, msg=...) at qpid/broker/SemanticState.cpp:453 #9 0x00007f7a85fce7cb in qpid::broker::SessionState::handleContent (this=0x7f7a70171df0, frame=<value optimized out>, id=<value optimized out>) at qpid/broker/SessionState.cpp:224 #10 0x00007f7a85fcf3f7 in qpid::broker::SessionState::handleIn (this=0x7f7a70171df0, frame=...) at qpid/broker/SessionState.cpp:283 #11 0x00007f7a85a93ee1 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x7f7a70171c00, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93 #12 0x00007f7a85f12d83 in operator() (this=0x7f7a70003180, frame=...) at qpid/framing/Handler.h:42 #13 qpid::broker::ConnectionHandler::handle (this=0x7f7a70003180, frame=...) at qpid/broker/ConnectionHandler.cpp:90 #14 0x00007f7a85f0b788 in qpid::broker::Connection::received (this=0x7f7a70002f70, frame=...) at qpid/broker/Connection.cpp:166 #15 0x00007f7a85ed61fd in qpid::amqp_0_10::Connection::decode (this=0x7f7a70003d50, buffer=<value optimized out>, size=<value optimized out>) at qpid/amqp_0_10/Connection.cpp:58 #16 0x00007f7a85ac460d in qpid::sys::AsynchIOHandler::readbuff (this=0x7f7a74005b60, buff=0x7f7a74000f80) at qpid/sys/AsynchIOHandler.cpp:170 #17 0x00007f7a859e60b1 in operator() (this=0x7f7a74004d20, h=...) at /usr/include/boost/function/function_template.hpp:1013 #18 qpid::sys::posix::AsynchIO::readable (this=0x7f7a74004d20, h=...) at qpid/sys/posix/AsynchIO.cpp:462 #19 0x00007f7a85aca523 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=<value optimized out>, a0=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013 #20 0x00007f7a85ac7726 in qpid::sys::DispatchHandle::processEvent (this=0x7f7a74004d28, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:280 #21 0x00007f7a859f30cd in process (this=0x2078d90) at qpid/sys/Poller.h:131 #22 qpid::sys::Poller::run (this=0x2078d90) at qpid/sys/epoll/EpollPoller.cpp:524 #23 0x00007f7a85eea7b2 in qpid::broker::Broker::run (this=<value optimized out>) at qpid/broker/Broker.cpp:432 ---Type <return> to continue, or q <return> to quit--- #24 0x000000000040dea1 in qpid::broker::QpiddBroker::execute (this=<value optimized out>, options=<value optimized out>) at posix/QpiddBroker.cpp:206 #25 0x000000000040aa34 in qpid::broker::run_broker (argc=1, argv=0x7fffbf21f8f8, hidden=<value optimized out>) at qpidd.cpp:106 #26 0x00000032c441ed1d in __libc_start_main () from /lib64/libc.so.6 #27 0x000000000040a2a9 in _start ()
Updating.. Original segfault is _not_ present in 0.18-26, exactly due to the change introduced in -26: -void RingQueuePolicy::enqueued(const QueuedMessage& m) +void RingQueuePolicy::enqueued(const QueuedMessage& m, bool inUpdate) { uint priority = m.payload->getPriority(); assert(priority < queue.size()); - // Need to insert in correct location based on position - typedef OrderedMessages::nth_index<0>::type MessageSeq; - MessageSeq& seq = queue[priority].get<0>(); - MessageSeq::iterator i = lower_bound(seq.begin(), seq.end(), m); - seq.insert(i, m); + if (inUpdate) { + // If this is an acquired message being enqueued during a cluster update + // then we need to ensure it goes in order by position. + typedef OrderedMessages::nth_index<0>::type MessageSeq; + MessageSeq& seq = queue[priority].get<0>(); + MessageSeq::iterator i = lower_bound(seq.begin(), seq.end(), m); + seq.insert(i, m); + } + else { + // If this is a normal enqueue then we know it goes to the back of the + // queue, so avoid performance overhead of sorting. + queue[priority].push_back(m); + } } BUT 0.18-26 suffers by another segfault with reproducer (again, sending priority=10 is the key point): qpid-config add queue q -b train2 --limit-policy=ring --max-queue-count=100 --flow-stop-count=0 --flow-stop-size=0 --argument=x-qpid-priorities=10 for i in $(seq 1 10); do qpid-send -a q -m 1000 -b train2 --content-size=128 --priority=$i & done qpid-receive -a q -m 10000 -b train2 --print-content=no -f & and backtrace: #0 find<qpid::framing::SequenceNumber, boost::hash<qpid::framing::SequenceNumber>, std::equal_to<qpid::framing::SequenceNumber> > ( this=<value optimized out>, m=<value optimized out>, q=..., remove=false) at /usr/include/boost/multi_index/hashed_index.hpp:443 #1 find<qpid::framing::SequenceNumber> (this=<value optimized out>, m=<value optimized out>, q=..., remove=false) at /usr/include/boost/multi_index/hashed_index.hpp:429 #2 qpid::broker::RingQueuePolicy::find (this=<value optimized out>, m=<value optimized out>, q=..., remove=false) at qpid/broker/QueuePolicy.cpp:330 #3 0x00007f74486e1ecd in qpid::broker::Queue::isEnqueued (this=0x7f742c05d480, msg=...) at qpid/broker/Queue.cpp:1867 #4 0x00007f74486ecef7 in qpid::broker::Queue::dequeue (this=0x7f742c05d480, ctxt=0x0, msg=..., removed=false) at qpid/broker/Queue.cpp:1089 #5 0x00007f744868f1df in qpid::broker::DeliveryRecord::accept (this=0x7f74380d36b0, ctxt=0x0) at qpid/broker/DeliveryRecord.cpp:120 #6 0x00007f7448731687 in operator() (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/mem_fn_template.hpp:181 #7 operator()<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list1<qpid::broker::DeliveryRecord&> > (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/bind.hpp:296 #8 operator()<qpid::broker::DeliveryRecord> (__first=..., __last=..., __pred=...) at /usr/include/boost/bind/bind_template.hpp:32 #9 operator() (__first=..., __last=..., __pred=...) at qpid/broker/SemanticState.cpp:786 #10 std::__find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., __last=..., __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:222 #11 0x00007f7448731c99 in find_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., __last=..., __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:4248 #12 std::remove_if<std::_Deque_iterator<qpid::broker::DeliveryRecord, qpid::broker::DeliveryRecord&, qpid::broker::DeliveryRecord*>, qpid::broker::IsInSequenceSetAnd<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, qpid::broker::DeliveryRecord, qpid::broker::TransactionContext*>, boost::_bi::list2<boost::arg<1>, boost::_bi::value<qpid::broker::TransactionContext*> > > > > (__first=..., __last=..., __pred=...) at /usr/include/c++/4.4.7/bits/stl_algo.h:1153 #13 0x00007f744872981d in qpid::broker::SemanticState::accepted (this=0x7f7434005f38, commands=...) at qpid/broker/SemanticState.cpp:823 #14 0x00007f74481ad4d3 in invoke<qpid::framing::AMQP_ServerOperations::MessageHandler> (this=<value optimized out>, body=<value optimized out>) at qpid/framing/MessageAcceptBody.h:64 ..
Fixed http://git.app.eng.bos.redhat.com/git/rh-qpid.git/commit/?h=0.18-mrg-aconway-bz1121615 Treat priorities > 9 as priority 9.
Confirmed that this is not a problem on 0.22-mrg or trunk.
Rebased to include: 0632fb6 Bug 860691,QPID-4287: Poor performance when a priority queue with a ring queue policy has a large backlog New branch: http://git.app.eng.bos.redhat.com/git/rh-qpid.git/log/?h=0.18-mrg-aconway-bz1121615-2
Stress-tested 0.18-27 by sending & consuming messages with various (wrong) priority, no issue found. No impact to performance (wrt. orig bz 860691).
This issue is fixed. Verified OS: RHEL 6 x86_64, i386; RHEL 5 x86_64, i386 Packages: qpid-cpp-client-devel-docs-0.18-25.el6 qpid-cpp-client-devel-0.18-32.el6 qpid-cpp-client-ssl-0.18-32.el6 qpid-cpp-client-0.18-32.el6 qpid-cpp-debuginfo-0.18-32.el6 qpid-cpp-server-devel-0.18-32.el6 qpid-cpp-server-ssl-0.18-32.el6 qpid-cpp-server-store-0.18-32.el6 qpid-cpp-server-xml-0.18-32.el6 qpid-cpp-server-0.18-32.el6 qpid-tools-0.18-10.el6_4 -> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-1682.html