Bug 606357

Summary: Support for dropping messages first by priority, then by age within priority
Product: Red Hat Enterprise MRG Reporter: Andy Goldstein <agoldste>
Component: qpid-cppAssignee: Gordon Sim <gsim>
Status: CLOSED ERRATA QA Contact: Petra Svobodová <psvobodo>
Severity: medium Docs Contact:
Priority: medium    
Version: DevelopmentCC: agoldste, freznice, iboverma, jeder, jneedle, jross, jsarenik, mhusnain, psvobodo
Target Milestone: 2.0Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qpid-cpp-0.9.1079953 Doc Type: Enhancement
Doc Text:
Cause: A queue can be configured to remain within a given size by removing the oldest messages when needed to make space for newer messages. However this took no account of message priority. Consequence: In some cases messages that were more valuable would be removed while there were less valuable messages left on the queue. Change: If such a queue has also been configured to recognise message priority (i.e. qpid.priorities is specified) then it will discard lower priority messages first. Result: The application can configure the queue (and mark the messages) such that less useful/less important messages are discarded first. Release Note Entry: When the Messaging component runs out of space, it must remove older messages to make space for new incoming messages. The messages to be deleted first are selected using an algorithm that takes into account both the priority and the age of the message. This algorithm allows the oldest of the low priority messages to be considered expendable while the high priority messages are preserved.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-06-23 15:47:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
The behavior reproducer none

Description Andy Goldstein 2010-06-21 13:30:56 UTC
ITC has the following requirement: When the messaging system runs out of resources and needs to drop messages, it should drop messages by priority (lowest to highest) and by age within a priority (oldest to newest).

I don't believe this is supported by MRG at the moment.  We would like to see this added to MRG so we can meet all ITC requirements.

In terms of criticality, I would say that both priority queues (bug 453538) and automatic queue deletion (bug 585844) are more important than this one.

Comment 2 Gordon Sim 2011-03-02 13:39:00 UTC
For ring queues configured to recognise different priority levels (qpid.priorities is set), the lowest priority messages will be deleted first when the maximum size is reached.

Comment 3 Gordon Sim 2011-03-07 14:41:18 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
A queue can be configured to remain within a given size by removing the oldest messages when needed to make space for newer messages. However this took no account of message priority.

Consequence:
In some cases messages that were more valuable would be removed while there were less valuable messages left on the queue.

Change:
If such a queue has also been configured to recognise message priority (i.e. qpid.priorities is specified) then it will discard lower priority messages first.

Result:
The application can configure the queue (and mark the messages) such that less useful/less important messages are discarded first.

Comment 5 Petra Svobodová 2011-05-18 14:09:21 UTC
Created attachment 499601 [details]
The behavior reproducer

Queues can have defined their priority levels.
A queue with declared "limit-policy" option value "ring" after its overfilling (if its limit is exceeded) removes a message with the lowest priority value or the latest message; this is expected behaviour.

Queues with "limit-policy" value "ring-strict" have the same behaviour. 
Is it right? Should the "ring-strict" queues remove messages before their accepting? This seems to be in contrary to documentation

Messaging_User_Guide chapter 3

(search for 'Similar to the ring policy, but will not remove messages that have not yet been accepted')

Could you possibly double-check whether described ring-strict behavior is correct, please?


Raising NEEDINFO.


The testing steps:
  service qpidd restart
  ./bug606357_prior_mod &>log_ring
  service qpidd restart
  ./bug606357_prior_mod ring-strict &>log_ring_strict

Comment 6 Gordon Sim 2011-05-18 14:22:04 UTC
The difference with ring-strict is that if the message to be removed to make space for a new message has been delivered to a consumer but not yet acknowledged (and thus dequeued), it should not be replaced and the enqueue should fail.

Your reproducer appears to use drain (in browse), so I don't see how it could trigger the ring-strict specific behaviour. Am I missing something?

Comment 7 Gordon Sim 2011-05-18 14:37:31 UTC
One way to see the ring-strict behaviour is to use the qpid-receive test program (currently only in the svn tree I think?) with options designed to prevent immediate acknowledgement:

E.g.

qpid-config add queue my-queue --limit-policy ring-strict --max-queue-count 5
for i in `seq 1 10`; do spout --content "msg-$i" my-queue; done

then start (and leave running):

qpid-receive --ack-frequency 10 --messages 10 --address my-queue -f

and in another window:

spout my-queue

That last spout will fail for the ring-strict policy. However if you kill the qpid-receive program and try again it will work.

Comment 8 Petra Svobodová 2011-05-19 08:26:25 UTC
Gordon, thank you for explanation; the "ring-strict" queues behave as you write. The issue is clearer for me now.

Messages in queues are dropped first by priority, then by age; this is expected behaviour.

Tested packages:

[root@dhcp-26-233 ~]# rpm -qa | grep -E '(qpid|qmf|sesame)' | sort
python-qpid-0.10-1.el5
python-qpid-qmf-0.10-6.el5
qpid-cpp-client-0.10-7.el5
qpid-cpp-client-devel-0.10-7.el5
qpid-cpp-client-devel-docs-0.10-7.el5
qpid-cpp-client-ssl-0.10-7.el5
qpid-cpp-mrg-debuginfo-0.10-4.el5
qpid-cpp-server-0.10-7.el5
qpid-cpp-server-cluster-0.10-7.el5
qpid-cpp-server-devel-0.10-7.el5
qpid-cpp-server-ssl-0.10-7.el5
qpid-cpp-server-store-0.10-7.el5
qpid-cpp-server-xml-0.10-7.el5
qpid-java-client-0.10-6.el5
qpid-java-common-0.10-6.el5
qpid-java-example-0.10-6.el5
qpid-qmf-0.10-6.el5
qpid-qmf-devel-0.10-6.el5
qpid-tools-0.10-5.el5
rh-qpid-cpp-tests-0.10-7.el5
sesame-0.10-1.el5

verified on RHEL 5.6 / 6.1rc i[36]86 / x86_64

--> VERIFIED

Comment 9 Misha H. Ali 2011-05-31 06:06:00 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -8,4 +8,8 @@
 If such a queue has also been configured to recognise message priority (i.e. qpid.priorities is specified) then it will discard lower priority messages first.
 
 Result:
-The application can configure the queue (and mark the messages) such that less useful/less important messages are discarded first.+The application can configure the queue (and mark the messages) such that less useful/less important messages are discarded first.
+
+Release Note Entry:
+
+When the Messaging component runs out of space, it must remove older messages to make space for new incoming messages. The messages to be deleted first are selected using an algorithm that takes into account both the priority and the age of the message. This algorithm allows the oldest of the low priority messages to be considered expendable while the high priority messages are preserved.

Comment 10 Misha H. Ali 2011-06-06 03:15:54 UTC
Technical note can be viewed in the release notes for 2.0 at the documentation stage here:

http://documentation-stage.bne.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/2.0/html-single/MRG_Release_Notes/index.html#tabl-MRG_Release_Notes-RHM_Update_Notes-RHM_Update_Notes

Comment 11 errata-xmlrpc 2011-06-23 15:47:41 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2011-0890.html