Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 509449

Summary: JMS client releases messages in an unpredictable order on recover
Product: Red Hat Enterprise MRG Reporter: Rajith Attapattu <rattapat+nobody>
Component: qpid-javaAssignee: Rajith Attapattu <rattapat+nobody>
Status: CLOSED ERRATA QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 1.1.2CC: gsim
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being re-queued in an unpredictable order even if there was only one consumer. This unpredictability was due to undispatched messages ending up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-14 16:15:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rajith Attapattu 2009-07-02 20:23:13 UTC
Description of problem:
When messages are released on recover, the order of release is unpredictable.
This results in messages being requeued in an unpredictable order even if there
is only one consumer. The unpredictability is because messages that have yet to
be dispatched end up being released in the dispatch thread and this may
sometimes happen before or after the messages arrive in the consumers fetch
queue.

Version-Release number of selected component (if applicable):
1.1.6

How reproducible:
The problem is intermittent since it depends on the thread timings. It may take
multiple trials to reproduce.

Steps to 
1. Populate a queue with 5 to 10 messages. The exact number may effect the
timing, and so some tweaking might be necessary, but it has to be more than
one. The message content should be numbered with the order the messages are
placed on the queue.
2. From the JMS client, create a session and a consumer on the queue.
3. Receive a message and assert that it is the correct message based on the seq. 
If you haven't seen the same message 5 times call recover, if not ack the message. You can use a map with message-id to keep count the number of times you have seen it.
  
(A standalone test case will be attached)

Actual results:
Eventually you will messages being received out of order.
Ex.

0) DelayedAckListener: - Ignored 
	Body: Msg0
1) DelayedAckListener: - Ignored 
	Body: Msg0
2) DelayedAckListener: - Ignored 
	Body: Msg0
3) DelayedAckListener: - Ignored 
	Body: Msg0
4) DelayedAckListener: - Ignored 
	Body: Msg0
DelayedAckListener: - Accepted 
	Body: Msg0
Listener will ignore message ID:b6317b8d-4f2a-3f24-a65f-71fc491d0820 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg5
1) DelayedAckListener: - Ignored 
	Body: Msg5
2) DelayedAckListener: - Ignored 
	Body: Msg5
3) DelayedAckListener: - Ignored 
	Body: Msg5
4) DelayedAckListener: - Ignored 
	Body: Msg5
DelayedAckListener: - Accepted 
	Body: Msg5
Listener will ignore message ID:9a7d5973-bc44-3f06-a8bb-19639628b95d 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg6
1) DelayedAckListener: - Ignored 
	Body: Msg6
2) DelayedAckListener: - Ignored 
	Body: Msg6
3) DelayedAckListener: - Ignored 
	Body: Msg6
4) DelayedAckListener: - Ignored 
	Body: Msg6
Listener will ignore message ID:df2336ae-b909-33d4-89d2-3bdeb50bfcf4 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg1
1) DelayedAckListener: - Ignored 
	Body: Msg1
2) DelayedAckListener: - Ignored 
	Body: Msg1
3) DelayedAckListener: - Ignored 
	Body: Msg1
4) DelayedAckListener: - Ignored 
	Body: Msg1
DelayedAckListener: - Accepted 
	Body: Msg1
Listener will ignore message ID:4b1ab213-f9ea-3b4d-88a8-29775e528062 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg2
1) DelayedAckListener: - Ignored 
	Body: Msg2
2) DelayedAckListener: - Ignored 
	Body: Msg2
3) DelayedAckListener: - Ignored 
	Body: Msg2
4) DelayedAckListener: - Ignored 
	Body: Msg2
DelayedAckListener: - Accepted 
	Body: Msg2
Listener will ignore message ID:50c4eea9-1b7b-3f9d-a752-a64b8bb9f8e2 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg3
1) DelayedAckListener: - Ignored 
	Body: Msg3
2) DelayedAckListener: - Ignored 
	Body: Msg3
3) DelayedAckListener: - Ignored 
	Body: Msg3
4) DelayedAckListener: - Ignored 
	Body: Msg3
DelayedAckListener: - Accepted 
	Body: Msg3
Listener will ignore message ID:737065af-ded0-3d3f-aac5-ff763b72f16e 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg4
1) DelayedAckListener: - Ignored 
	Body: Msg4
2) DelayedAckListener: - Ignored 
	Body: Msg4
3) DelayedAckListener: - Ignored 
	Body: Msg4
4) DelayedAckListener: - Ignored 
	Body: Msg4
DelayedAckListener: - Accepted 
	Body: Msg4
Listener will ignore message ID:46c933e6-0faa-3002-80aa-560802eef2b4 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg7
1) DelayedAckListener: - Ignored 
	Body: Msg7
2) DelayedAckListener: - Ignored 
	Body: Msg7
3) DelayedAckListener: - Ignored 
	Body: Msg7
4) DelayedAckListener: - Ignored 
	Body: Msg7
DelayedAckListener: - Accepted 
	Body: Msg7
DelayedAckListener: - Accepted 
	Body: Msg6

Expected results:
Should receive messages in order

Comment 2 Rajith Attapattu 2010-04-23 01:24:53 UTC
This is fixed in rev 934236 in Qpid trunk.

Comment 3 Rajith Attapattu 2010-04-29 18:12:19 UTC
Added two test cases in rev 939410 in Qpid trunk.
You could run them by doing,
"ant test -Dprofile=cpp -Dtest=RecoverTest"

The two specific test cases are,
testOderingWithAsyncConsumer
testOderingWithSyncConsumer

Comment 5 Rajith Attapattu 2010-10-07 18:07:34 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: JMS client releases messages in an unpredictable order on recover.

Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there
is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue.


Fix: The messages are now released in the order they arrived in the queue.


Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.

Comment 6 Martin Prpič 2010-10-10 07:59:55 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,10 +1 @@
-Cause: JMS client releases messages in an unpredictable order on recover.
+Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.-
-Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there
-is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue.
-
-
-Fix: The messages are now released in the order they arrived in the queue.
-
-
-Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.

Comment 7 Douglas Silas 2010-10-11 11:53:25 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.+Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being re-queued in an unpredictable order even if there was only one consumer. This unpredictability was due to undispatched messages ending up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.

Comment 9 errata-xmlrpc 2010-10-14 16:15:59 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html