Bug 509449 - JMS client releases messages in an unpredictable order on recover
JMS client releases messages in an unpredictable order on recover
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-java (Show other bugs)
1.1.2
All Linux
low Severity medium
: 1.3
: ---
Assigned To: Rajith Attapattu
MRG Quality Engineering
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-02 16:23 EDT by Rajith Attapattu
Modified: 2010-10-14 12:15 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being re-queued in an unpredictable order even if there was only one consumer. This unpredictability was due to undispatched messages ending up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-10-14 12:15:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rajith Attapattu 2009-07-02 16:23:13 EDT
Description of problem:
When messages are released on recover, the order of release is unpredictable.
This results in messages being requeued in an unpredictable order even if there
is only one consumer. The unpredictability is because messages that have yet to
be dispatched end up being released in the dispatch thread and this may
sometimes happen before or after the messages arrive in the consumers fetch
queue.

Version-Release number of selected component (if applicable):
1.1.6

How reproducible:
The problem is intermittent since it depends on the thread timings. It may take
multiple trials to reproduce.

Steps to 
1. Populate a queue with 5 to 10 messages. The exact number may effect the
timing, and so some tweaking might be necessary, but it has to be more than
one. The message content should be numbered with the order the messages are
placed on the queue.
2. From the JMS client, create a session and a consumer on the queue.
3. Receive a message and assert that it is the correct message based on the seq. 
If you haven't seen the same message 5 times call recover, if not ack the message. You can use a map with message-id to keep count the number of times you have seen it.
  
(A standalone test case will be attached)

Actual results:
Eventually you will messages being received out of order.
Ex.

0) DelayedAckListener: - Ignored 
	Body: Msg0
1) DelayedAckListener: - Ignored 
	Body: Msg0
2) DelayedAckListener: - Ignored 
	Body: Msg0
3) DelayedAckListener: - Ignored 
	Body: Msg0
4) DelayedAckListener: - Ignored 
	Body: Msg0
DelayedAckListener: - Accepted 
	Body: Msg0
Listener will ignore message ID:b6317b8d-4f2a-3f24-a65f-71fc491d0820 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg5
1) DelayedAckListener: - Ignored 
	Body: Msg5
2) DelayedAckListener: - Ignored 
	Body: Msg5
3) DelayedAckListener: - Ignored 
	Body: Msg5
4) DelayedAckListener: - Ignored 
	Body: Msg5
DelayedAckListener: - Accepted 
	Body: Msg5
Listener will ignore message ID:9a7d5973-bc44-3f06-a8bb-19639628b95d 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg6
1) DelayedAckListener: - Ignored 
	Body: Msg6
2) DelayedAckListener: - Ignored 
	Body: Msg6
3) DelayedAckListener: - Ignored 
	Body: Msg6
4) DelayedAckListener: - Ignored 
	Body: Msg6
Listener will ignore message ID:df2336ae-b909-33d4-89d2-3bdeb50bfcf4 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg1
1) DelayedAckListener: - Ignored 
	Body: Msg1
2) DelayedAckListener: - Ignored 
	Body: Msg1
3) DelayedAckListener: - Ignored 
	Body: Msg1
4) DelayedAckListener: - Ignored 
	Body: Msg1
DelayedAckListener: - Accepted 
	Body: Msg1
Listener will ignore message ID:4b1ab213-f9ea-3b4d-88a8-29775e528062 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg2
1) DelayedAckListener: - Ignored 
	Body: Msg2
2) DelayedAckListener: - Ignored 
	Body: Msg2
3) DelayedAckListener: - Ignored 
	Body: Msg2
4) DelayedAckListener: - Ignored 
	Body: Msg2
DelayedAckListener: - Accepted 
	Body: Msg2
Listener will ignore message ID:50c4eea9-1b7b-3f9d-a752-a64b8bb9f8e2 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg3
1) DelayedAckListener: - Ignored 
	Body: Msg3
2) DelayedAckListener: - Ignored 
	Body: Msg3
3) DelayedAckListener: - Ignored 
	Body: Msg3
4) DelayedAckListener: - Ignored 
	Body: Msg3
DelayedAckListener: - Accepted 
	Body: Msg3
Listener will ignore message ID:737065af-ded0-3d3f-aac5-ff763b72f16e 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg4
1) DelayedAckListener: - Ignored 
	Body: Msg4
2) DelayedAckListener: - Ignored 
	Body: Msg4
3) DelayedAckListener: - Ignored 
	Body: Msg4
4) DelayedAckListener: - Ignored 
	Body: Msg4
DelayedAckListener: - Accepted 
	Body: Msg4
Listener will ignore message ID:46c933e6-0faa-3002-80aa-560802eef2b4 5 times before acknowledging
0) DelayedAckListener: - Ignored 
	Body: Msg7
1) DelayedAckListener: - Ignored 
	Body: Msg7
2) DelayedAckListener: - Ignored 
	Body: Msg7
3) DelayedAckListener: - Ignored 
	Body: Msg7
4) DelayedAckListener: - Ignored 
	Body: Msg7
DelayedAckListener: - Accepted 
	Body: Msg7
DelayedAckListener: - Accepted 
	Body: Msg6

Expected results:
Should receive messages in order
Comment 2 Rajith Attapattu 2010-04-22 21:24:53 EDT
This is fixed in rev 934236 in Qpid trunk.
Comment 3 Rajith Attapattu 2010-04-29 14:12:19 EDT
Added two test cases in rev 939410 in Qpid trunk.
You could run them by doing,
"ant test -Dprofile=cpp -Dtest=RecoverTest"

The two specific test cases are,
testOderingWithAsyncConsumer
testOderingWithSyncConsumer
Comment 5 Rajith Attapattu 2010-10-07 14:07:34 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: JMS client releases messages in an unpredictable order on recover.

Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there
is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue.


Fix: The messages are now released in the order they arrived in the queue.


Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Comment 6 Martin Prpic 2010-10-10 03:59:55 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,10 +1 @@
-Cause: JMS client releases messages in an unpredictable order on recover.
+Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.-
-Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there
-is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue.
-
-
-Fix: The messages are now released in the order they arrived in the queue.
-
-
-Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Comment 7 Douglas Silas 2010-10-11 07:53:25 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.+Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being re-queued in an unpredictable order even if there was only one consumer. This unpredictability was due to undispatched messages ending up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Comment 9 errata-xmlrpc 2010-10-14 12:15:59 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html

Note You need to log in before you can comment on or make changes to this bug.