Red Hat Bugzilla – Bug 509449
JMS client releases messages in an unpredictable order on recover
Last modified: 2010-10-14 12:15:59 EDT
Description of problem: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue. Version-Release number of selected component (if applicable): 1.1.6 How reproducible: The problem is intermittent since it depends on the thread timings. It may take multiple trials to reproduce. Steps to 1. Populate a queue with 5 to 10 messages. The exact number may effect the timing, and so some tweaking might be necessary, but it has to be more than one. The message content should be numbered with the order the messages are placed on the queue. 2. From the JMS client, create a session and a consumer on the queue. 3. Receive a message and assert that it is the correct message based on the seq. If you haven't seen the same message 5 times call recover, if not ack the message. You can use a map with message-id to keep count the number of times you have seen it. (A standalone test case will be attached) Actual results: Eventually you will messages being received out of order. Ex. 0) DelayedAckListener: - Ignored Body: Msg0 1) DelayedAckListener: - Ignored Body: Msg0 2) DelayedAckListener: - Ignored Body: Msg0 3) DelayedAckListener: - Ignored Body: Msg0 4) DelayedAckListener: - Ignored Body: Msg0 DelayedAckListener: - Accepted Body: Msg0 Listener will ignore message ID:b6317b8d-4f2a-3f24-a65f-71fc491d0820 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg5 1) DelayedAckListener: - Ignored Body: Msg5 2) DelayedAckListener: - Ignored Body: Msg5 3) DelayedAckListener: - Ignored Body: Msg5 4) DelayedAckListener: - Ignored Body: Msg5 DelayedAckListener: - Accepted Body: Msg5 Listener will ignore message ID:9a7d5973-bc44-3f06-a8bb-19639628b95d 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg6 1) DelayedAckListener: - Ignored Body: Msg6 2) DelayedAckListener: - Ignored Body: Msg6 3) DelayedAckListener: - Ignored Body: Msg6 4) DelayedAckListener: - Ignored Body: Msg6 Listener will ignore message ID:df2336ae-b909-33d4-89d2-3bdeb50bfcf4 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg1 1) DelayedAckListener: - Ignored Body: Msg1 2) DelayedAckListener: - Ignored Body: Msg1 3) DelayedAckListener: - Ignored Body: Msg1 4) DelayedAckListener: - Ignored Body: Msg1 DelayedAckListener: - Accepted Body: Msg1 Listener will ignore message ID:4b1ab213-f9ea-3b4d-88a8-29775e528062 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg2 1) DelayedAckListener: - Ignored Body: Msg2 2) DelayedAckListener: - Ignored Body: Msg2 3) DelayedAckListener: - Ignored Body: Msg2 4) DelayedAckListener: - Ignored Body: Msg2 DelayedAckListener: - Accepted Body: Msg2 Listener will ignore message ID:50c4eea9-1b7b-3f9d-a752-a64b8bb9f8e2 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg3 1) DelayedAckListener: - Ignored Body: Msg3 2) DelayedAckListener: - Ignored Body: Msg3 3) DelayedAckListener: - Ignored Body: Msg3 4) DelayedAckListener: - Ignored Body: Msg3 DelayedAckListener: - Accepted Body: Msg3 Listener will ignore message ID:737065af-ded0-3d3f-aac5-ff763b72f16e 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg4 1) DelayedAckListener: - Ignored Body: Msg4 2) DelayedAckListener: - Ignored Body: Msg4 3) DelayedAckListener: - Ignored Body: Msg4 4) DelayedAckListener: - Ignored Body: Msg4 DelayedAckListener: - Accepted Body: Msg4 Listener will ignore message ID:46c933e6-0faa-3002-80aa-560802eef2b4 5 times before acknowledging 0) DelayedAckListener: - Ignored Body: Msg7 1) DelayedAckListener: - Ignored Body: Msg7 2) DelayedAckListener: - Ignored Body: Msg7 3) DelayedAckListener: - Ignored Body: Msg7 4) DelayedAckListener: - Ignored Body: Msg7 DelayedAckListener: - Accepted Body: Msg7 DelayedAckListener: - Accepted Body: Msg6 Expected results: Should receive messages in order
This is fixed in rev 934236 in Qpid trunk.
Added two test cases in rev 939410 in Qpid trunk. You could run them by doing, "ant test -Dprofile=cpp -Dtest=RecoverTest" The two specific test cases are, testOderingWithAsyncConsumer testOderingWithSyncConsumer
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: JMS client releases messages in an unpredictable order on recover. Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue. Fix: The messages are now released in the order they arrived in the queue. Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,10 +1 @@ -Cause: JMS client releases messages in an unpredictable order on recover. +Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.- -Consequence: When messages are released on recover, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there -is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue. - - -Fix: The messages are now released in the order they arrived in the queue. - - -Result: If client calls recover messages are not released in the order they arrived in the queue, hence the next time it's sent to the client the original ordering is preserved. Please note this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being requeued in an unpredictable order even if there was only one consumer. The unpredictability was caused by the messages that had yet to be dispatched ended up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.+Previously, a JMS client released messages in an unpredictable order when trying to recover. This resulted in messages being re-queued in an unpredictable order even if there was only one consumer. This unpredictability was due to undispatched messages ending up being released in the dispatch thread. This may have happened before or after the messages arrived in the consumers fetch queue. With this update, the messages are now released in the order they arrived in the queue. If a client calls for a recover, messages are not released in the order they arrived in the queue, hence, the next time they are sent to the client, the original ordering is preserved. Please note that this is only valid if there is only one consumer present on the queue. If there are multiple consumers the ordering cannot be guaranteed.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0773.html