Bug 501552 - java client releases messages in an unpredictable order on rollback
java client releases messages in an unpredictable order on rollback
Status: CLOSED ERRATA
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-java (Show other bugs)
1.1.1
All Linux
medium Severity medium
: 1.1.2
: ---
Assigned To: Rafael H. Schloming
Jan Sarenik
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-19 13:43 EDT by Rafael H. Schloming
Modified: 2014-12-01 18:14 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-06-12 13:39:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch with fix and test (4.74 KB, patch)
2009-05-19 13:45 EDT, Rafael H. Schloming
no flags Details | Diff
Test case provided by customer (2.93 KB, application/x-gzip)
2009-05-25 10:55 EDT, Rajith Attapattu
no flags Details

  None (edit)
Description Rafael H. Schloming 2009-05-19 13:43:51 EDT
Description of problem:

When messages are released on rollback, the order of release is unpredictable. This results in messages being requeued in an unpredictable order even if there is only one consumer. The unpredictability is because messages that have yet to be dispatched end up being released in the dispatch thread and this may sometimes happen before or after the messages arrive in the consumers fetch queue.

How reproducible:

The problem is intermittent since it depends on the thread timings. It may take multiple trials to reproduce.

Steps to Reproduce:
1. Populate a queue with 5 to 10 messages. The exact number may effect the timing, and so some tweaking might be necessary, but it has to be more than one. The message content should be numbered with the order the messages are placed on the queue.
2. From the JMS client, create a transactional session.
3. Using the transactional session, create a consumer on the queue.
4. In a loop, pull one message off of the queue using receive(), assert that this is the first message placed on the queue (using the numbered content), and then immediately rollback the session.
  
Actual results:

The assert should eventually fail because the messages will eventually be reordered if the bug is present.

Expected results:

The loop should be able to proceed indefinitely without the assertion failing, assuming there are no other consumers pulling from the same queue.

Additional info:

The patch includes a unit test that performs the steps described above. It stops after 10 iterations, so it won't reproduce the issue 100% of the time.
Comment 1 Rafael H. Schloming 2009-05-19 13:45:16 EDT
Created attachment 344656 [details]
patch with fix and test
Comment 4 Jan Sarenik 2009-05-25 08:00:57 EDT
How should I reproduce it and test it is corrected already?

If I do 'ant test -Dtest=RollbackOrderTest' in qpid-svn/java,
the test runs but fails both with old qpid-java-client
package and latest one. So I assume there is something
wrong with what I am doing.
Comment 5 Rajith Attapattu 2009-05-25 10:44:36 EDT
Use -Dprofile=cpp when running the svn tests, other wise the test will be run against the in-vm java broker.
Comment 6 Rajith Attapattu 2009-05-25 10:55:13 EDT
Created attachment 345327 [details]
Test case provided by customer

The attached tar file contains a test case used by the customer to demonstrate the rollback issue.
Comment 7 Jan Sarenik 2009-05-26 07:04:50 EDT
Hello Rajith!
Testing is in progress, but I came upon few problems:

  1. The first one is not so vital, but I would like
     to know, whether there is possibility to run
     a java class without moving it to full package
     path, e.g. I wanted to run "java GSTests/Publisher"
     which did not work, but after moving Publisher to
     org/apache/qpid/example/jmsexample/pubsub/ it works.

  2. On qpid-java-client-0.5.751061-2.el5 (1.1.1) when using
     your attached test and grepping its output for
     "^message", I am getting "message 1" few times
     (random) but then also "message 2" and 3 4 5 6.

     On qpid-java-client-0.5.751061-4.el5 (from mrg-devel
     repo), I am getting always "message 1". Is this the
     right way to prove it works as expected?

  3. From time to time I get message on stderr of the
     Subscriber class:

-----------------------------------------------------------------
Exception in thread "Dispatcher-Channel-1" java.lang.IllegalStateException: dispatcher is not started
        at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:2683)
        at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:55)
        at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:2814)
        at java.lang.Thread.run(Thread.java:619)
-----------------------------------------------------------------

Now I run a longer test in which I would like to see if
there is a chance to get messages other than "message 1"
on the new packages... I have not read the code too much
to understand whether it is possible at all, so please
comment on my 3 points first.

  Thanks, Jasan
Comment 8 Jan Sarenik 2009-05-26 07:17:36 EDT
By the way, I am using java-1.6.0-sun-devel package for java.
Comment 9 Rafael H. Schloming 2009-05-26 08:11:30 EDT
I believe the issue with (1) is that the Publisher and Subscriber classes are declared in the org.apache.qpid.example.jmsexample.pubsub packages. If you modify the package declarations in the source file, you should be able to compile and run them without placing them in the examples directory. You will just need to include the qpid client jar in your classpath when you compile and run.

As for (2), that is exactly as expected.

I noticed (3) as well. We should probably file that as an independent issue.

Probably the easiest way to run a longer test would be to modify RollbackOrderTest and change the loop count to some large number.
Comment 10 Jan Sarenik 2009-05-27 10:11:16 EDT
Can you please build fixed package for RHEL4 as well?
Comment 13 Jan Sarenik 2009-06-01 23:57:50 EDT
------------------------------------------------------------------
root@rhel4:~/bz501552# sh runit.sh
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
message 1
message 5
message 5
message 5
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 1
message 5
message 5
message 5
message 5
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 2
message 6
message 6
message 6
message 6
message 6
message 6
message 4
message 4
message 4
message 4
message 4
message 4
message 4
------------------------------------------------------------------

But on RHEL5 with package rev -4 it is message 1 all the time.
Comment 15 Jan Sarenik 2009-06-03 03:10:35 EDT
Verified on
 RHEL4
  qpid-java-client-0.5.751061-4.el4

 RHEL5
  qpid-java-client-0.5.751061-5.el5
Comment 17 errata-xmlrpc 2009-06-12 13:39:46 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1097.html

Note You need to log in before you can comment on or make changes to this bug.