Bug 1295808

RemoteJcaWithHighCpuLoadTestCase.loadInClusterWithRestartLodhMdb test failed with 1 duplicated message.

Test scenario - "100% cpu load in remote JCA topology in cluster with restart" - see one [1]
1. Start EAP servers 1 and 3 in HornetQ cluster (jms cluster) with deployed queues InQueue and OutQueue
2. Start other 2 EAP servers 2 and 4 (mdb servers) which has configured resource adapter to connect to jms cluster
3. Send 50 000 small to InQueue to jms cluster
4. Deploy MDB to mdb servers
   - MDB consumes messages from InQueue and for each message send new message to OutQueue. MDB makes JNDI lookup for OutQueue for each message.
5. During processing of messages by MDB cause 100% cpu load on server 2 
6. Wait until all messages are processed from InQueue
7. Restart all servers - clean shutdown of servers, first mdb servers and then jms cluster, then start servers from jms cluster and then mdb servers
8. Wait some time until the rest (if there are some) of messages are consumed from InQueue
9. Receive messages from OutQueue

Test failed because 1 message which was sent to InQueue was consumed 2 times by MDB and thus there are 2 messages in OutQueue which were created for message with messageID: ID:0c1613f0-b35a-11e5-9570-8fa72efbcb3f

Test detects those messages by comparing inMessageId property of each message in OutQueue at the end of the test and marks duplicted messages:
Duplicated message detected: [ID:ac087851-b35b-11e5-b0f3-fbef93c87d4b]

By quick check of server logs there is HeuristicMixedException in log of server 2:
22:15:09,505 WARN  [org.hornetq.ra] (Thread-65 (HornetQ-client-global-threads-1484100913)) HQ152006: Unable to call after delivery: javax.resource.spi.LocalTransactionException: javax.transaction.HeuristicMixedException
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_65]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_65]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_65]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_65]
        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60)
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:136)
        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)
        at com.sun.proxy.$Proxy12.afterDelivery(Unknown Source)
        at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:349) [hornetq-ra-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]
Caused by: javax.transaction.HeuristicMixedException
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1197) [jbossjts-jacorb-4.17.30.Final-redhat-1.jar:4.17.30.Final-redhat-1]
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) [jbossjts-jacorb-4.17.30.Final-redhat-1.jar:4.17.30.Final-redhat-1]
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:72)
        ... 16 more


[1] https://mojo.redhat.com/docs/DOC-1058185
[2] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP-6.4.x-CP/view/EAP-6.4.x-CP-jobs/job/eap-64x-patched-hornetq-cpu-load-tests-2/lastCompletedBuild/testReport/org.jboss.qa.hornetq.test.remote.jca/RemoteJcaWithHighCpuLoadTestCase/loadInClusterWithRestartLodhMdb/