Bug 1295808 - one-off (BZ1290841) - Duplicated message with MDB consuming from remote HornetQ cluster with 100% CPU load
Summary: one-off (BZ1290841) - Duplicated message with MDB consuming from remote Horne...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: HornetQ
Version: 6.4.5
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Clebert Suconic
QA Contact: Miroslav Novak
URL:
Whiteboard:
Depends On: 1290841
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-05 14:05 UTC by Miroslav Novak
Modified: 2016-01-08 12:19 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-01-08 12:19:05 UTC
Type: Bug


Attachments (Terms of Use)

Description Miroslav Novak 2016-01-05 14:05:48 UTC
RemoteJcaWithHighCpuLoadTestCase.loadInClusterWithRestartLodhMdb test failed with 1 duplicated message.

Test scenario - "100% cpu load in remote JCA topology in cluster with restart" - see one [1]
1. Start EAP servers 1 and 3 in HornetQ cluster (jms cluster) with deployed queues InQueue and OutQueue
2. Start other 2 EAP servers 2 and 4 (mdb servers) which has configured resource adapter to connect to jms cluster
3. Send 50 000 small to InQueue to jms cluster
4. Deploy MDB to mdb servers
   - MDB consumes messages from InQueue and for each message send new message to OutQueue. MDB makes JNDI lookup for OutQueue for each message.
5. During processing of messages by MDB cause 100% cpu load on server 2 
6. Wait until all messages are processed from InQueue
7. Restart all servers - clean shutdown of servers, first mdb servers and then jms cluster, then start servers from jms cluster and then mdb servers
8. Wait some time until the rest (if there are some) of messages are consumed from InQueue
9. Receive messages from OutQueue

Test failed because 1 message which was sent to InQueue was consumed 2 times by MDB and thus there are 2 messages in OutQueue which were created for message with messageID: ID:0c1613f0-b35a-11e5-9570-8fa72efbcb3f

Test detects those messages by comparing inMessageId property of each message in OutQueue at the end of the test and marks duplicted messages:
Duplicated message detected: [ID:ac087851-b35b-11e5-b0f3-fbef93c87d4b]

By quick check of server logs there is HeuristicMixedException in log of server 2:
22:15:09,505 WARN  [org.hornetq.ra] (Thread-65 (HornetQ-client-global-threads-1484100913)) HQ152006: Unable to call after delivery: javax.resource.spi.LocalTransactionException: javax.transaction.HeuristicMixedException
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:82)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_65]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_65]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_65]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_65]
        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60)
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:136)
        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)
        at com.sun.proxy.$Proxy12.afterDelivery(Unknown Source)
        at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:349) [hornetq-ra-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) [hornetq-core-client-2.3.25.SP6-redhat-1-bz-1290841.jar:]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]
Caused by: javax.transaction.HeuristicMixedException
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1197) [jbossjts-jacorb-4.17.30.Final-redhat-1.jar:4.17.30.Final-redhat-1]
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) [jbossjts-jacorb-4.17.30.Final-redhat-1.jar:4.17.30.Final-redhat-1]
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:72)
        ... 16 more


[1] https://mojo.redhat.com/docs/DOC-1058185
[2] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP-6.4.x-CP/view/EAP-6.4.x-CP-jobs/job/eap-64x-patched-hornetq-cpu-load-tests-2/lastCompletedBuild/testReport/org.jboss.qa.hornetq.test.remote.jca/RemoteJcaWithHighCpuLoadTestCase/loadInClusterWithRestartLodhMdb/


Note You need to log in before you can comment on or make changes to this bug.