Bug 1291688

Summary: one-off (BZ1290841) - Lost message after suspending server with MDB consuming from HornetQ cluster
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Miroslav Novak <mnovak>
Component: HornetQAssignee: jboss-set
Status: CLOSED WONTFIX QA Contact: Miroslav Novak <mnovak>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.5CC: clichybi, csuconic, msvehla, toross
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-01 12:28:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Miroslav Novak 2015-12-15 12:53:09 UTC 
Description of problem:
There is missing one message in scenario with suspended EAP 6 server with MDB.

Test scenario - "suspend server in remote JCA topology in cluster"
1. Start servers 1 and 3 in HornetQ cluster (jms cluster) with deployed queues InQueue and OutQueue
2. Start servers 2 and 4 (mdb servers) which has configured resource adapter to connect to jms cluster
3. Start sending 10 000  messages to InQueue to jms cluster and deploy MDB to each mdb server
   - MDB consumes messages from InQueue and for each message sends a new message to OutQueue. MDB makes JNDI lookup for OutQueue for each message.
5. During processing of messages by MDB, suspend process of server 2 (with MDB) for 10 minutes and resume
6. Wait until all messages are processed from InQueue

Pass criteria:
Number of sent and received messages is the same. There are no lost or duplicated messages.

Test failed because there was 1 message sent to InQueue which did not have its corresponding message in OutQueue:
Lost message detected - there are not corresponding messages for: [ID:e613ea9c-a30f-11e5-9e86-f7c57f59158b]

Attaching traces from all 4 servers and clients (test-suite.log).

Link to failed Jenkins job - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-64x-patched-hornetq-remote-jca-suspend-tests/9/testReport/org.jboss.qa.hornetq.test.remote.jca/RemoteJcaWithSuspendTestCase/suspendOfMdbInClusterWithLodhLikeMdbMixMessages/


Version-Release number of selected component (if applicable):
EAP 6.4.5.CP + bz#1290841

Steps to Reproduce:
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/
git checkout cpu-load
# copy patched EAP 6.4.5.CP with BZ1290841 from messaging-10
scp 10.16.100.24:/home/mnovak/tmp/jboss-eap-6.4.5-patched-bz1290841.zip .

cd scripts
# this groovy script takes EAP 6.4.x zip and unzips to 4 directories, it also makes better "default" config, change path to EAP zip per your machine
groovy -DEAP_ZIP_URL=file://$PWD/../jboss-eap-6.4.5-patched-bz1290841.zip PrepareServers.groovy

export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=RemoteJcaWithSuspendTestCase#suspendOfMdbInClusterWithLodhLikeMdbMixMessages -DfailIfNoTests=false   | tee log

Actual results:
There is missing one message.

Expected results:
There should no lost messages.