Bug 1292388

Summary: one-off (BZ1290841) - Not processed messages when suspending "jms" server in HornetQ cluster
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Miroslav Novak <mnovak>
Component: HornetQAssignee: jboss-set
Status: NEW --- QA Contact: Miroslav Novak <mnovak>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.5CC: clichybi, csuconic, msvehla, toross
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Miroslav Novak 2015-12-17 04:48:01 EST
There are missing 5 messages in scenario with suspended EAP 6 server from which are consuming remote MDBs.

Test scenario - "suspend server in remote JCA topology in cluster" [1]
1. Start servers 1 and 3 in HornetQ cluster (jms cluster) with deployed queues InQueue and OutQueue
2. Start servers 2 and 4 (mdb servers) which has configured resource adapter to connect to jms cluster
3. Start sending 10 000  messages to InQueue to jms cluster and deploy MDB to each mdb server
   - MDB consumes messages from InQueue and for each message sends a new message to OutQueue. MDB makes JNDI lookup for OutQueue for each message.
5. During processing of messages by MDB, suspend process of server 3 (jms server) for 10 minutes and resume
6. Wait until all messages are processed from InQueue
7. Receive messages from OutQueue

Test failed because there were 5 messages sent to InQueue which did not have its corresponding message in OutQueue:
Lost message detected - there are not corresponding messages for: [ID:fafd2d81-a416-11e5-8d99-2d9061c53a0c, ID:ff18cf91-a416-11e5-8d99-2d9061c53a0c, ID:ff438995-a416-11e5-8d99-2d9061c53a0c, ID:0bf3860c-a417-11e5-8d99-2d9061c53a0c, ID:16f472fe-a417-11e5-8d99-2d9061c53a0c]

I've checked the logs and see that messages are "stuck" in InQueue the whole time. Strange thing is that once receiver (step 7.) finishes consumption of messages from OutQueue then MDBs process those 5 messages and they get to OutQueue. (I can see them in OutQueue when printing content of journals from server1 and server3 after the test.)

To download traces from all 4 servers, journal directories, configuration and clients (test-suite.log) - see comments below to download it
Link to failed Jenkins job - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-64x-patched-hornetq-remote-jca-suspend-tests-2/11/
Complete "info" log from the test - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-64x-patched-hornetq-remote-jca-suspend-tests-2/11/consoleText

[1] https://mojo.redhat.com/docs/DOC-1058185