Bug 1292388 - one-off (BZ1290841) - Not processed messages when suspending "jms" server in HornetQ cluster
one-off (BZ1290841) - Not processed messages when suspending "jms" server in ...
Status: NEW
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: HornetQ (Show other bugs)
6.4.5
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: jboss-set
Miroslav Novak
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-17 04:48 EST by Miroslav Novak
Modified: 2015-12-17 05:36 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Miroslav Novak 2015-12-17 04:48:01 EST
There are missing 5 messages in scenario with suspended EAP 6 server from which are consuming remote MDBs.

Test scenario - "suspend server in remote JCA topology in cluster" [1]
1. Start servers 1 and 3 in HornetQ cluster (jms cluster) with deployed queues InQueue and OutQueue
2. Start servers 2 and 4 (mdb servers) which has configured resource adapter to connect to jms cluster
3. Start sending 10 000  messages to InQueue to jms cluster and deploy MDB to each mdb server
   - MDB consumes messages from InQueue and for each message sends a new message to OutQueue. MDB makes JNDI lookup for OutQueue for each message.
5. During processing of messages by MDB, suspend process of server 3 (jms server) for 10 minutes and resume
6. Wait until all messages are processed from InQueue
7. Receive messages from OutQueue

Test failed because there were 5 messages sent to InQueue which did not have its corresponding message in OutQueue:
Lost message detected - there are not corresponding messages for: [ID:fafd2d81-a416-11e5-8d99-2d9061c53a0c, ID:ff18cf91-a416-11e5-8d99-2d9061c53a0c, ID:ff438995-a416-11e5-8d99-2d9061c53a0c, ID:0bf3860c-a417-11e5-8d99-2d9061c53a0c, ID:16f472fe-a417-11e5-8d99-2d9061c53a0c]

I've checked the logs and see that messages are "stuck" in InQueue the whole time. Strange thing is that once receiver (step 7.) finishes consumption of messages from OutQueue then MDBs process those 5 messages and they get to OutQueue. (I can see them in OutQueue when printing content of journals from server1 and server3 after the test.)

To download traces from all 4 servers, journal directories, configuration and clients (test-suite.log) - see comments below to download it
Link to failed Jenkins job - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-64x-patched-hornetq-remote-jca-suspend-tests-2/11/
Complete "info" log from the test - https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-64x-patched-hornetq-remote-jca-suspend-tests-2/11/consoleText

[1] https://mojo.redhat.com/docs/DOC-1058185

Note You need to log in before you can comment on or make changes to this bug.