Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 959641

Summary: NPE when node without destinations joins cluster which is used by remote HornetQ Resource Adapter
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Miroslav Novak <mnovak>
Component: HornetQAssignee: Andy Taylor <ataylor>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: anmiller, ataylor, csuconic, dosoudil, dpogrebn, mnovak, msvehla, myarboro
Target Milestone: ER6   
Target Release: EAP 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-15 16:15:32 UTC Type: Enhancement
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1016141    
Bug Blocks:    
Attachments:
Description Flags
reproducer.zip
none
patch to stop partial init of MDB none

Description Miroslav Novak 2013-05-04 15:13:18 UTC
This enhancement is based on bz#912688.

If EAP 6.1.0 server has MDB which has configured HornetQ resource adapter connected to another server in hq cluster then if any node in the cluster doesn't have one queue/topic which MDB is using deployed, the MDB starts throw NPEs.

Comment 1 Clebert Suconic 2013-05-31 02:25:53 UTC
I don't understand how this could be. BZ912688 was happening because of the same issues you found on SpecJMS before the release where the destination was not being cleared.

At this point I have no clue how to work on this issue as I have no clue how to replicate it...


I will assign it back to you since I need more information.

Comment 2 Miroslav Novak 2013-05-31 09:56:49 UTC
Steps to Reproduce:
0. Download and unzip reproducer.zip from attachemnt. Next steps excexute in unzipped "reproducer" directory
1. First create an IP alias by "ifconfig eth1:0 192.168.40.1 up" under root user (change command according your environment)
2. run - "sh prepare.sh"
  - this downloads EAP 6.1.0.GA
  - creates 3 directories server1,2,3
  - copies directory jboss-eap-6.1 to server1,2,3
  - copies configuration standalone-full-ha-jms.xml to server1
  - copies configuration standalone-full-ha-mdb.xml to server2
  - copies mdb1.jar to server2's deployments directory
3. start first server by "sh start-server1.sh 192.168.40.1"
4. start thrird server by "sh start-server3.sh <third_ip>"
5. start second server by "sh start-server2.sh <second_ip>"
6. start jms producer by "sh start-producer.sh 192.168.40.1 1000"
7. see exceptions in server2 log

It's important to let server1 and server3 create a cluster before server3 is started otherwise exceptions won't occur.

Comment 3 Miroslav Novak 2013-05-31 09:58:11 UTC
Created attachment 755223 [details]
reproducer.zip

Comment 5 Miroslav Novak 2013-08-03 10:34:38 UTC
From feedback from dev (bz#912688) this is not an issue. I've set jboss-eap-6.1.1 flag not to loose focus from this bz.

Comment 6 Miroslav Novak 2013-08-26 11:04:53 UTC
My previous comment may look confusing without knowing context.

By default HornetQ RA load-balances connections to all nodes in cluster. When one node does not have specific destinations deployed then HQ RA starts to throw NPEs and breaks the delivery to MDB which is using those destinations. MDB does not receive any message.

Purpose of this BZ is to make HornetQ RA more robust when it connects to node in cluster which does not have deployed destinations needed by MDB so it can

Comment 7 Miroslav Novak 2013-08-26 11:25:42 UTC
... deliver messages from "well configured" nodes.

Comment 8 Andy Taylor 2013-08-27 10:05:00 UTC
nodes arent aware of the configuration of backup nodes and since the backup may not be available when the live startes there is not much we can do on the live. We could however a check for the queue on failover and if its not present stop the MDB and log a warning.

Comment 9 Miroslav Novak 2013-08-29 06:51:35 UTC
After irc chat with Andy, dev will discuss possible solutions for this.

Comment 10 Andy Taylor 2013-09-02 12:57:27 UTC
Created attachment 792818 [details]
patch to stop partial init of MDB

The issue is that if in HA and sessions are round robinned if the 1st succeeds and is started  the second may fail. At this point we stop the MSB but the first session is akready handling messages. This patch postpones the starting of the session until all sessions are created and then starts them all together. If there is a single failure we stop the whole MDB rather than partially init it.

Comment 11 Miroslav Novak 2013-09-16 11:52:30 UTC
*** Bug 914758 has been marked as a duplicate of this bug. ***

Comment 12 Clebert Suconic 2013-09-17 21:16:26 UTC
Andy: it seems that the patch makes sense. Can you do it?

Comment 13 Andy Taylor 2013-09-18 07:45:10 UTC
pr sent - https://github.com/hornetq/hornetq/pull/1287

Comment 17 Martin Svehla 2013-10-29 07:58:56 UTC
Tested with EAP 6.2.0.ER6 / HornetQ 2.3.9