Bug 1806709 - [GSS](6.4.z) Make HornetQ UDP Clustering Client discovery resilient
Summary: [GSS](6.4.z) Make HornetQ UDP Clustering Client discovery resilient
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: HornetQ
Version: unspecified
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: EAP 6.4.23
Assignee: Romain Pelisse
QA Contact: Peter Mackay
URL:
Whiteboard:
Depends On:
Blocks: eap6423-payload 1806052 1700855
TreeView+ depends on / blocked
 
Reported: 2020-02-24 19:36 UTC by Brad Maxwell
Modified: 2020-08-26 16:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Apache JIRA ARTEMIS-2637 0 None None None 2020-02-29 02:11:20 UTC
Red Hat Issue Tracker JBEAP-18839 0 Critical Verified [GSS](7.2.z) ARTEMIS-2637 - Resilience around UDP Discovery 2020-08-26 16:24:31 UTC
Red Hat Issue Tracker JBEAP-18840 0 Critical Verified [GSS](7.3.z) ARTEMIS-2637 - Resilience around UDP Discovery 2020-08-26 16:24:31 UTC

Description Brad Maxwell 2020-02-24 19:36:44 UTC

Comment 4 Brad Maxwell 2020-02-29 02:10:30 UTC
An EAP server setup as a HornetQ client using Discovery to locate the HornetQ JMS Brokers via UDP such as in the configuration below.  Can get into a state when there is an unknown issue occur such as a network blip, JVM Bug, OS bug where in it times out with the error below but is unable to recover and even when the network messages are received and will continue to fail with the error below.  The HornetQ UDP Clustering Client Discovery needs to be more resilient so that it can recover and reconnect.

            <subsystem xmlns="urn:jboss:domain:messaging:1.4">
                <hornetq-server>
                    <persistence-enabled>true</persistence-enabled>
                    <message-expiry-scan-period>300000</message-expiry-scan-period>
                    <journal-type>NIO</journal-type>
                    <journal-min-files>2</journal-min-files>

                    <discovery-groups>
                        <discovery-group name="dg-group1">
                            <socket-binding>messaging-group</socket-binding>
                            <refresh-timeout>10000</refresh-timeout>
                            <initial-wait-timeout>20000</initial-wait-timeout>
                        </discovery-group>
                    </discovery-groups>

                    <jms-connection-factories>
                        <pooled-connection-factory name="hornetq-ra">
                            <user>jmsuser</user>
                            <password>redhat1!</password>
                            <min-pool-size>10</min-pool-size>
                            <max-pool-size>30</max-pool-size>
                            <discovery-group-ref discovery-group-name="dg-group1"/>
                            <entries>
                                <entry name="java:/JmsXA"/>
                            </entries>
                            <retry-interval>4000</retry-interval>
                            <max-retry-interval>12000</max-retry-interval>
                            <reconnect-attempts>15</reconnect-attempts>
                            <connection-load-balancing-policy-class-name>org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy</connection-load-balancing-policy-class-name>
                        </pooled-connection-factory>
                    </jms-connection-factories>
                </hornetq-server>
            </subsystem>

        <socket-binding-group name="full-sockets" default-interface="public">
            ...
            <socket-binding name="messaging" port="5445"/>
            <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
            <socket-binding name="messaging-throughput" port="5455"/>
        </socket-binding-group>


[Server:client] 19:50:08,293 ERROR [org.hornetq.ra] (JMSTest-Runner-0) HQ154002: Could not create session: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/JmsXA
[Server:client] 	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:410)
[Server:client] 	at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:367)
[Server:client] 	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:499)
[Server:client] 	at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:832)
[Server:client] 	at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:465)
[Server:client] 	at test.JmsConnectionFactoryRestService$Runner.run(JmsConnectionFactoryRestService.java:164) [classes:]
[Server:client] 	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_202]
[Server:client] Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null
[Server:client] 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:457)
[Server:client] 	at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:479)
[Server:client] 	at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:451)
[Server:client] 	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:344)
[Server:client] 	... 6 more
[Server:client] Caused by: javax.resource.ResourceException: Error during setup
[Server:client] 	at org.hornetq.ra.HornetQRAManagedConnection.<init>(HornetQRAManagedConnection.java:171)
[Server:client] 	at org.hornetq.ra.HornetQRAManagedConnectionFactory.createManagedConnection(HornetQRAManagedConnectionFactory.java:157)
[Server:client] 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:874)
[Server:client] 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:416)
[Server:client] 	... 9 more
[Server:client] Caused by: javax.resource.ResourceException: Failed to create session factory
[Server:client] 	at org.hornetq.ra.HornetQRAManagedConnection.setup(HornetQRAManagedConnection.java:887)
[Server:client] 	at org.hornetq.ra.HornetQRAManagedConnection.<init>(HornetQRAManagedConnection.java:160)
[Server:client] 	... 12 more
[Server:client] Caused by: javax.jms.JMSException: Failed to create session factory
[Server:client] 	at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:587)
[Server:client] 	at org.hornetq.jms.client.HornetQConnectionFactory.createXAConnection(HornetQConnectionFactory.java:143)
[Server:client] 	at org.hornetq.ra.HornetQRAManagedConnection.setup(HornetQRAManagedConnection.java:871)
[Server:client] 	... 13 more
[Server:client] Caused by: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119012: Timed out waiting to receive initial broadcast from cluster]
[Server:client] 	at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:859)
[Server:client] 	at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:583)
[Server:client] 	... 15 more

Comment 8 Peter Mackay 2020-06-30 11:57:21 UTC
Verified with EAP 6.4.23.CP.CR2


Note You need to log in before you can comment on or make changes to this bug.