Description of problem: When a live server dies, its backup becomes active. And the backup becomes inactive again after the live server has restarted (fail-back). Though the backup server keeps warning the following messages after the fail-back every about 2 minutes. ~~~ 10:42:52,345 WARN [org.hornetq.jms.server] (Periodic Recovery) HQ122015: Can not connect to XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=5f3417b7-41cc-11e3-b1cc-9fd7243525cb, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=null] on auto-generated resource recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:852) [hornetq-core-client-2.3.1.Final-redhat-1-bz-1017943.jar:] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:378) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:287) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:75) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:534) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:168) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] 10:42:52,347 WARN [org.hornetq.jms.server] (Periodic Recovery) HQ122008: XA Recovery can not connect to any hornetq server on recovery [XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=5f3417b7-41cc-11e3-b1cc-9fd7243525cb, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=null]] 10:42:52,348 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:314) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:75) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:534) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:168) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] Caused by: HornetQException[errorType=NOT_CONNECTED message=null] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:427) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:287) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] ... 5 more 10:42:52,351 WARN [org.hornetq.jms.server] (Periodic Recovery) HQ122015: Can not connect to XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=5f3417b7-41cc-11e3-b1cc-9fd7243525cb, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=null] on auto-generated resource recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.] at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:852) [hornetq-core-client-2.3.1.Final-redhat-1-bz-1017943.jar:] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:378) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:287) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:75) [hornetq-jms-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoveryFirstPass(XARecoveryModule.java:548) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:168) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] 10:42:52,352 WARN [org.hornetq.jms.server] (Periodic Recovery) HQ122008: XA Recovery can not connect to any hornetq server on recovery [XARecoveryConfig [transportConfiguration = [TransportConfiguration(name=5f3417b7-41cc-11e3-b1cc-9fd7243525cb, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryConfiguration = null, username=null, password=null]] ~~~ Steps to Reproduce: 1. Set up a HornetQ live-backup domain as in https://access.redhat.com/site/solutions/239013 or use attached domain.xml and host.xml ("live-full-ha" profile for live-sever and "backup-full-ha" for backup-server). 2. Launch the domain and confirm "HQ221007: Server is now live" in live-server and "HQ221031: backup announced". 3. Kill live-server and confirm "HQ221010: Backup Server is now live" in backup-server. 4. Start live-server (/host=master/server-config=server-one:start) and confirm a failback happened. 5. Warning messages in the description keep recording in backup-server. Additional Info: In step 4., you'll probably need '/host=master/server-config=server-one:stop' first because of Bug 965374.
Created attachment 825356 [details] domain.xml for EAP610
Created attachment 825357 [details] host.xml for EAP610
Setting flags for EAP 6.2 so it can be tracked for next releases of EAP 6. It seems that it's to late for getting changes to current release. This is dedicated topology with shared store. Problem is that HQ RA is configured on backup server (with in-vm connector) where actually nothing except HQ backup should be hosted. We should prepare example configuration for HQ backup for EAP 6 with just necessary and supported services similarly as we have for EAP 5.
As far as Im aware this is not one of the supported configurations, we support colocated, dedicated and dedicated with remote HQ server and in all 3 there should be no MDB's. Its the pooled connection factory that registers a resource with the transaction manager when it first is able to connect and it is valid from then on for the transaction manager to keep rying to recover transactions if it has knowledge of an xa resource
Hi Osamu, thanks for creating this bz. Problem is that there is pooled-connection-factory in configuration of HQ backup server which is using in-vm connector. This should be removed from configuration. EAP 6 server with HQ backup (in dedicated topology) should not host any deployment (like MDB, Servlet or SLSB) which is using pooled CF or HQ resource adapter connecting to this backup. I've created new bz for documentation team to document it to administration and configuration guide - bz#1032551 I'll create a new bz for EAP 6.3 to provide example configuration with just necessary services once "jboss-eap-6.3" flag is created. Let's keep this bz open for now so we don't loose track on this until I do so. Thanks, Mirek
Hi Mirek, Thank you for the review. I removed "/profile=backup-full-ha/subsystem=messaging/hornetq-server=default/pooled-connection-factory=hornetq-ra" and the warnings disappeared now. The example configuration will be definitely appreciated. Maybe we can add a new profile like "full-ha-backup" in the out-of-the-box domain.xml. Regards, Osamu
One more question. Some fix is done in WFLY-1710, don't we need to back port it?
Osamu, thanks for pointing this out. WFLY-1710 has already been back ported as part of https://bugzilla.redhat.com/show_bug.cgi?id=1003913.
I cannot reproduce this issue in EAP 6.3.0.ER9. Closing.