Test scenario: 1. EAP 6.1.0 ER4 with HornetQ configured as backup 2. EAP 6.1.0 ER4 with HornetQ configured as live 3. Shutdown EAP 6.1.0 ER4 with backup Result: After step 3. EAP 6.1.0 ER4 with HornetQ configured as backup does not start and throws: 18:15:03,296 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=7c2c4c70-9c79-11e2-abd5-538a3c182844) HQ224000: Failure in initialisation: java.lang.IllegalStateEx ception: ServiceBuilder is already installed at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:302) at org.jboss.as.messaging.jms.JMSService$1.activated(JMSService.java:89) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at org.hornetq.core.server.impl.HornetQServerImpl.callActivateCallbacks(HornetQServerImpl.java:1358) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1581) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.access$1300(HornetQServerImpl.java:169) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation.run(HornetQServerImpl.java:2117) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] As a consequence JMS clients cannot failover to backup. Reproducer attached (reproducer.zip): 1. Download and unzip it - following steps execute in unzipped "reproducer" directory 2. Prepare servers - "sh prepare.sh" - downloads EAP 6.1.0.ER4 + natives - copies jboss-eap-6.1 to server1 and server2 directory - copies standalone-full-ha-live.xml, standalone-full-ha-backup.xml 3. Start backup server - sh start-server2.sh <ip_of_backup> and wait until it's started 4. Start live server - sh start-server1.sh <ip_of_live> 5. Shutdown/Kill live server
Created attachment 731566 [details] reproducer.zip
I did a typo in step 3. in description of test scenario. (bloody bz I can't edit it) There should be: 3. Shutdown EAP 6.1.0 ER4 with LIVE Also sometimes there are java.lang.IllegalStateException in console of backup server: 14:09:15,937 WARN [org.jboss.messaging] (ServerService Thread Pool -- 64) JBAS011603: Failed to destroy queue: testQueue0: java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active at org.hornetq.jms.server.impl.JMSServerManagerImpl.checkInitialised(JMSServerManagerImpl.java:1620) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.access$1100(JMSServerManagerImpl.java:105) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl$3.runException(JMSServerManagerImpl.java:806) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.runAfterActive(JMSServerManagerImpl.java:1832) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.removeQueueFromJNDI(JMSServerManagerImpl.java:795) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.jboss.as.messaging.jms.JMSQueueService$2.run(JMSQueueService.java:89) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1] 14:09:15,937 WARN [org.jboss.messaging] (ServerService Thread Pool -- 63) JBAS011603: Failed to destroy jms topic: testTopic0: java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active at org.hornetq.jms.server.impl.JMSServerManagerImpl.checkInitialised(JMSServerManagerImpl.java:1620) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.access$1100(JMSServerManagerImpl.java:105) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl$4.runException(JMSServerManagerImpl.java:854) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.runAfterActive(JMSServerManagerImpl.java:1832) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.removeTopicFromJNDI(JMSServerManagerImpl.java:843) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.jboss.as.messaging.jms.JMSTopicService$2.run(JMSTopicService.java:84) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
the issue does not occur the 1st time the backup fails over but a 2nd time after it *failed back* (ie restarting itself to let the restarted live server becomes active) the events to reproduce the issues are: 6. Restart the live server => the backup server will failback and restart itself 7. Kill a 2nd time the live server => the described exception happens there
This issue blocks our HA testing.
Jeff Mesnil <jmesnil> made a comment on jira AS7-6840 PR for master branch
Jeff Mesnil <jmesnil> made a comment on jira AS7-6840 PR merged in master branch
I can't hit this issue in EAP 6.1.0.ER5. Thanks, Jeff!
Details inside vpn: https://post-office.corp.redhat.com/mailman/private/eap-pm-list/2013-July/msg00105.html