Bug 948247
| Summary: | Backup can't start after fallback - java.lang.IllegalStateException: ServiceBuilder is already installed | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Miroslav Novak <mnovak> | ||||
| Component: | JMS | Assignee: | Jeff Mesnil <jmesnil> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.1.0 | CC: | brian.stansberry, jason.greene, jdoyle, jmesnil, myarboro, smcgowan | ||||
| Target Milestone: | ER5 | Keywords: | Regression, TestBlocker | ||||
| Target Release: | EAP 6.1.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 731566 [details]
reproducer.zip
I did a typo in step 3. in description of test scenario. (bloody bz I can't edit it) There should be: 3. Shutdown EAP 6.1.0 ER4 with LIVE Also sometimes there are java.lang.IllegalStateException in console of backup server: 14:09:15,937 WARN [org.jboss.messaging] (ServerService Thread Pool -- 64) JBAS011603: Failed to destroy queue: testQueue0: java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active at org.hornetq.jms.server.impl.JMSServerManagerImpl.checkInitialised(JMSServerManagerImpl.java:1620) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.access$1100(JMSServerManagerImpl.java:105) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl$3.runException(JMSServerManagerImpl.java:806) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.runAfterActive(JMSServerManagerImpl.java:1832) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.removeQueueFromJNDI(JMSServerManagerImpl.java:795) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.jboss.as.messaging.jms.JMSQueueService$2.run(JMSQueueService.java:89) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1] 14:09:15,937 WARN [org.jboss.messaging] (ServerService Thread Pool -- 63) JBAS011603: Failed to destroy jms topic: testTopic0: java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active at org.hornetq.jms.server.impl.JMSServerManagerImpl.checkInitialised(JMSServerManagerImpl.java:1620) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.access$1100(JMSServerManagerImpl.java:105) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl$4.runException(JMSServerManagerImpl.java:854) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.runAfterActive(JMSServerManagerImpl.java:1832) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.jms.server.impl.JMSServerManagerImpl.removeTopicFromJNDI(JMSServerManagerImpl.java:843) [hornetq-jms-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.jboss.as.messaging.jms.JMSTopicService$2.run(JMSTopicService.java:84) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1] the issue does not occur the 1st time the backup fails over but a 2nd time after it *failed back* (ie restarting itself to let the restarted live server becomes active) the events to reproduce the issues are: 6. Restart the live server => the backup server will failback and restart itself 7. Kill a 2nd time the live server => the described exception happens there This issue blocks our HA testing. Jeff Mesnil <jmesnil> made a comment on jira AS7-6840 PR for master branch Jeff Mesnil <jmesnil> made a comment on jira AS7-6840 PR merged in master branch I can't hit this issue in EAP 6.1.0.ER5. Thanks, Jeff! |
Test scenario: 1. EAP 6.1.0 ER4 with HornetQ configured as backup 2. EAP 6.1.0 ER4 with HornetQ configured as live 3. Shutdown EAP 6.1.0 ER4 with backup Result: After step 3. EAP 6.1.0 ER4 with HornetQ configured as backup does not start and throws: 18:15:03,296 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=7c2c4c70-9c79-11e2-abd5-538a3c182844) HQ224000: Failure in initialisation: java.lang.IllegalStateEx ception: ServiceBuilder is already installed at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:302) at org.jboss.as.messaging.jms.JMSService$1.activated(JMSService.java:89) [jboss-as-messaging-7.2.0.Final-redhat-4.jar:7.2.0.Final-redhat-4] at org.hornetq.core.server.impl.HornetQServerImpl.callActivateCallbacks(HornetQServerImpl.java:1358) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1581) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.access$1300(HornetQServerImpl.java:169) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation.run(HornetQServerImpl.java:2117) [hornetq-server-2.3.0.CR2-redhat-1.jar:2.3.0.CR2-redhat-1] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] As a consequence JMS clients cannot failover to backup. Reproducer attached (reproducer.zip): 1. Download and unzip it - following steps execute in unzipped "reproducer" directory 2. Prepare servers - "sh prepare.sh" - downloads EAP 6.1.0.ER4 + natives - copies jboss-eap-6.1 to server1 and server2 directory - copies standalone-full-ha-live.xml, standalone-full-ha-backup.xml 3. Start backup server - sh start-server2.sh <ip_of_backup> and wait until it's started 4. Start live server - sh start-server1.sh <ip_of_live> 5. Shutdown/Kill live server