Hide Forgot
We can see "JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back." which causes EAP server to abort startup in our failover tests with invalidation cache and shared cache store with disabled sticky sessions. Scenario description: HTTP traffic accessing clustered web application that has replicated sessions (uses a mod_cluster load balancer). Delay between sending a new request after receiving a response is 4000 ms (for each client). Session size is 34 KB. 4-node EAP cluster + 4-node JDG cluster, one EAP node at time is shut down and after some time started again, while 2000 standalone clients keep calling the application. Sticky sessions are disabled. Configuration: - 4-node EAP cluster with an invalidation cache + a shared cache store (remote JDG cluster) - 4-node JDG cluster with distributed cache - 4 nodes generating load (2000 clients in total) - cache mode: ASYNC or SYNC (for both invalidation and distributed caches, also "write-behind" element is set for "remote-store" element accordingly) Links to configuration files: EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/4/artifact/report/config/jboss-perf18/standalone-ha.xml JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/4/artifact/report/config/jboss-perf22/clustered.xml When EAP server is being started up (after previous shutdown), it sometimes logs this error, which causes EAP server to abort startup: [JBossINF] [0m[31m03:18:03,725 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[ [JBossINF] ("core-service" => "management"), [JBossINF] ("management-interface" => "native-interface") Before that, it was repeatedly logging these two WARN messages: WARN [org.jgroups.protocols.MPING] (MPING) perf19/ejb: discarding discovery request for cluster 'web' from perf19/web; our cluster name is 'ejb'. Please separate your clusters cleanly. WARN [org.jgroups.protocols.MPING] (MPING) perf19/web: discarding discovery request for cluster 'ejb' from perf19/ejb; our cluster name is 'web'. Please separate your clusters cleanly. Server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/4/console-perf19/