Hide Forgot
We can see some IllegalStateExceptions during server shutdown in our failover tests with invalidation cache and shared cache store with disabled sticky sessions. Scenario description: HTTP traffic accessing clustered web application that has replicated sessions (uses a mod_cluster load balancer). Delay between sending a new request after receiving a response is 4000 ms (for each client). Session size is 34 KB. 4-node EAP cluster + 4-node JDG cluster, one EAP node at time is shut down and after some time started again, while 2000 standalone clients keep calling the application. Sticky sessions are disabled. Configuration: - 4-node EAP cluster with an invalidation cache + a shared cache store (remote JDG cluster) - 4-node JDG cluster with distributed cache - 4 nodes generating load (2000 clients in total) - cache mode: ASYNC or SYNC (for both invalidation and distributed caches, also "write-behind" element is set for "remote-store" element accordingly) Links to configuration files: EAP http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/3/artifact/report/config/jboss-perf18/standalone-ha.xml JDG http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/3/artifact/report/config/jboss-perf22/clustered.xml These two errors were seen during server shutdown: [JBossINF] [0m[0m07:23:35,305 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 68) MODCLUSTER000002: Initiating mod_cluster shutdown [JBossINF] [0m[32m07:23:35,305 DEBUG [org.infinispan.CacheImpl] (ServerService Thread Pool -- 68) Stopping cache default-host/clusterbench-offload on perf19/web [JBossINF] [0m[32m07:23:35,365 DEBUG [org.infinispan.transaction.TransactionTable] (ServerService Thread Pool -- 68) Wait for on-going transactions to finish for 30 seconds. [JBossINF] [0m[0m07:23:35,377 INFO [org.jboss.weld.deployer] (MSC service thread 1-11) JBAS016009: Stopping weld service for deployment clusterbench-ee6-offload.ear [JBossINF] [0m[31m07:23:35,331 ERROR [org.jboss.as.txn] (ajp-perf19/10.16.90.56:8009-198) JBAS010151: Unable to get transaction state: java.lang.IllegalStateException [JBossINF] at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47) [JBossINF] at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.checkTransactionStatus(TransactionRollbackSetupAction.java:112) [JBossINF] at org.jboss.as.txn.deployment.TransactionRollbackSetupAction.teardown(TransactionRollbackSetupAction.java:66) [JBossINF] at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.5.8.Final-redhat-2-bz-1370648.jar:7.5.8.Final-redhat-SNAPSHOT] [JBossINF] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:189) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91] : : [JBossINF] [0m[31m07:23:35,447 ERROR [org.apache.coyote.ajp] (ajp-perf19/10.16.90.56:8009-1372) JBWEB003038: Error processing request: java.lang.IllegalStateException: JBAS010217: Must call start() before first call to unlock() [JBossINF] at org.jboss.as.clustering.lock.YieldingGloballyExclusiveClusterLockSupport.unlock(YieldingGloballyExclusiveClusterLockSupport.java:57) [JBossINF] at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.unlock(SharedLocalYieldingClusterLockManager.java:518) [JBossINF] at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.relinquishSessionOwnership(DistributedCacheManager.java:411) [JBossINF] at org.jboss.as.web.session.ClusteredSession.relinquishSessionOwnership(ClusteredSession.java:575) [jboss-as-web-7.5.8.Final-redhat-2-bz-1370648.jar:7.5.8.Final-redhat-SNAPSHOT] [JBossINF] at org.jboss.as.web.session.ClusteredSession.endAccess(ClusteredSession.java:567) [jboss-as-web-7.5.8.Final-redhat-2-bz-1370648.jar:7.5.8.Final-redhat-SNAPSHOT] [JBossINF] at org.apache.catalina.connector.Request.recycle(Request.java:565) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:387) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.16.Final-redhat-1.jar:7.5.16.Final-redhat-1] [JBossINF] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_91] Link to server logs: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-remote-jdg-session-shutdown-invalidation-sync-4nodes-no-sticky-sessions-perf17/3/console-perf19/