Bug 900206 (JBPAPP6-752)
| Summary: | CLONE - Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y" | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Radoslav Husar <rhusar> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED DUPLICATE | QA Contact: | |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 6.0.0 | CC: | rachmato, simone.gotti, sjacobs |
| Target Milestone: | --- | ||
| Target Release: | EAP 6.0.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | http://jira.jboss.org/jira/browse/JBPAPP6-752 | ||
| Whiteboard: | eap6_clustering eap6_need_triage failover_testing | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: |
Linux
|
|
| Last Closed: | 2012-05-14 11:47:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 900611 | ||
Link: Added: This issue Cloned from AS7-4143 Link: Added: This issue is related to AS7-4144 Link: Added: This issue is incorporated by JBPAPP-7577 Workflow: Removed: GIT Pull Request workflow Added: jira Security: Added: Public Docs QE Status: Added: NEW Just a tracker Jira for EAP, discussion see upstream Jira. Link: Removed: This issue is incorporated by JBPAPP-7577 Link: Added: This issue is related to JBPAPP-7577 Labels: Removed: eap6_need_triage failover_testing Added: eap6_clustering eap6_need_triage failover_testing Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828 Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828 Based on todays update, this issue is still be worked on upstream. Retarget for CR1. Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828 This exception *has not* been seen in EAP6-Failover runs as of Tuesday, May 8 (using AS7 master, at least 2 weeks more recent than ER6). Link: Added: This issue duplicates JBPAPP-8423 Link: Added: This issue is a dependency of JBPAPP-9290 Docs QE Status: Removed: NEW |
project_key: JBPAPP6 Most of the positive failover scenarios under little load result in StateTransferInProgressException. Load is on average 50r/s (which is 200x less than AS5 easily handles). {noformat} [JBossINF] 16:04:42,099 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf20-10.16.90.58-8009-25) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 23 [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1] [JBossINF] at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1] [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1530) [JBossINF] at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865) [JBossINF] at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:133) [JBossINF] at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91) [JBossINF] at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88) [JBossINF] at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56) [JBossINF] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:154) [JBossINF] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [JBossINF] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [JBossINF] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [JBossINF] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [JBossINF] at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [JBossINF] at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [JBossINF] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] {noformat} This likely results in timeout on the clients (that you see in the runs). {noformat} Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out> org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:218) at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51) at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:132) ... 3 more {noformat}