900206 – (JBPAPP6-752) CLONE - Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"

Bug 900206 (JBPAPP6-752) - CLONE - Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"

Summary: CLONE - Failover scenarios often fail with "StateTransferInProgressException:...

Keywords:
Status:	CLOSED DUPLICATE of bug 900021
Alias:	JBPAPP6-752
Product:	JBoss Enterprise Application Platform 6
Classification:	JBoss
Component:	Clustering
Sub Component:
Version:	6.0.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	EAP 6.0.0
Assignee:	Paul Ferraro
QA Contact:
Docs Contact:
URL:	http://jira.jboss.org/jira/browse/JBP...
Whiteboard:	eap6_clustering eap6_need_triage fail...
Depends On:
Blocks:	JBPAPP6-1422
TreeView+	depends on / blocked

Reported:	2012-04-13 13:25 UTC by Radoslav Husar
Modified:	2014-06-28 12:32 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Clone Of:
Environment:	Linux
Last Closed:	2012-05-14 11:47:10 UTC
Type:	Bug
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	899596	1	None	None	None	2021-01-20 06:05:38 UTC
Red Hat Issue Tracker	JBPAPP6-752	0	Blocker	Closed	CLONE - Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer l...	2013-03-22 21:23:08 UTC

Internal Links: 899596

Description Radoslav Husar 2012-04-13 13:25:00 UTC

project_key: JBPAPP6

Most of the positive failover scenarios under little load result in StateTransferInProgressException. 

Load is on average 50r/s (which is 200x less than AS5 easily handles).

{noformat}
[JBossINF] 16:04:42,099 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-perf20-10.16.90.58-8009-25) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 23
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:174) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:272) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:321) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:90) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75) [jboss-as-clustering-web-spi-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1530)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:865)
[JBossINF] 	at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47)
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:133)
[JBossINF] 	at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
[JBossINF] 	at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
[JBossINF] 	at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
[JBossINF] 	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:154)
[JBossINF] 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
[JBossINF] 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[JBossINF] 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[JBossINF] 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[JBossINF] 	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
[JBossINF] 	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
[JBossINF] 	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{noformat}

This likely results in timeout on the clients (that you see in the runs).

{noformat}
Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
        org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:218)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:132)
	... 3 more
{noformat}

Comment 1 Radoslav Husar 2012-04-13 13:25:02 UTC

Link: Added: This issue Cloned from AS7-4143

Comment 2 Radoslav Husar 2012-04-13 13:25:02 UTC

Link: Added: This issue is related to AS7-4144

Comment 3 Radoslav Husar 2012-04-13 13:25:03 UTC

Link: Added: This issue is incorporated by JBPAPP-7577

Comment 4 Radoslav Husar 2012-04-13 13:25:48 UTC

Workflow: Removed: GIT Pull Request workflow  Added: jira
Security: Added: Public
Docs QE Status: Added: NEW

Comment 5 Radoslav Husar 2012-04-13 13:26:20 UTC

Just a tracker Jira for EAP, discussion see upstream Jira.

Comment 6 Rajesh Rajasekaran 2012-04-13 15:14:15 UTC

Link: Removed: This issue is incorporated by JBPAPP-7577

Comment 7 Rajesh Rajasekaran 2012-04-13 15:14:37 UTC

Link: Added: This issue is related to JBPAPP-7577

Comment 8 Rajesh Rajasekaran 2012-04-13 17:58:35 UTC

Labels: Removed: eap6_need_triage failover_testing Added: eap6_clustering eap6_need_triage failover_testing

Comment 9 Shelly McGowan 2012-04-19 17:23:05 UTC

Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828

Comment 10 Anne-Louise Tangring 2012-04-19 17:24:46 UTC

Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828

Comment 11 Shelly McGowan 2012-05-03 16:14:01 UTC

Based on todays update, this issue is still be worked on upstream. Retarget for CR1.

Comment 12 Shelly McGowan 2012-05-03 16:14:02 UTC

Git Pull Request: Removed: https://github.com/jbossas/jboss-as/pull/1828 Added: https://github.com/jbossas/jboss-as/pull/1828

Comment 13 Richard Achmatowicz 2012-05-08 14:55:09 UTC

This exception *has not* been seen in EAP6-Failover runs as of Tuesday, May 8 (using AS7 master, at least 2 weeks more recent than ER6).

Comment 14 Radoslav Husar 2012-05-14 11:47:20 UTC

Link: Added: This issue duplicates JBPAPP-8423

Comment 15 Rajesh Rajasekaran 2012-06-06 20:50:30 UTC

Link: Added: This issue is a dependency of JBPAPP-9290

Comment 16 Anne-Louise Tangring 2012-11-05 17:51:05 UTC

Docs QE Status: Removed: NEW

Note You need to log in before you can comment on or make changes to this bug.