Bug 856101 - Cache cannot shut down when disconnected
Cache cannot shut down when disconnected
Status: CLOSED NOTABUG
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan (Show other bugs)
6.0.1
Unspecified Unspecified
unspecified Severity medium
: ---
: 6.1.0
Assigned To: Tristan Tarrant
Martin Gencur
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-11 04:22 EDT by Radim Vansa
Modified: 2012-09-18 01:50 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
When the cache is disconnected from the network, it may become impossible to shut down. As a result, graceful node shutdown not possible. As a workaround, the process can be killed manually. The node may continue to run if it is not manually killed.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-13 10:31:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-2283 Major Closed Cache cannot shut down when disconnected 2015-10-12 06:49 EDT

  None (edit)
Description Radim Vansa 2012-09-11 04:22:49 EDT
Description of problem:
When the DISCARD protocol is set to discard all messages and then the cache is shut down using cacheManager.stop() an infinite loop is entered throwing these exceptions:

09:59:23,538 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Installing new view CacheView{viewId=11, members=[rvansa-31823]} for cache x
09:59:23,539 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) ISPN000172: Failed to prepare view CacheView{viewId=11, members=[rvansa-31823]} for cache  x, rolling back to view CacheView{viewId=10, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]}
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
	at java.util.concurrent.FutureTask.get(FutureTask.java:119)
	at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:323)
	at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
	at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:894)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down
	at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:494)
	at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:314)
	... 5 more
09:59:23,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) x: Rolling back to cache view 10, new view id is 12
09:59:23,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Rolled back to view CacheView{viewId=10, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} for cache x
09:59:24,538 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Installing new view CacheView{viewId=13, members=[rvansa-31823]} for cache x
09:59:24,539 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) ISPN000172: Failed to prepare view CacheView{viewId=13, members=[rvansa-31823]} for cache  x, rolling back to view CacheView{viewId=12, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]}
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
	at java.util.concurrent.FutureTask.get(FutureTask.java:119)
	at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:323)
	at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250)
	at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:894)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down
	at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:494)
	at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:314)
	... 5 more
09:59:24,539 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) x: Rolling back to cache view 12, new view id is 14
09:59:24,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Rolled back to view CacheView{viewId=12, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} for cache x
Comment 2 JBoss JIRA Server 2012-09-13 03:11:11 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-2283

I've found that this can occur not only when the node is disconnected but also when the whole cluster is shutting down and therefore the cache view is changed after the node has shut down.
Comment 3 JBoss JIRA Server 2012-09-13 08:43:02 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-2283

In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a timeout in RSVP protocol which was set to 10 minutes. I'll check why the timeout is as high.
Comment 4 JBoss JIRA Server 2012-09-13 09:51:35 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-2283

In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a transport.distributedSyncTimeout elapsed which was set to 10 minutes. I'll check why the timeout is as high.
Comment 5 JBoss JIRA Server 2012-09-13 10:24:01 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-2283

In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a RSVP protocol timeout elapsed which was set to 10 minutes.
Comment 6 JBoss JIRA Server 2012-09-13 10:29:47 EDT
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-2283 to Resolved
Comment 7 JBoss JIRA Server 2012-09-13 10:29:47 EDT
Radim Vansa <rvansa@redhat.com> made a comment on jira ISPN-2283

Not a bug - caused by improper configuration of RSVP protocol, was just waiting too long due to configuration.
Comment 8 JBoss JIRA Server 2012-09-13 10:30:13 EDT
Radim Vansa <rvansa@redhat.com> updated the status of jira ISPN-2283 to Closed

Note You need to log in before you can comment on or make changes to this bug.