Description of problem: When the DISCARD protocol is set to discard all messages and then the cache is shut down using cacheManager.stop() an infinite loop is entered throwing these exceptions: 09:59:23,538 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Installing new view CacheView{viewId=11, members=[rvansa-31823]} for cache x 09:59:23,539 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) ISPN000172: Failed to prepare view CacheView{viewId=11, members=[rvansa-31823]} for cache x, rolling back to view CacheView{viewId=10, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} java.util.concurrent.ExecutionException: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:323) at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250) at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:894) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:494) at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:314) ... 5 more 09:59:23,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) x: Rolling back to cache view 10, new view id is 12 09:59:23,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Rolled back to view CacheView{viewId=10, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} for cache x 09:59:24,538 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Installing new view CacheView{viewId=13, members=[rvansa-31823]} for cache x 09:59:24,539 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) ISPN000172: Failed to prepare view CacheView{viewId=13, members=[rvansa-31823]} for cache x, rolling back to view CacheView{viewId=12, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} java.util.concurrent.ExecutionException: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:323) at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:250) at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:894) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.IllegalStateException: x: Received cache view prepare request after the local node has already shut down at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:494) at org.infinispan.cacheviews.CacheViewsManagerImpl$3.call(CacheViewsManagerImpl.java:314) ... 5 more 09:59:24,539 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) x: Rolling back to cache view 12, new view id is 14 09:59:24,540 DEBUG [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-1,rvansa-31823) Rolled back to view CacheView{viewId=12, members=[rvansa-12248, rvansa-31823, rvansa-2915, rvansa-56921]} for cache x
Radim Vansa <rvansa> made a comment on jira ISPN-2283 I've found that this can occur not only when the node is disconnected but also when the whole cluster is shutting down and therefore the cache view is changed after the node has shut down.
Radim Vansa <rvansa> made a comment on jira ISPN-2283 In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a timeout in RSVP protocol which was set to 10 minutes. I'll check why the timeout is as high.
Radim Vansa <rvansa> made a comment on jira ISPN-2283 In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a transport.distributedSyncTimeout elapsed which was set to 10 minutes. I'll check why the timeout is as high.
Radim Vansa <rvansa> made a comment on jira ISPN-2283 In fact the loop is not as infinite as I thought at first: DefaultCacheManager stops all caches and THEN stops the CacheViewManagerImpl. However, when stopping the cache, CacheViewManagerImpl.leave(cacheName) is called, which gets stuck in transport.invokeRemotely(request leave command), until a RSVP protocol timeout elapsed which was set to 10 minutes.
Radim Vansa <rvansa> updated the status of jira ISPN-2283 to Resolved
Radim Vansa <rvansa> made a comment on jira ISPN-2283 Not a bug - caused by improper configuration of RSVP protocol, was just waiting too long due to configuration.
Radim Vansa <rvansa> updated the status of jira ISPN-2283 to Closed