Red Hat Bugzilla – Bug 918791
Starting a server with multiple web apps, causes deployment failures
Last modified: 2014-06-12 08:47:21 EDT
When the server starts with multiple web apps , and each one tries to create its cache , the first one locks all other because of a lock on cache manager see ISPN-2898 this results in deployment failure , and the apps will not get redeployed
Dennis Reed <dereed@redhat.com> made a comment on jira ISPN-2898 The issue is the cacheCreateLock in org.infinispan.manager.DefaultCacheManager#wireCache When multiple caches are started at the same time, some are timing out waiting for this lock.
jaikiran pai <jpai@redhat.com> made a comment on jira AS7-6685 Which version of AS7 is this reproducible against and is it always reproducible?
You really need to specify affected versions. Bug reports without it are useless. Also please paste the exception, makes much easier looking for whether its a known issue or not.
version numbers added to all 3 bugs
the issue depends on the time it takes to start the cachemanager and the number of deployed apps , i assume that if the manager goes up really fast deployments might not fail, but generally this is pretty consistent
:22:40,823 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC000001: Failed to start service jboss.infinispan.web.default-host/FT: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/FT: org.infinispan.CacheException: Unable to acquire lock on cache with name default-host/FT at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) [jboss-as-clustering-common-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2] Caused by: org.infinispan.CacheException: Unable to acquire lock on cache with name default-host/FT at org.infinispan.manager.DefaultCacheManager.wireCache(DefaultCacheManager.java:675) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:125) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:116) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) [jboss-as-clustering-common-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] ... 4 more
Dan Berindei <dberinde@redhat.com> made a comment on jira ISPN-2898 What is the lockAcquisitionTimeout on the caches being started? I think increasing that timeout should be a good enough workaround. As for Infinispan code changes, we should add a global startupTimeout that should be used for the global cacheCreationLock acquisition instead of each cache's lockAcquisitionTimeout. Changing DefaultCacheManager.start() to start all the components in the global component registry, including the transport, would help as well.
Dan Berindei <dberinde@redhat.com> made a comment on jira ISPN-2898 Correction: the proper workaround is to increase the lockAcquisitionTimeout of the *default* cache, not of the cache being started. I have tested it and it works.
We can backport the GlobalComponentRegistryService from upstream master. This should alleviate the problem significantly.
https://github.com/jbossas/jboss-eap/pull/327