Bug 918791
| Summary: | Starting a server with multiple web apps, causes deployment failures | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Shay Matasaro <smatasar> |
| Component: | Clustering | Assignee: | Paul Ferraro <paul.ferraro> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.0.1 | CC: | dereed, jkudrnac, myarboro, paul.ferraro, pjindal, rhusar, rjanik |
| Target Milestone: | DR1 | ||
| Target Release: | EAP 6.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
In some cases, web applications on a clustered server would fail to deploy if multiple applications were being deployed. Each application in this situation would attempt to lock the cache manager to create its cache, and the first application that obtained a lock would deploy successfully. However, depending on the time taken to deploy, any other deployments could timeout while waiting for access to the cache manager and fail to deploy. JBoss EAP 6 now includes a `GlobalComponentRegistryService` which handles this scenario and applications now deploy successfully in this situation.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-12-15 16:12:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Shay Matasaro
2013-03-06 21:08:45 UTC
Dennis Reed <dereed> made a comment on jira ISPN-2898 The issue is the cacheCreateLock in org.infinispan.manager.DefaultCacheManager#wireCache When multiple caches are started at the same time, some are timing out waiting for this lock. jaikiran pai <jpai> made a comment on jira AS7-6685 Which version of AS7 is this reproducible against and is it always reproducible? You really need to specify affected versions. Bug reports without it are useless. Also please paste the exception, makes much easier looking for whether its a known issue or not. version numbers added to all 3 bugs the issue depends on the time it takes to start the cachemanager and the number of deployed apps , i assume that if the manager goes up really fast deployments might not fail, but generally this is pretty consistent :22:40,823 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC000001: Failed to start service jboss.infinispan.web.default-host/FT: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/FT: org.infinispan.CacheException: Unable to acquire lock on cache with name default-host/FT at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) [jboss-as-clustering-common-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_37] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_37] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2] Caused by: org.infinispan.CacheException: Unable to acquire lock on cache with name default-host/FT at org.infinispan.manager.DefaultCacheManager.wireCache(DefaultCacheManager.java:675) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:125) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:116) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) [jboss-as-clustering-common-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4] ... 4 more Dan Berindei <dberinde> made a comment on jira ISPN-2898 What is the lockAcquisitionTimeout on the caches being started? I think increasing that timeout should be a good enough workaround. As for Infinispan code changes, we should add a global startupTimeout that should be used for the global cacheCreationLock acquisition instead of each cache's lockAcquisitionTimeout. Changing DefaultCacheManager.start() to start all the components in the global component registry, including the transport, would help as well. Dan Berindei <dberinde> made a comment on jira ISPN-2898 Correction: the proper workaround is to increase the lockAcquisitionTimeout of the *default* cache, not of the cache being started. I have tested it and it works. We can backport the GlobalComponentRegistryService from upstream master. This should alleviate the problem significantly. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |