Affects: Release Notes project_key: JBPAPP6 Just undeploying application from one node in the cluster. Application is still being accessed by remote ejb clients. {noformat} [JBossINF] 18:01:15,248 INFO [org.jboss.as.osgi] (MSC service thread 1-2) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader [JBossINF] 18:01:15,249 INFO [org.jboss.as.osgi] (MSC service thread 1-8) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear:main" from Service Module Loader [JBossINF] 18:01:15,253 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-15) ISPN000029: Passivating all entries to disk [JBossINF] 18:01:15,249 INFO [org.jboss.as.osgi] (MSC service thread 1-6) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-web.war:main" from Service Module Loader [JBossINF] 18:01:15,255 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-15) ISPN000030: Passivated 0 entries in 2 milliseconds [JBossINF] 18:01:15,269 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk [JBossINF] 18:01:15,273 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000029: Passivating all entries to disk [JBossINF] 18:01:15,270 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container [JBossINF] 18:01:15,275 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000030: Passivated 0 entries in 1 milliseconds [JBossINF] 18:01:15,296 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010282: Stopped //default-host//clusterbench cache from web container [JBossINF] 18:01:15,303 INFO [org.jboss.weld.deployer] (MSC service thread 1-10) JBAS016009: Stopping weld service for deployment clusterbench-ee6.ear [JBossINF] 18:01:15,311 INFO [org.infinispan.eviction.PassivationManagerImpl] (pool-20-thread-2) ISPN000029: Passivating all entries to disk [JBossINF] 18:01:15,313 INFO [org.infinispan.eviction.PassivationManagerImpl] (pool-20-thread-2) ISPN000030: Passivated 0 entries in 1 milliseconds [JBossINF] 18:01:15,314 INFO [org.infinispan.eviction.PassivationManagerImpl] (pool-15-thread-2) ISPN000029: Passivating all entries to disk [JBossINF] 18:01:15,317 INFO [org.jboss.as.clustering.infinispan] (pool-20-thread-2) JBAS010282: Stopped repl cache from ejb container [JBossINF] 18:01:15,326 INFO [org.jboss.as.server.deployment] (MSC service thread 1-16) JBAS015877: Stopped deployment clusterbench-ee6-ejb.jar in 82ms [JBossINF] 18:01:15,328 INFO [org.jboss.as.server.deployment] (MSC service thread 1-12) JBAS015877: Stopped deployment clusterbench-ee6-web.war in 84ms [JBossINF] 18:01:15,328 INFO [org.jboss.as.clustering.infinispan] (pool-21-thread-2) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container [JBossINF] 18:01:15,330 INFO [org.jboss.as.server.deployment] (MSC service thread 1-14) JBAS015877: Stopped deployment clusterbench-ee6.ear in 88ms [JBossINF] 18:01:15,352 INFO [org.infinispan.eviction.PassivationManagerImpl] (pool-15-thread-2) ISPN000030: Passivated 3 entries in 37 milliseconds [JBossINF] 18:01:15,354 INFO [org.jboss.as.clustering.infinispan] (pool-15-thread-2) JBAS010282: Stopped repl cache from web container [JBossINF] 18:01:15,476 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-2) ISPN000082: Stopping the RpcDispatcher [JBossINF] 18:01:15,498 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-19-thread-2) ISPN000082: Stopping the RpcDispatcher [JBossINF] 18:01:15,523 WARN [org.jboss.msc.service.fail] (MSC service thread 1-9) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee6.ear"."clusterbench-ee6-ejb.jar".component.RemoteStatefulSBImpl.START: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl [JBossINF] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:882) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:672) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:551) [JBossINF] at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:198) [JBossINF] at org.infinispan.CacheImpl.stop(CacheImpl.java:516) [JBossINF] at org.infinispan.CacheImpl.stop(CacheImpl.java:508) [JBossINF] at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:291) [JBossINF] at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:107) [JBossINF] at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:308) [JBossINF] at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:315) [JBossINF] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:102) [JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:306) [JBossINF] at org.jboss.as.ee.component.ComponentStartService.stop(ComponentStartService.java:49) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.loaders.bucket.Bucket [JBossINF] Caused by: an exception which occurred: [JBossINF] in object org.infinispan.loaders.bucket.Bucket@40a995c5 [JBossINF] [JBossINF] 18:01:15,916 INFO [org.jboss.as.server] (management-handler-thread - 1) JBAS018558: Undeployed "clusterbench-ee6.ear" {noformat}
Whether the issue is present upstream not yet known (needs testing).
Link: Added: This issue is related to JBPAPP-7577
I'm raising the priority on this JIRA as I believe this blocks getting valid runs for ejb3 SFSB replication performance. Rado, please correct me if I am wrong.
Labels: Added: eap6_clustering eap6_need_triage
Why would replication performance testing be undeploying applications?
Jason is right, of course this does not have effect on performance tests. If people got here looking for the perf issue, it's this one JBPAPP-8721. The issue has been spotted at failover tests.
Link: Added: This issue depends ISPN-1981
Link: Added: This issue depends AS7-4559
This turns out to be due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent DistributedCacheManagerFactory, which depends on the cache container. So, what happens is the cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null - causing the NotSerializableException.
Link: Added: This issue depends AS7-4612
Scheduling for ER7 as the upstream issue seems resolved. Requires an Infinispan upgrade. Assigning to Shelly to coordinate.
Link: Removed: This issue depends AS7-4559
The commits reference in the AS7 JIRAs are in ER7. ER7 also has an Infinispan upgrade (5.1.4.Final) but note the linked ISPN-1981 issue was Rejected.
The issue is still present in ER8 upon shutdown in EJB3 test scenarios: {noformat} [JBossINF] 07:42:59,546 WARN [org.jboss.msc.service.fail] MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee6.ear"."clusterbench-ee6-ejb.jar".component.RemoteStatefulSBImpl.START: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl [JBossINF] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:674) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:553) [JBossINF] at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:199) [JBossINF] at org.infinispan.CacheImpl.stop(CacheImpl.java:539) [JBossINF] at org.infinispan.CacheImpl.stop(CacheImpl.java:531) [JBossINF] at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348) [JBossINF] at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:117) [JBossINF] at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319) [JBossINF] at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321) [JBossINF] at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118) [JBossINF] at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:334) [JBossINF] at org.jboss.as.ee.component.ComponentStartService.stop(ComponentStartService.java:49) [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30] [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30] [JBossINF] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30] [JBossINF] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.loaders.bucket.Bucket [JBossINF] Caused by: an exception which occurred: [JBossINF] in object org.infinispan.loaders.bucket.Bucket@3590ec87 [JBossINF] [JBossINF] 07:42:59,851 INFO [org.jboss.as] JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 581ms {noformat} http://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-shutdown-repl-sync/6/console-perf20/
Yes, this issue is still valid and affects CR1, e.g. in this run: https://hudson.qa.jboss.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-dist-async/2/ Marking docs as required and moving forward.
Release Notes Docs Status: Added: Not Yet Documented
Link: Added: This issue Cloned to JBPAPP-9248
Link: Added: This issue is a dependency of JBPAPP-9290
Link: Removed: This issue is related to JBPAPP-7577
Release Notes Docs Status: Removed: Not Yet Documented Added: Documented as Known Issue Release Notes Text: Added: Undeploying an application from one node in a cluster can result in <code>NotSerializableException</code> exceptions. This is due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent <code>DistributedCacheManagerFactory</code>, which depends on the cache container. The cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null, causing the NotSerializableException. The fix for this issue is still under investigation. Affects: Added: Release Notes
Writer: Added: mistysj
Am I clear to redocument this as a fixed issue?
Release Notes Docs Status: Removed: Documented as Known Issue Added: Needs More Info
We need feedback from development here. Paul, can you please comment?
Pinged Paul Ferraro today on IRC for comment on this one.
I've closed the parent jira (AS7-4950).
Again, I've documented this as resolved, but will leave it up to others to resolve the JIRA.
Release Notes Docs Status: Removed: Needs More Info Added: Documented as Resolved Issue Release Notes Text: Removed: Undeploying an application from one node in a cluster can result in <code>NotSerializableException</code> exceptions. This is due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent <code>DistributedCacheManagerFactory</code>, which depends on the cache container. The cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null, causing the NotSerializableException. The fix for this issue is still under investigation. Added: Undeploying an application from one node in a cluster could result in `NotSerializableException` exceptions. This was due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent `DistributedCacheManagerFactory`, which depends on the cache container. The cache container stopped while the web session cache was in the process of stopping and the externalizer on which it depended suddenly became null, causing the `NotSerializableException`. This has been resolved by upgrading the Infinispan component, and the exception no longer occurs.
Release Notes Docs Status: Removed: Documented as Resolved Issue Writer: Removed: mistysj Release Notes Text: Removed: Undeploying an application from one node in a cluster could result in `NotSerializableException` exceptions. This was due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent `DistributedCacheManagerFactory`, which depends on the cache container. The cache container stopped while the web session cache was in the process of stopping and the externalizer on which it depended suddenly became null, causing the `NotSerializableException`. This has been resolved by upgrading the Infinispan component, and the exception no longer occurs. Docs QE Status: Removed: NEW