Bug 900199 (JBPAPP6-1397) - NotSerializableException: org.infinispan.loaders.bucket.Bucket on undeploying application
Summary: NotSerializableException: org.infinispan.loaders.bucket.Bucket on undeploying...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: JBPAPP6-1397
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: TBD EAP 6
Assignee: Paul Ferraro
QA Contact:
URL: http://jira.jboss.org/jira/browse/JBP...
Whiteboard: eap6_clustering eap6_need_triage
Depends On:
Blocks: JBPAPP6-1422
TreeView+ depends on / blocked
 
Reported: 2012-04-12 22:16 UTC by Radoslav Husar
Modified: 2014-06-28 12:32 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-15 17:01:45 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker AS7-4950 0 Major Closed CLONE - NotSerializableException: org.infinispan.loaders.bucket.Bucket on undeploying application 2017-07-24 02:02:45 UTC
Red Hat Issue Tracker JBPAPP6-1397 0 Major Closed NotSerializableException: org.infinispan.loaders.bucket.Bucket on undeploying application 2017-07-24 02:02:46 UTC

Description Radoslav Husar 2012-04-12 22:16:43 UTC
Affects: Release Notes
project_key: JBPAPP6

Just undeploying application from one node in the cluster. Application is still being accessed by remote ejb clients.

{noformat}
[JBossINF] 18:01:15,248 INFO  [org.jboss.as.osgi] (MSC service thread 1-2) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-ejb.jar:main" from Service Module Loader
[JBossINF] 18:01:15,249 INFO  [org.jboss.as.osgi] (MSC service thread 1-8) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear:main" from Service Module Loader
[JBossINF] 18:01:15,253 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-15) ISPN000029: Passivating all entries to disk
[JBossINF] 18:01:15,249 INFO  [org.jboss.as.osgi] (MSC service thread 1-6) JBAS011943: Unregister module: Module "deployment.clusterbench-ee6.ear.clusterbench-ee6-web.war:main" from Service Module Loader
[JBossINF] 18:01:15,255 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-15) ISPN000030: Passivated 0 entries in 2 milliseconds
[JBossINF] 18:01:15,269 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk
[JBossINF] 18:01:15,273 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000029: Passivating all entries to disk
[JBossINF] 18:01:15,270 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container
[JBossINF] 18:01:15,275 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-8) ISPN000030: Passivated 0 entries in 1 milliseconds
[JBossINF] 18:01:15,296 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010282: Stopped //default-host//clusterbench cache from web container
[JBossINF] 18:01:15,303 INFO  [org.jboss.weld.deployer] (MSC service thread 1-10) JBAS016009: Stopping weld service for deployment clusterbench-ee6.ear
[JBossINF] 18:01:15,311 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-20-thread-2) ISPN000029: Passivating all entries to disk
[JBossINF] 18:01:15,313 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-20-thread-2) ISPN000030: Passivated 0 entries in 1 milliseconds
[JBossINF] 18:01:15,314 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-15-thread-2) ISPN000029: Passivating all entries to disk
[JBossINF] 18:01:15,317 INFO  [org.jboss.as.clustering.infinispan] (pool-20-thread-2) JBAS010282: Stopped repl cache from ejb container
[JBossINF] 18:01:15,326 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-16) JBAS015877: Stopped deployment clusterbench-ee6-ejb.jar in 82ms
[JBossINF] 18:01:15,328 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-12) JBAS015877: Stopped deployment clusterbench-ee6-web.war in 84ms
[JBossINF] 18:01:15,328 INFO  [org.jboss.as.clustering.infinispan] (pool-21-thread-2) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container
[JBossINF] 18:01:15,330 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-14) JBAS015877: Stopped deployment clusterbench-ee6.ear in 88ms
[JBossINF] 18:01:15,352 INFO  [org.infinispan.eviction.PassivationManagerImpl] (pool-15-thread-2) ISPN000030: Passivated 3 entries in 37 milliseconds
[JBossINF] 18:01:15,354 INFO  [org.jboss.as.clustering.infinispan] (pool-15-thread-2) JBAS010282: Stopped repl cache from web container
[JBossINF] 18:01:15,476 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-2) ISPN000082: Stopping the RpcDispatcher
[JBossINF] 18:01:15,498 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-19-thread-2) ISPN000082: Stopping the RpcDispatcher
[JBossINF] 18:01:15,523 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-9) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee6.ear"."clusterbench-ee6-ejb.jar".component.RemoteStatefulSBImpl.START: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl
[JBossINF] 	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:882)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:672)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:551)
[JBossINF] 	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:198)
[JBossINF] 	at org.infinispan.CacheImpl.stop(CacheImpl.java:516)
[JBossINF] 	at org.infinispan.CacheImpl.stop(CacheImpl.java:508)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:291)
[JBossINF] 	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:107)
[JBossINF] 	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:308)
[JBossINF] 	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:315)
[JBossINF] 	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:102)
[JBossINF] 	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:306)
[JBossINF] 	at org.jboss.as.ee.component.ComponentStartService.stop(ComponentStartService.java:49)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.loaders.bucket.Bucket
[JBossINF] Caused by: an exception which occurred:
[JBossINF] 	in object org.infinispan.loaders.bucket.Bucket@40a995c5
[JBossINF] 
[JBossINF] 18:01:15,916 INFO  [org.jboss.as.server] (management-handler-thread - 1) JBAS018558: Undeployed "clusterbench-ee6.ear"
{noformat}

Comment 2 Radoslav Husar 2012-04-12 22:17:47 UTC
Whether the issue is present upstream not yet known (needs testing).

Comment 3 Rajesh Rajasekaran 2012-04-13 15:03:37 UTC
Link: Added: This issue is related to JBPAPP-7577


Comment 4 Rajesh Rajasekaran 2012-04-13 17:53:53 UTC
I'm raising the priority on this JIRA as I believe this blocks getting valid runs for ejb3 SFSB replication performance. Rado, please correct me if I am wrong.

Comment 5 Rajesh Rajasekaran 2012-04-13 17:54:10 UTC
Labels: Added: eap6_clustering eap6_need_triage


Comment 6 Jason Greene 2012-04-13 21:07:03 UTC
Why would replication performance testing be undeploying applications? 

Comment 7 Radoslav Husar 2012-04-16 15:48:44 UTC
Jason is right, of course this does not have effect on performance tests. If people got here looking for the perf issue, it's this one JBPAPP-8721.

The issue has been spotted at failover tests. 

Comment 8 Radoslav Husar 2012-04-16 18:19:58 UTC
Link: Added: This issue depends ISPN-1981


Comment 9 Paul Ferraro 2012-04-19 13:04:45 UTC
Link: Added: This issue depends AS7-4559


Comment 10 Paul Ferraro 2012-04-20 21:45:45 UTC
This turns out to be due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent DistributedCacheManagerFactory, which depends on the cache container.  So, what happens is the cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null - causing the NotSerializableException.

Comment 11 Paul Ferraro 2012-04-24 18:17:19 UTC
Link: Added: This issue depends AS7-4612


Comment 12 Rajesh Rajasekaran 2012-04-30 15:18:25 UTC
Scheduling for ER7 as the upstream issue seems resolved. Requires an Infinispan upgrade. Assigning to Shelly to coordinate.

Comment 13 Paul Ferraro 2012-04-30 22:20:56 UTC
Link: Removed: This issue depends AS7-4559 


Comment 14 Shelly McGowan 2012-05-03 16:45:16 UTC
The commits reference in the AS7 JIRAs are in ER7.  ER7 also has an Infinispan upgrade (5.1.4.Final) but note the linked ISPN-1981 issue was Rejected.

Comment 17 Radoslav Husar 2012-05-24 12:33:55 UTC
The issue is still present in ER8 upon shutdown in EJB3 test scenarios:

{noformat}
[JBossINF] 07:42:59,546 WARN  [org.jboss.msc.service.fail] MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee6.ear"."clusterbench-ee6-ejb.jar".component.RemoteStatefulSBImpl.START: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.eviction.PassivationManagerImpl.passivateAll() throws org.infinispan.loaders.CacheLoaderException on object of type PassivationManagerImpl
[JBossINF] 	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:674)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:553)
[JBossINF] 	at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:199)
[JBossINF] 	at org.infinispan.CacheImpl.stop(CacheImpl.java:539)
[JBossINF] 	at org.infinispan.CacheImpl.stop(CacheImpl.java:531)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:348)
[JBossINF] 	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.stop(InfinispanBackingCacheEntryStore.java:117)
[JBossINF] 	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.stop(SerializationGroupMemberContainer.java:319)
[JBossINF] 	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.stop(PassivatingBackingCacheImpl.java:321)
[JBossINF] 	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.stop(AbstractCache.java:118)
[JBossINF] 	at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.stop(StatefulSessionComponent.java:334)
[JBossINF] 	at org.jboss.as.ee.component.ComponentStartService.stop(ComponentStartService.java:49)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] Caused by: org.infinispan.marshall.NotSerializableException: org.infinispan.loaders.bucket.Bucket
[JBossINF] Caused by: an exception which occurred:
[JBossINF] 	in object org.infinispan.loaders.bucket.Bucket@3590ec87
[JBossINF] 
[JBossINF] 07:42:59,851 INFO  [org.jboss.as] JBAS015950: JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) stopped in 581ms
{noformat}

http://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-shutdown-repl-sync/6/console-perf20/

Comment 19 Radoslav Husar 2012-06-05 13:12:47 UTC
Yes, this issue is still valid and affects CR1, e.g. in this run:
https://hudson.qa.jboss.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-dist-async/2/

Marking docs as required and moving forward.

Comment 20 Radoslav Husar 2012-06-05 13:12:47 UTC
Release Notes Docs Status: Added: Not Yet Documented


Comment 21 Radoslav Husar 2012-06-05 13:13:58 UTC
Link: Added: This issue Cloned to JBPAPP-9248


Comment 22 Rajesh Rajasekaran 2012-06-06 20:43:25 UTC
Link: Added: This issue is a dependency of JBPAPP-9290


Comment 23 Rajesh Rajasekaran 2012-06-06 20:51:13 UTC
Link: Removed: This issue is related to JBPAPP-7577 


Comment 24 Misty Stanley-Jones 2012-06-12 10:11:21 UTC
Release Notes Docs Status: Removed: Not Yet Documented Added: Documented as Known Issue
Release Notes Text: Added: Undeploying an application from one node in a cluster can result in <code>NotSerializableException</code> exceptions. This is due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent <code>DistributedCacheManagerFactory</code>, which depends on the cache container. The cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null, causing the NotSerializableException. The fix for this issue is still under investigation.
Affects: Added: Release Notes


Comment 27 Dana Mison 2012-10-16 05:54:34 UTC
Writer: Added: mistysj


Comment 28 Misty Stanley-Jones 2012-10-18 02:31:01 UTC
Am I clear to redocument this as a fixed issue?

Comment 29 Misty Stanley-Jones 2012-10-18 02:31:15 UTC
Release Notes Docs Status: Removed: Documented as Known Issue Added: Needs More Info


Comment 30 Jitka Kozana 2012-10-18 06:09:53 UTC
We need feedback from development here.
Paul, can you please comment? 

Comment 31 sgilda 2012-10-29 19:47:41 UTC
Pinged Paul Ferraro today on IRC for comment on this one.

Comment 32 Paul Ferraro 2012-10-30 17:06:18 UTC
I've closed the parent jira (AS7-4950).

Comment 33 Misty Stanley-Jones 2012-11-07 06:24:48 UTC
Again, I've documented this as resolved, but will leave it up to others to resolve the JIRA.

Comment 34 Misty Stanley-Jones 2012-11-07 06:24:48 UTC
Release Notes Docs Status: Removed: Needs More Info Added: Documented as Resolved Issue
Release Notes Text: Removed: Undeploying an application from one node in a cluster can result in <code>NotSerializableException</code> exceptions. This is due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent <code>DistributedCacheManagerFactory</code>, which depends on the cache container. The cache container stops while the web session cache is in the process of stopping and the externalizer on which it depends suddenly becomes null, causing the NotSerializableException. The fix for this issue is still under investigation. Added: Undeploying an application from one node in a cluster could result in `NotSerializableException` exceptions. This was due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent `DistributedCacheManagerFactory`, which depends on the cache container. The cache container stopped while the web session cache was in the process of stopping and the externalizer on which it depended suddenly became null, causing the `NotSerializableException`. This has been resolved by upgrading the Infinispan component, and the exception no longer occurs.


Comment 35 Anne-Louise Tangring 2012-11-13 20:57:54 UTC
Release Notes Docs Status: Removed: Documented as Resolved Issue 
Writer: Removed: mistysj 
Release Notes Text: Removed: Undeploying an application from one node in a cluster could result in `NotSerializableException` exceptions. This was due to a race condition between the shutdown of the web deployment (including the session manager) and the stopping of the dependent `DistributedCacheManagerFactory`, which depends on the cache container. The cache container stopped while the web session cache was in the process of stopping and the externalizer on which it depended suddenly became null, causing the `NotSerializableException`. This has been resolved by upgrading the Infinispan component, and the exception no longer occurs. 
Docs QE Status: Removed: NEW 



Note You need to log in before you can comment on or make changes to this bug.