Red Hat Bugzilla – Bug 1058216
Struts portlet prevents cluster node from starting up properly
Last modified: 2014-07-04 00:08:25 EDT
Created attachment 855971 [details] Simple Struts portlet to reproduce the issue Description of problem: Using a struts based portlet in a cluster prevents new nodes from starting up properly. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Build attached simplest-hello-world-portlet 2. Set up a 2-node JPP 6.1 cluster 3. Deploy simplest-hello-world-portlet on both nodes 4. Start the first node, add the portlet to the main page, access the main page and submit the form 5. Start the second node Actual results: Second node fails to start up properly: 15:21:32,330 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 70) MSC000001: Failed to start service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217) at org.infinispan.CacheImpl.start(CacheImpl.java:582) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) ... 4 more Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/simplest-hello-world-portlet on node2/web at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) ... 18 more On the first node, the following exception is thrown: 15:20:33,853 ERROR [org.infinispan.marshall.VersionAwareMarshaller] (transport-thread-11) ISPN000065: Exception while marshalling object: java.io.NotActiveException: Fields were never written at org.jboss.marshalling.river.RiverObjectOutputStream.finish(RiverObjectOutputStream.java:175) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1009) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85) at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175) at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47) at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36) at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:459) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59) at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63) at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:47) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:250) at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:247) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:103) at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:88) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257) at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] Caused by: an exception which occurred: in field parameters in field navigationalState in object java.util.HashMap@7b2c68fe in object org.jboss.as.clustering.SimpleMarshalledValue@7b2c68fe Expected results: Starting of the second cluster node works without issues Additional info: This is caused by a serialization issue in ParameterMap, see https://issues.jboss.org/browse/GTNCOMMON-21
Boleslaw Dawidowicz <bdawidow@redhat.com> updated the status of jira GTNCOMMON-21 to Resolved
It will get fixed with upgrading version of gatein common
Unfortunatelly I can still reproduce this problem on both 6.2.0.ER1 and ER2.
Created attachment 898445 [details] exception in log of node1 when node2 is starting up (with 6.2.0.ER1)
Created attachment 898446 [details] exception in node2 log during startup (6.2.0.ER1)
Created attachment 898447 [details] exception in node2 log when page with portlet is accessed (6.2.0.ER1)
1) When node2 is starting up: - node1 log: ... org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) ... - node2 log: (the originally reported exception) ... org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl ... 2) When the page with the portlet is accessed on node2: - node2 log: ... org.gatein.pc.api.NoSuchPortletException: No such portlet /simplest-hello-world-portlet.SimplestHelloWorldPortlet ...
I have set up a 2 cluster nodes in ER2. The only thing I have seen is the following: - In the example attached, there are two configuration files: web.xml and portlet.xml In web.xml configuration parameter 'PortletScopeStrutsSession' is set a false but in portlet.xml is set a true. A true parameter indicates that ServletPortletSessionProxy will be used to handle portlet sessions and this class is not serializable, so it won't work under cluster environment. After setting this parameter to false in portlet.xml I can not reproduce this issue. Please, can you try it and confirm it ? Thanks, Lucas
With the "PortletScopeStrutsSession" init param set to "false" in both portlet.xml and web.xml I'm also unable to reproduce the problem in 6.2.0.ER2, however it's still present in 6.1.0.