DIST_ASYNC. At server shutdown, we saw the following exception: 21:22:49,833 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-17) Failed to send entries to node perf20/web : java.lang.RuntimeException: Failure to marshal argument(s): org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257) at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_43] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_43] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43] at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43] Caused by: java.lang.RuntimeException: Failure to marshal argument(s) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ... 14 more Caused by: java.lang.InterruptedException: Cache manager is shutting down, so type write externalizer for type=java.lang.String cannot be resolved. Interruption being pushed up. at org.infinispan.marshall.jboss.ExternalizerTable.getObjectWriter(ExternalizerTable.java:185) at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:159) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:139) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:153) at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:103) at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:88) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50) at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119) at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96) at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92) at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279) ... 16 more Caused by: an exception which occurred: in object java.lang.String@f6a5adbc -> toString = qFQ00RtiqF3fi+HFCKsRV3oT in object org.infinispan.container.entries.ImmortalCacheEntry@fdc1c569 -> toString = ImmortalCacheEntry{key=qFQ00RtiqF3fi+HFCKsRV3oT, value=ImmortalCacheValue {value=AtomicHashMap}} in object java.util.ArrayList@9cf562ca Link to server log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/39/artifact/report/config/jboss-perf21/server.log In the same job, this issue was seen also in the following server.log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/39/artifact/report/config/jboss-perf20/server.log In the log of perf20, BZ 908333 was seen prior to this one. Link to job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/39/
Seen again during 6.1.0.ER8 testing. Link to server log: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/41/artifact/report/config/jboss-perf20/server.log Link to job: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-jvmkill-dist-async/41/
Still seeing this with EAP 6.1.1.ER7. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-sync/28/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/58/
Infinispan does not yet support clean shutdown. This is anticipated behavior.
This may no longer be an issue. Please retest against 6.3.0.GA.
I was instructed to retest against 6.3.0.DR1 and it's present there: for example shutdown, dist async cache: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-async/68/console-perf19/ undeploy, dist async cache: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/44/console-perf20/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-undeploy-dist-async/44/console-perf21/ but also dist sync cache (shutdown): https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/70/console-perf19/ https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-shutdown-dist-sync/70/console-perf20/
Still an issue, moving to 6.4.
Deferring clean shutdown issues to EAP7.