ISPN-2589 affects EAP 6.1.0.ER1: WARN [org.infinispan.topology.CacheTopologyControlCommand] (OOB-18,shared=udp) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, type=CH_UPDATE, sender=perf21/ejb, joinInfo=null, topologyId=9, currentCH=ReplicatedConsistentHash{members=[perf21/ejb, perf20/ejb]}, pendingCH=null, throwable=null, viewId=5}: java.lang.NullPointerException at org.infinispan.commands.write.InvalidateL1Command.perform(InvalidateL1Command.java:109) at org.infinispan.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:110) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.CacheLoaderInterceptor.visitInvalidateCommand(CacheLoaderInterceptor.java:128) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) at org.infinispan.interceptors.EntryWrappingInterceptor.visitInvalidateL1Command(EntryWrappingInterceptor.java:158) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitInvalidateL1Command(AbstractLockingInterceptor.java:98) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.MarshalledValueInterceptor.visitInvalidateCommand(MarshalledValueInterceptor.java:296) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251) at org.infinispan.interceptors.TxInterceptor.visitInvalidateCommand(TxInterceptor.java:221) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.statetransfer.StateTransferInterceptor.visitInvalidateL1Command(StateTransferInterceptor.java:173) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:85) at org.infinispan.commands.AbstractVisitor.visitInvalidateCommand(AbstractVisitor.java:141) at org.infinispan.commands.AbstractVisitor.visitInvalidateL1Command(AbstractVisitor.java:146) at org.infinispan.commands.write.InvalidateL1Command.acceptVisitor(InvalidateL1Command.java:192) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:834) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:319) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:194) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:60) at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:120) at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:194) at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:165) at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:137) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:253) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:484) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:707) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020) at org.jgroups.protocols.RSVP.up(RSVP.java:172) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453) at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD.up(FD.java:253) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE3.up(MERGE3.java:290) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616) at org.jgroups.protocols.TP.passMessageUp(TP.java:1263) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_38] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_38] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_38] Seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbremote-netDown-repl-async/10/console-perf19/
Still seeing this with EAP 6.1.0.ER8. For example: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-netDown-repl-sync/11/artifact/report/config/jboss-perf19/server.log https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-http-session-netDown-repl-async/11/artifact/report/config/jboss-perf20/server.log
N.B. L1 cache is not enabled by default in DIST caches in EAP 6.1, so this would only affect people who enable it explicitly.
This should already be fixed. Please verify.
Haven't seen this in 6.2 nor 6.3.0.DR0. Verified.
Paul, does this issue need a release note for the 6.3.0 release? The linked JIRA seems to suggest is was an unexpected but harmless message indicating a situation that was rarely (should never?) be encountered. If a release note is warranted, could you please provide some details in the Doc Text field above to allow ECS to craft one. Thanks.
This should not need a release note, since it was addressed in the last release cycle (6.2), even though it was finally verified in this cycle.