https://issues.jboss.org/browse/ISPN-2787
Tristan Tarrant <ttarrant> made a comment on jira ISPN-2787 Looking at the code it appears that the only possible situation in which an NPE can happen, is that there is a pre-existing entry with a null value and a conditional replace is performed with the oldValue set to null.
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Coding In Progress
Adrian Nistor <anistor> made a comment on jira ISPN-2787 I thought we do not support null values in our caches ... weird. Investigating this now.
Adrian Nistor <anistor> made a comment on jira ISPN-2787 I suspect this happens during state transfer, because of ISPN-2688. Will try to create a unit test for ISPN-2688 first.
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Resolved
Adrian Nistor <anistor> made a comment on jira ISPN-2787 Fixed by ISPN-2688.
Still present in JDG 6.1.0.ER12 (Infinispan 5.2.2.Final) https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-client-stress-test-rest/92/artifact/report/size4/loganalysis/server/index.html
Adrian Nistor <anistor> made a comment on jira ISPN-2787 I noticed only now that the cache is non-tx (here is a snippet from sever node03 log). ISPN-2688 was solved only for the TX case so this failure in REST suite is still present and the issue should be reopened. {noformat} 19:26:03,063 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (http-/172.18.1.5:8080-792) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node04/default, see cause for remote stack trace at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:310) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:241) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:236) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.distribution.NonTxConcurrentDistributionInterceptor.handleLocalWrite(NonTxConcurrentDistributionInterceptor.java:104) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleWriteCommand(NonTxDistributionInterceptor.java:106) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitReplaceCommand(BaseDistributionInterceptor.java:122) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.EntryWrappingInterceptor.visitReplaceCommand(EntryWrappingInterceptor.java:207) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitReplaceCommand(NonTransactionalLockingInterceptor.java:150) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.statetransfer.StateTransferInterceptor.visitReplaceCommand(StateTransferInterceptor.java:156) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1169) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.CacheImpl.replaceInternal(CacheImpl.java:835) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.CacheImpl.replace(CacheImpl.java:828) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.CacheImpl.replace(CacheImpl.java:823) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.AbstractDelegatingCache.replace(AbstractDelegatingCache.java:153) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1] at org.infinispan.rest.Server.putOrReplace(Server.scala:186) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at org.infinispan.rest.Server.org$infinispan$rest$Server$$putInCache(Server.scala:157) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:133) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at org.infinispan.rest.Server.protectCacheNotFound(Server.scala:254) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at org.infinispan.rest.Server.putEntry(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:] at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.6.0_38] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:536) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:513) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2] {noformat}
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Reopened
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Resolved
Mircea Markus <mmarkus> made a comment on jira ISPN-2787 Duplicate of ISPN-2846.
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Reopened
Mircea Markus <mmarkus> made a comment on jira ISPN-2787 the number of failures was reduced about 10 times, but there still are some failure. Looking into it
Adrian Nistor <anistor> made a comment on jira ISPN-2787 After adding TRACE logging to the REST suite the errors no longer happen. Unfortunately the logs without TRACE do not provide enough info to investigate this.
Adrian Nistor <anistor> made a comment on jira ISPN-2787 This only happens during cluster merge. I added MergeDuringReplaceTest to highlight the issue. The problem is in RpcManagerImpl.invokeRemotely(Collection<Address> recipients, ReplicableCommand rpcCommand, ResponseMode mode, long timeout, boolean usePriorityQueue, ResponseFilter responseFilter). There is an optimization there to avoid the remote call if the sender is the only member of the cluster. In that case it just returns an empty response map. This is most of the times incorrect (unless we are told to ignore leavers). A suspect exception for any of the destinations that are no longer in the cluster should be thrown instead. Not doing so will cause a null to be propagated to caller instead of an actual response which can later cause NPE in a seemingly unrelated place.
Fine in JDG 6.1.0.CR1