Bug 907499

Summary: NPE after ReplaceCommand
Product: [JBoss] JBoss Data Grid 6 Reporter: Michal Linhard <mlinhard>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: CLOSED UPSTREAM QA Contact: Nobody <nobody>
Severity: high Docs Contact:
Priority: high    
Version: 6.1.0CC: jdg-bugs, nobody
Target Milestone: CR1   
Target Release: 6.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Occasionally during a conditional replace operation performed via the REST interface, a NullPointerException is thrown. This causes a NullPointerException ERROR to be reported during the replace operation. The actual outcome of the replace operation in this case is undetermined, and there is currently no workaround for this issue.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2025-02-10 03:27:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michal Linhard 2013-02-04 14:58:59 UTC 
https://issues.jboss.org/browse/ISPN-2787
Comment 1 JBoss JIRA Server 2013-02-07 13:43:34 UTC 
Tristan Tarrant <ttarrant> made a comment on jira ISPN-2787

Looking at the code it appears that the only possible situation in which an NPE can happen, is that there is a pre-existing entry with a null value and a conditional replace is performed with the oldValue set to null.
Comment 2 JBoss JIRA Server 2013-02-15 12:04:28 UTC 
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Coding In Progress
Comment 3 JBoss JIRA Server 2013-02-15 15:33:28 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I thought we do not support null values in our caches ... weird. Investigating this now.
Comment 4 JBoss JIRA Server 2013-02-18 14:07:32 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I suspect this happens during state transfer, because of ISPN-2688. Will try to create a unit test for ISPN-2688 first.
Comment 5 JBoss JIRA Server 2013-02-22 09:58:59 UTC 
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Resolved
Comment 6 JBoss JIRA Server 2013-02-22 09:58:59 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

Fixed by ISPN-2688.
Comment 7 Michal Linhard 2013-02-25 10:41:02 UTC 
Still present in JDG 6.1.0.ER12 (Infinispan 5.2.2.Final)

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg-60-perf-client-stress-test-rest/92/artifact/report/size4/loganalysis/server/index.html
Comment 8 JBoss JIRA Server 2013-02-25 11:55:17 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I noticed only now that the cache is non-tx (here is a snippet from sever node03 log). ISPN-2688 was solved only for the TX case so this failure in REST suite is still present and the issue should be reopened.

{noformat}
19:26:03,063 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (http-/172.18.1.5:8080-792) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node04/default, see cause for remote stack trace
	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:310) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:241) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:236) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.NonTxConcurrentDistributionInterceptor.handleLocalWrite(NonTxConcurrentDistributionInterceptor.java:104) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleWriteCommand(NonTxDistributionInterceptor.java:106) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitReplaceCommand(BaseDistributionInterceptor.java:122) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitReplaceCommand(EntryWrappingInterceptor.java:207) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitReplaceCommand(NonTransactionalLockingInterceptor.java:150) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitReplaceCommand(StateTransferInterceptor.java:156) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1169) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replaceInternal(CacheImpl.java:835) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replace(CacheImpl.java:828) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replace(CacheImpl.java:823) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.AbstractDelegatingCache.replace(AbstractDelegatingCache.java:153) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.rest.Server.putOrReplace(Server.scala:186) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.org$infinispan$rest$Server$$putInCache(Server.scala:157) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:133) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.protectCacheNotFound(Server.scala:254) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.putEntry(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.6.0_38]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38]
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:536) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:513) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
{noformat}
Comment 9 JBoss JIRA Server 2013-02-25 12:51:04 UTC 
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Reopened
Comment 10 JBoss JIRA Server 2013-02-28 17:33:35 UTC 
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Resolved
Comment 11 JBoss JIRA Server 2013-02-28 17:33:35 UTC 
Mircea Markus <mmarkus> made a comment on jira ISPN-2787

Duplicate of ISPN-2846.
Comment 12 JBoss JIRA Server 2013-03-01 11:09:09 UTC 
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Reopened
Comment 13 JBoss JIRA Server 2013-03-01 11:09:09 UTC 
Mircea Markus <mmarkus> made a comment on jira ISPN-2787

the number of failures was reduced about 10 times, but there still are some failure. Looking into it
Comment 14 JBoss JIRA Server 2013-03-01 15:00:50 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

After adding TRACE logging to the REST suite the errors no longer happen. Unfortunately the logs without TRACE do not provide enough info to investigate this.
Comment 15 JBoss JIRA Server 2013-03-05 16:47:56 UTC 
Adrian Nistor <anistor> made a comment on jira ISPN-2787

This only happens during cluster merge. I added MergeDuringReplaceTest to highlight the issue. The problem is in RpcManagerImpl.invokeRemotely(Collection<Address> recipients, ReplicableCommand rpcCommand, ResponseMode mode, long timeout, boolean usePriorityQueue, ResponseFilter responseFilter). There is an optimization there to avoid the remote call if the sender is the only member of the cluster. In that case it just returns an empty response map. This is most of the times incorrect (unless we are told to ignore leavers). A suspect exception for any of the destinations that are no longer in the cluster should be thrown instead. Not doing so will cause a null to be propagated to caller instead of an actual response which can later cause NPE in a seemingly unrelated place.
Comment 16 Michal Linhard 2013-03-06 09:31:15 UTC 
Fine in JDG 6.1.0.CR1
Comment 21 Red Hat Bugzilla 2025-02-10 03:27:24 UTC 
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.