Bug 907499 - NPE after ReplaceCommand
Summary: NPE after ReplaceCommand
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: CR1
: 6.1.0
Assignee: Tristan Tarrant
QA Contact: Nobody
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-02-04 14:58 UTC by Michal Linhard
Modified: 2018-09-12 22:30 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Occasionally during a conditional replace operation performed via the REST interface, a NullPointerException is thrown. This causes a NullPointerException ERROR to be reported during the replace operation. The actual outcome of the replace operation in this case is undetermined, and there is currently no workaround for this issue.
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-2787 0 Critical Resolved NPE after ReplaceCommand 2017-02-22 18:52:59 UTC
Red Hat Issue Tracker ISPN-2846 0 Major Resolved OperationsDuringStateTransferTest fails if run on non-tx cache 2017-02-22 18:52:59 UTC

Description Michal Linhard 2013-02-04 14:58:59 UTC
https://issues.jboss.org/browse/ISPN-2787

Comment 1 JBoss JIRA Server 2013-02-07 13:43:34 UTC
Tristan Tarrant <ttarrant> made a comment on jira ISPN-2787

Looking at the code it appears that the only possible situation in which an NPE can happen, is that there is a pre-existing entry with a null value and a conditional replace is performed with the oldValue set to null.

Comment 2 JBoss JIRA Server 2013-02-15 12:04:28 UTC
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Coding In Progress

Comment 3 JBoss JIRA Server 2013-02-15 15:33:28 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I thought we do not support null values in our caches ... weird. Investigating this now.

Comment 4 JBoss JIRA Server 2013-02-18 14:07:32 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I suspect this happens during state transfer, because of ISPN-2688. Will try to create a unit test for ISPN-2688 first.

Comment 5 JBoss JIRA Server 2013-02-22 09:58:59 UTC
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Resolved

Comment 6 JBoss JIRA Server 2013-02-22 09:58:59 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

Fixed by ISPN-2688.

Comment 8 JBoss JIRA Server 2013-02-25 11:55:17 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

I noticed only now that the cache is non-tx (here is a snippet from sever node03 log). ISPN-2688 was solved only for the TX case so this failure in REST suite is still present and the issue should be reopened.

{noformat}
19:26:03,063 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (http-/172.18.1.5:8080-792) ISPN000136: Execution error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from node04/default, see cause for remote stack trace
	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:310) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:241) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:236) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.NonTxConcurrentDistributionInterceptor.handleLocalWrite(NonTxConcurrentDistributionInterceptor.java:104) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleWriteCommand(NonTxDistributionInterceptor.java:106) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitReplaceCommand(BaseDistributionInterceptor.java:122) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitReplaceCommand(EntryWrappingInterceptor.java:207) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitReplaceCommand(NonTransactionalLockingInterceptor.java:150) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.statetransfer.StateTransferInterceptor.visitReplaceCommand(StateTransferInterceptor.java:156) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.AbstractVisitor.visitReplaceCommand(AbstractVisitor.java:72) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.commands.write.ReplaceCommand.acceptVisitor(ReplaceCommand.java:68) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1169) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replaceInternal(CacheImpl.java:835) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replace(CacheImpl.java:828) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.CacheImpl.replace(CacheImpl.java:823) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.AbstractDelegatingCache.replace(AbstractDelegatingCache.java:153) [infinispan-core-5.2.2.Final-redhat-1.jar:5.2.2.Final-redhat-1]
	at org.infinispan.rest.Server.putOrReplace(Server.scala:186) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.org$infinispan$rest$Server$$putInCache(Server.scala:157) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:133) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server$$anonfun$putEntry$1.apply(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.protectCacheNotFound(Server.scala:254) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at org.infinispan.rest.Server.putEntry(Server.scala:120) [infinispan-server-rest-5.2.2.Final-redhat-1-classes.jar:]
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.6.0_38]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_38]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_38]
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:536) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:513) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.4.Final-redhat-2.jar:2.3.4.Final-redhat-2]
{noformat}

Comment 9 JBoss JIRA Server 2013-02-25 12:51:04 UTC
Adrian Nistor <anistor> updated the status of jira ISPN-2787 to Reopened

Comment 10 JBoss JIRA Server 2013-02-28 17:33:35 UTC
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Resolved

Comment 11 JBoss JIRA Server 2013-02-28 17:33:35 UTC
Mircea Markus <mmarkus> made a comment on jira ISPN-2787

Duplicate of ISPN-2846.

Comment 12 JBoss JIRA Server 2013-03-01 11:09:09 UTC
Mircea Markus <mmarkus> updated the status of jira ISPN-2787 to Reopened

Comment 13 JBoss JIRA Server 2013-03-01 11:09:09 UTC
Mircea Markus <mmarkus> made a comment on jira ISPN-2787

the number of failures was reduced about 10 times, but there still are some failure. Looking into it

Comment 14 JBoss JIRA Server 2013-03-01 15:00:50 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

After adding TRACE logging to the REST suite the errors no longer happen. Unfortunately the logs without TRACE do not provide enough info to investigate this.

Comment 15 JBoss JIRA Server 2013-03-05 16:47:56 UTC
Adrian Nistor <anistor> made a comment on jira ISPN-2787

This only happens during cluster merge. I added MergeDuringReplaceTest to highlight the issue. The problem is in RpcManagerImpl.invokeRemotely(Collection<Address> recipients, ReplicableCommand rpcCommand, ResponseMode mode, long timeout, boolean usePriorityQueue, ResponseFilter responseFilter). There is an optimization there to avoid the remote call if the sender is the only member of the cluster. In that case it just returns an empty response map. This is most of the times incorrect (unless we are told to ignore leavers). A suspect exception for any of the destinations that are no longer in the cluster should be thrown instead. Not doing so will cause a null to be propagated to caller instead of an actual response which can later cause NPE in a seemingly unrelated place.

Comment 16 Michal Linhard 2013-03-06 09:31:15 UTC
Fine in JDG 6.1.0.CR1


Note You need to log in before you can comment on or make changes to this bug.