Bug 1163332

Summary: Cross site state transfer - CLI ops throw NPE when backup is not defined
Product: [JBoss] JBoss Data Grid 6 Reporter: Matej Čimbora <mcimbora>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: jdg-bugs, pruivo, slaskawi
Target Milestone: ER7   
Target Release: 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Matej Čimbora 2014-11-12 14:16:52 UTC
Tested version: ER3, server

When <backups><backup site="XYZ"/></backups> is not present in configuration of a given cache, "site" CLI operations are still available on the node. However, their usage leads to NPEs being thrown, e.g.   

31m08:33:50,092 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 1) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:46) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

[0m[31m08:34:11,802 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 2) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:82) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

Comment 2 Sebastian Łaskawiec 2014-11-25 13:20:04 UTC
PR: https://github.com/infinispan/jdg/pull/365

Comment 3 Matej Čimbora 2014-12-04 11:11:33 UTC
I confirm the issue has been fixed by displaying appropriate message. 

One more NPE found on consumer site after invoking push operation on producer site. Occurs when corresponding cache does not exist there (either with the same name as the main cache or specified via <backup-for> option).

Producer site CLI:

Unable to pushState to 'LON'. org.infinispan.commons.CacheException: Problems invoking command.

Consumer site log:

[33m11:50:26,235 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (remote-thread-0) ISPN000071: Caught exception when handling command XSiteStateTransferControlCommand{control=START_RECEIVE, siteName='BRN', statusOk=false, cacheName='brnCache'}: java.lang.NullPointerException
	at org.infinispan.xsite.BackupReceiverImpl.invokeRemotelyInLocalSite(BackupReceiverImpl.java:212) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.xsite.BackupReceiverImpl.handleStateTransferControl(BackupReceiverImpl.java:89) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.xsite.statetransfer.XSiteStateTransferControlCommand.performInLocalSite(XSiteStateTransferControlCommand.java:41) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$3.run(CommandAwareRpcDispatcher.java:250) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]

[33m11:50:26,821 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (remote-thread-0) ISPN000071: Caught exception when handling command XSiteStateTransferControlCommand{control=FINISH_RECEIVE, siteName='BRN', statusOk=false, cacheName='brnCache'}: java.lang.NullPointerException
	at org.infinispan.xsite.BackupReceiverImpl.invokeRemotelyInLocalSite(BackupReceiverImpl.java:212) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.xsite.BackupReceiverImpl.handleStateTransferControl(BackupReceiverImpl.java:89) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.xsite.statetransfer.XSiteStateTransferControlCommand.performInLocalSite(XSiteStateTransferControlCommand.java:41) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$3.run(CommandAwareRpcDispatcher.java:250) [infinispan-core-6.2.0-redhat-SNAPSHOT.jar:6.2.0-redhat-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]

Comment 4 Matej Čimbora 2014-12-09 09:05:16 UTC
Verified, new bz will be created for aforementioned error.