Bug 1158845

Summary: Add Cross-Site state transfer operations and metrics to CLI
Product: [JBoss] JBoss Data Grid 6 Reporter: Pedro Ruivo <pruivo>
Component: InfinispanAssignee: Pedro Ruivo <pruivo>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: jdg-bugs, mcimbora, slaskawi
Target Milestone: ER3   
Target Release: 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-28 13:25:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pedro Ruivo 2014-10-30 11:19:18 UTC
Add the operations to control the cross-site state transfer and the metrics to check the state of running state transfer to CLI.

Comment 2 Sebastian Łaskawiec 2014-11-03 13:09:24 UTC
PR: https://github.com/infinispan/jdg/pull/316

Comment 3 Matej Čimbora 2014-11-10 08:54:56 UTC
I came across 2 issues described below. They may be resolved as a part of this bz or this one can be closed (as the functionality is present as described) & new bugzillas will be created for each of the issues. Please comment which option is more appropriate and whether the behavior is unexpected.

1. When <backups><backup site="XYZ"/></backups> is not present in configuration of a given cache, "site" CLI operations are still available on the node. However, their usage leads to NPEs being thrown, e.g.   

31m08:33:50,092 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 1) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:46) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

[0m[31m08:34:11,802 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 2) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:82) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

-----------------------------------------

2.

After invoking: site --cancelpush backupSite on the producer site, status of the push operation seems to get stuck at "SENDING" value (tested by site --pushstatus), even if state transfer is not currently in progress. Invoking site --cancelreceive mainSite on the consumer site works correctly. New invocation of 'push backupsite' leads to "X-Site state transfer to '%s' already started!" being displayed. The issue seems to be caused by XSiteStateTransferManagerImpl.siteCollector not being cleared.

Comment 4 Pedro Ruivo 2014-11-12 13:56:56 UTC
Hi Matej,

yep, they are both bugs. Please create a BZ for them and assign them to me. 

About the first, I'll try to find a way to disable the operations if the cross-site replication is not active. Otherwise, I'll report an error.

About the second, is it happening always?

Comment 5 Matej Čimbora 2014-11-12 15:55:15 UTC
Hi Pedro,

I tried to run the scenario again in lab instead of local machine & got the same error. 

Used configuration: 
distributed caches, site A: 2 nodes, site B: 3 nodes, B is a backup for A.

Scenario
- Start A,B
- Take B offline using takeSiteOffline
- Load data into A
- Push state into B
- CancelPushState B
-- PushStateStatus remains stuck at SENDING & new push is not possible


I'll mark this issue as verified as the purpose has been met and feature has been delivered. Other issues will be addressed in separate bugzillas, such as [1][2].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1163337
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1163332