Bug 1158845 - Add Cross-Site state transfer operations and metrics to CLI
Summary: Add Cross-Site state transfer operations and metrics to CLI
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: 6.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ER3
: 6.4.0
Assignee: Pedro Ruivo
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-10-30 11:19 UTC by Pedro Ruivo
Modified: 2015-01-28 13:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-28 13:25:13 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-4811 0 Major Resolved Add Cross-Site state transfer operations and metrics to CLI 2014-10-30 11:19:18 UTC

Description Pedro Ruivo 2014-10-30 11:19:18 UTC
Add the operations to control the cross-site state transfer and the metrics to check the state of running state transfer to CLI.

Comment 2 Sebastian Łaskawiec 2014-11-03 13:09:24 UTC
PR: https://github.com/infinispan/jdg/pull/316

Comment 3 Matej Čimbora 2014-11-10 08:54:56 UTC
I came across 2 issues described below. They may be resolved as a part of this bz or this one can be closed (as the functionality is present as described) & new bugzillas will be created for each of the issues. Please comment which option is more appropriate and whether the behavior is unexpected.

1. When <backups><backup site="XYZ"/></backups> is not present in configuration of a given cache, "site" CLI operations are still available on the node. However, their usage leads to NPEs being thrown, e.g.   

31m08:33:50,092 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 1) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:46) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

[0m[31m08:34:11,802 ERROR [org.infinispan.cli.interpreter.Interpreter] (management-handler-thread - 2) ISPN019003: Interpreter error: java.lang.NullPointerException
	at org.infinispan.cli.interpreter.statement.SiteStatement.execute(SiteStatement.java:82) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:149) [infinispan-cli-server-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:238) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions$6.run(SecurityActions.java:235) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.security.Security.doPrivileged(Security.java:89) [infinispan-core-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:56) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.infinispan.server.infinispan.SecurityActions.executeInterpreter(SecurityActions.java:241) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:49) [infinispan-server-infinispan-6.2.0.ER3-redhat-1.jar:6.2.0.ER3-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:251) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:152) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:203) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:132) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_60]
	at javax.security.auth.Subject.doAs(Subject.java:415) [rt.jar:1.7.0_60]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:149) [jboss-as-controller-7.4.2.Final-redhat-2.jar:7.4.2.Final-redhat-2]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]

-----------------------------------------

2.

After invoking: site --cancelpush backupSite on the producer site, status of the push operation seems to get stuck at "SENDING" value (tested by site --pushstatus), even if state transfer is not currently in progress. Invoking site --cancelreceive mainSite on the consumer site works correctly. New invocation of 'push backupsite' leads to "X-Site state transfer to '%s' already started!" being displayed. The issue seems to be caused by XSiteStateTransferManagerImpl.siteCollector not being cleared.

Comment 4 Pedro Ruivo 2014-11-12 13:56:56 UTC
Hi Matej,

yep, they are both bugs. Please create a BZ for them and assign them to me. 

About the first, I'll try to find a way to disable the operations if the cross-site replication is not active. Otherwise, I'll report an error.

About the second, is it happening always?

Comment 5 Matej Čimbora 2014-11-12 15:55:15 UTC
Hi Pedro,

I tried to run the scenario again in lab instead of local machine & got the same error. 

Used configuration: 
distributed caches, site A: 2 nodes, site B: 3 nodes, B is a backup for A.

Scenario
- Start A,B
- Take B offline using takeSiteOffline
- Load data into A
- Push state into B
- CancelPushState B
-- PushStateStatus remains stuck at SENDING & new push is not possible


I'll mark this issue as verified as the purpose has been met and feature has been delivered. Other issues will be addressed in separate bugzillas, such as [1][2].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1163337
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1163332


Note You need to log in before you can comment on or make changes to this bug.