Bug 736802

I am going to write a unit test for this functionality. I just did it for the plugin config update (LargeGroupPluginConfigurationTest) and was planning already to do it for the resource config updates.

assigning to bob - he's actually going to fix this. I'm just gonna write a unit test to see how bad/good the performance is.

unit test written - LargeGroupResourceConfigurationTest. Tests group of 1010 resources.

On Postgres, it took 39s for the initial configuration to be loaded (which included getting the live config from the agent and persisting that as the original config update). After updating the config, it took 15s to load the configs again. Relatively same numbers for Oracle.

This is only times for the loading of the initial and subsequent configs, although this unit test does exercise the actual scheduling of the config updates.

Pushing this fix to just the GWT timeout tracker

we improved this slightly by changing the RED error that pops up to a YELLOW warning when an outstanding config update is incomplete...


diff --git a/modules/enterprise/gui/coregui/src/main/java/org/rhq/enterprise/gui/coregui/client/inventory/groups/detail/configuration/GroupResourceConfigurationEditView.java b/modules/enterprise/gui/coregui/src/main/java/org/rhq/enterprise/gui/coregui/client/inventory/gro
index e75bfa2..454cbf7 100644
--- a/modules/enterprise/gui/coregui/src/main/java/org/rhq/enterprise/gui/coregui/client/inventory/groups/detail/configuration/GroupResourceConfigurationEditView.java
+++ b/modules/enterprise/gui/coregui/src/main/java/org/rhq/enterprise/gui/coregui/client/inventory/groups/detail/configuration/GroupResourceConfigurationEditView.java
@@ -198,7 +198,12 @@ public class GroupResourceConfigurationEditView extends LocatableVLayout impleme
 
     private void handleLoadFailure(Throwable caught) {
         refreshing = false;
-        CoreGUI.getErrorHandler().handleError(MSG.view_group_resConfig_edit_loadFail(group.toString()), caught);
+        if (caught.getMessage().contains("ConfigurationUpdateStillInProgressException")) {
+            final String msg = MSG.view_group_resConfig_edit_loadFail(group.toString());
+            CoreGUI.getMessageCenter().notify(new Message(msg, Message.Severity.Warning));
+        } else {
+            CoreGUI.getErrorHandler().handleError(MSG.view_group_resConfig_edit_loadFail(group.toString()), caught);
+        }
     }

  After some conversation and further thought, it would seem to me that this area of the product reveals a fundamental design approach that, while it works for small deployments with tens of nodes, it cannot scale to thousands of nodes. Product management has stated a requirement to support deployments in the low thousands of nodes. In fact the approach taken in much of the code similarly has as its basis an assumption that are not applicable in large deployments.

  One fundamental assumption is that consistency with respect to the observer, whose point of view is what is rendered in the JON console, is based upon and defined by what has been consistently enacted across all agents; consistency in this paradigm is achieved when state is guaranteed consistent across all agent nodes. As such, existing software commonly schedules asynchronous tasks as Quartz jobs which individually communicate with each agent node in sequence. Updating each node sequentially adds lots of latency and does not leverage compute power of the cluster.

  An alternative short term approach could better leverage compute resources by creating one Quartz job for every agent therefore transforming the sequential job into a parallel one. In the end, in the case associated to this defect, the user is blocked from making any further changes to the autogroup until such time as all agents have enacted the updated configuration. 

  So there are two fundamental design flaws here:

* sequential code rather than inherently parallelized code
* interface elements locked out awaiting completion events

  Leaving it as an exercise for the reader to lookup references on this topic, but promising references to these shortly, academic research material on the topic of distributed systems and consistency indicates that perfect consistency in distributed architectures is infeasible. While perfect consistency may be achieved in smaller deployments of ten or less nodes, and then only optimistically, in larger deployments where network partitions and other transient errors are commonplace, perfect consistency is not realistically feasible.

  An approach that would scale would be one that defines view consistency as one where user intent is paramount, not enacted/actualized state:

* views reflect deletions prior to any activity spanning multiple systems
  - perhaps through the use of offload tables, or queues
* views reflect insertions after all associated activities have completed
  - perhaps through the use of upload tables, or queues

  These suggest that deletions are asynchronous in nature, while insertions are synchronous - each with respect to state transfer. For the insertion case, once state has been transfered, control flow may return status codes or state synchronously. With regards to the user-client, all calls are synchronous, but internally activities such as those related to deletions may occur in the background, but the view MUST be consistent with user intent at the time control flow is returned.

  Let me provide some examples to make this a little more concrete; these are only provided as examples, and may not accurately reflect JON today, their intent is to make a point. Consider an group resource configuration updates. Updating the group resource configuration causes an update to each of the agents. Rather than updating all the agents synchronously, the "insertion" occurs in the database synchronously, but all updates to agents happen in parallel asynchronously. The user intent is immediately reflected in the view, represented in the database. Deployments could occur in the same manner. Deployments are interesting in that deletions are possible, the deletion request would occur synchronously with respect to the caller, and the (database) view would reflect this synchronously, but the undeployment activity would occur asynchronous with respect to the caller.

References:

[1] Byzantine Fault Tolerance
[2] Network Partitions
[3] View Consistency
[4] Eventual Consistency
[5] CAP Theorem

http://www.rhq-project.org/display/RHQ/Scaling+-+On+the+Definitions+of+View+Consistency