Bug 1004977

Summary: Datasource configuration changes SHOULD NOT require a complete reload/restart of EAP
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: Plugin -- JBoss EAP 6Assignee: Thomas Segismont <tsegismo>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: high Docs Contact:
Priority: high    
Version: JON 3.1.2CC: fbrychta, hrupp, lzoubek, onagano, rhatlapa, tsegismo
Target Milestone: ER05   
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The agent plug-in component supporting Datasource and XA Datasource resource types, like most JON plug-in components, used to re-apply every configuration property when a user changed resource configuration through the GUI or the CLI. After saving a datasource configuration change, the managed EAP 6 standalone or domain servers would put themselves in a "reload-required" state, even if the user only changed a property which does not require it. The plug-in component supporting Datasource and XA Datasource resource types now sends changed configuration properties only to the EAP 6 management interface. Additionally, the datasource "Disable" operation now has a "Allow service restart" parameter, which immediately stops the datasource service before changing a property which requires the datasource to be in a "Disabled" state. JON users can now manage datasources and XA-datasources following the workflows they are used to with the EAP6 CLI or Admin Console.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-11 14:03:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Larry O'Leary 2013-09-05 22:36:10 UTC
Description of problem:
Based on the EAP 6 console implementation, a user can update/change datasource configuration by disabling it, making their changes, and then enabling it. However, from JBoss ON, this is not possible. 

Although the datasource can be disabled, it can not be re-enabled making it impossible to apply the configuration update from JBoss ON. Instead, the user must resort to using the jboss-cli or the EAP management console.

Version-Release number of selected component (if applicable):
4.4.0.JON312GA

How reproducible:
Always

Steps to Reproduce:
1.  Start EAP 6 standalone.
2.  Start JON system.
3.  Set ExampleDS resource's connection pool max size property to 10.
4.  Execute the disable operation on ExampleDS.
5.  Execute the enable opeartion on ExampleDS.

Actual results:
Enable fails with detail message:

    java.lang.Exception: JBAS014749: Operation handler failed: Service jboss.data-source-config.ExampleDS is already registered, rolled-back=true
        at org.rhq.core.pc.operation.OperationInvocation.run(OperationInvocation.java:278)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

JON server log contains:
    DEBUG [WorkerThread#0[127.0.0.1:56922]] (enterprise.communications.command.server.CommandProcessor)- {CommandProcessor.executing}Executing command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[invokeOperation], targetInterfaceName=org.rhq.core.clientapi.agent.operation.OperationAgentService}]]
    DEBUG [WorkerThread#0[127.0.0.1:56922]] (enterprise.communications.command.server.CommandProcessor)- {CommandProcessor.executed}Executed command - response is [Command Response: isSuccessful=[true]; command=[null]; results=[null]; exception=[>> exception was null <<]]
    WARN  [ResourceContainer.invoker.nonDaemon-4] (rhq.modules.plugins.jbossas7.ASConnection)- We got a 500 with the following response body back: {"outcome" : "failed", "failure-description" : "JBAS014749: Operation handler failed: Service jboss.data-source-config.ExampleDS is already registered", "rolled-back" : true, "response-headers" : {"process-state" : "restart-required"}}

EAP server log contains:

    ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 26) JBAS014612: Operation ("enable") failed - address: ([
        {"subsystem" => "datasources"},
        {"data-source" => "ExampleDS"}
    ]): org.jboss.msc.service.DuplicateServiceException: Service jboss.data-source-config.ExampleDS is already registered
        at org.jboss.msc.service.ServiceRegistrationImpl.setInstance(ServiceRegistrationImpl.java:154) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl.startInstallation(ServiceControllerImpl.java:227) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:560) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:201) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2228) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:307) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
        at org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:955) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.connector.subsystems.datasources.DataSourceEnable.addServices(DataSourceEnable.java:182) [jboss-as-connector-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.connector.subsystems.datasources.DataSourceEnable$1.execute(DataSourceEnable.java:89) [jboss-as-connector-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        at org.jboss.as.connector.subsystems.datasources.DataSourceEnable.execute(DataSourceEnable.java:95) [jboss-as-connector-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
        ...

Expected results:
Datasource should be enabled and _AvailableCount_ should report `10`.

Additional info:
It appears that when the disable operation is invoked it does not apply the setting to the runtime meaning that although it is marked disabled, it isn't actually disabled. This is by design. 

The expectation is that the user must forcefully disable the datasource so that the acknowledge the fact that by doing so, all resource that depend on it will also be shutdown. 

I am not certain how the management console does this but using the JBoss CLI the following is necessary to successfully make the steps outlined about work:

    /subsystem=datasources/data-source=ExampleDS/statistics=pool:read-attribute(name="AvailableCount")
    /subsystem=datasources/data-source=ExampleDS:write-attribute(name="max-pool-size", value=10)
    /subsystem=datasources/data-source=ExampleDS:disable{allow-resource-service-restart=true}
    /subsystem=datasources/data-source=ExampleDS:enable
    /subsystem=datasources/data-source=ExampleDS/statistics=pool:read-attribute(name="AvailableCount")


The key is the request-header allow-resource-service-restart=true which indicates that we want the action to take affect right away resulting in a cascade action to all services that depend on it.

From a usability perspective, it seems that I should have the option to make configuration changes to be made available for the next scheduled restart or to force them to take effect now. Perhaps the configuration page itself should do this when I click _save_? It could ask me if I want to forcefully restart the service to apply the changes now or not? Then, the _disable_ operation could be updated to include an optional parameter to add the necessary header to force the action right now?

Comment 1 Thomas Segismont 2013-09-27 12:36:06 UTC
First look:

Datasources configuration properties have different behaviors in the management model:
* some of them (like pool size attributes) can be modified and saved even if the datasource is still enabled; but the changes will apply when the datasource is disabled then re-enabled
* some of them (like Connection URL) can only be modified and saved if the datasource was previously disabled

The admin console takes that into account (but does not indicate in which category a property falls).

Comment 2 Simeon Pinder 2013-10-24 04:16:22 UTC
Moving to ER05 as missed the cutoff for ER04.

Comment 3 Mike Foley 2013-11-11 16:23:29 UTC
moving to JON 3.2 per BZ Triage 11/11/2013 ...loleary, mfoley, heute, santos, yarboro

Comment 4 Mike Foley 2013-11-11 16:23:44 UTC
moving to JON 3.3 per BZ Triage 11/11/2013 ...loleary, mfoley, heute, santos, yarboro

Comment 6 Osamu Nagano 2013-12-11 06:19:38 UTC
Accroding to Bug 1024239, "allow-resource-service-restart" header is necessary to avoid server reloading like this in EAP 6 CLI.

~~~
[standalone@localhost:9999 /] /subsystem=datasources/data-source=ExampleDS:disable{allow-resource-service-restart=true}
{"outcome" => "success"}
~~~

EAP web console is doing this.  Probably JON can do the same.

Comment 8 Larry O'Leary 2014-02-13 18:14:16 UTC
As EAP and its management plug-in are not currently able to properly identify which properties can be updated without a complete restart and which properties actually have been modified (most likely a JBoss ON issue) this issue is being re-targeted for consideration for 3.3.

Comment 10 Thomas Segismont 2014-08-22 15:52:33 UTC
*** Bug 1036787 has been marked as a duplicate of this bug. ***

Comment 12 Thomas Segismont 2014-09-05 09:57:11 UTC
*** Bug 1020880 has been marked as a duplicate of this bug. ***

Comment 13 Thomas Segismont 2014-10-08 15:53:39 UTC
Pull request sent
https://github.com/rhq-project/rhq/pull/136

Comment 14 Thomas Segismont 2014-10-09 15:34:47 UTC
Pull request reviewed and tested by Libor. Waiting for PR Jenkins build to finish.

See https://github.com/rhq-project/rhq/pull/136#issuecomment-58510888

Comment 15 Thomas Segismont 2014-10-09 16:17:18 UTC
Fixed in master

commit bd8402ecec746fdd600e58daa1952e448ebffd52
Author: Thomas Segismont <tsegismo>
Date:   Wed Oct 8 17:42:09 2014 +0200
    
    "Enabled" attribute is no longer an editable configuration property (it will still be available on (xa-)datasource creation).
    Instead, users will have to use the "Enable"/"Disable" operations.
    
    "Disable" operation definition has been changed to include a new "allow-resource-service-restart" property, defaulting to false.
    If set to true, the corresponding management request header will be added. This allows to disable a datasource without a full server reload.
    
    The datasource component update configuration logic has been enhanced to send only configuration changes. It requires an extra request
    to read the current datasource config, but it avoids a server reload or datasource disable/enable cycle if the user only wants to
    change a set of properties which do not require it.
    
    Also:
    
    1. A new specific component class has been added for the datasources subsystem resource type. It used to be shared with the (xa-)datasource component.
    While in the beginning it probably was easier to have a single class for code sharing between the "Create Child Resource" process and the
    "Update Configuration Process", the (xa-)datasource component logic has become quite complex so it's safer to have separate component classes.
    
    2. The plugin descriptor has been updated to document that some (xa-)datasource properties cannot be unset if they already have a value.
    
    3. Perf: avoid compiling regular expression patterns over and over in AS7 plugin hotspots
    
    4. Code cleanup, like avoid string concat when working with a stringBuilder (it creates a useless additional builder)

Comment 16 Libor Zoubek 2014-10-10 13:23:19 UTC
branch:  release/jon3.3.x
link:    https://github.com/rhq-project/rhq/commit/dfee834e2
time:    2014-10-10 15:22:42 +0200
commit:  dfee834e28751fbb41a9609a5c6bc33e641b858e
author:  Thomas Segismont - tsegismo
message: Bug 1004977 - Datasource configuration changes SHOULD NOT require a
         complete reload/restart of EAP
         "Enabled" attribute is no longer an editable configuration
         property (it will still be available on (xa-)datasource
         creation). Instead, users will have to use the
         "Enable"/"Disable" operations.
         "Disable" operation definition has been changed to include a
         new "allow-resource-service-restart" property, defaulting to
         false. If set to true, the corresponding management request
         header will be added. This allows to disable a datasource
         without a full server reload.
         The datasource component update configuration logic has been
         enhanced to send only configuration changes. It requires an
         extra request to read the current datasource config, but it
         avoids a server reload or datasource disable/enable cycle if
         the user only wants to change a set of properties which do not
         require it.
         Also:
         1. A new specific component class has been added for the
         datasources subsystem resource type. It used to be shared with
         the (xa-)datasource component. While in the beginning it
         probably was easier to have a single class for code sharing
         between the "Create Child Resource" process and the
         "Update Configuration Process", the (xa-)datasource component
         logic has become quite complex so it's safer to have separate
         component classes.
         2. The plugin descriptor has been updated to document that some
         (xa-)datasource properties cannot be unset if they already have
         a value.
         3. Perf: avoid compiling regular expression patterns over and
         over in AS7 plugin hotspots
         4. Code cleanup, like avoid string concat when working with a
         stringBuilder (it creates a useless additional builder)
         (cherry picked from commit
         bd8402ecec746fdd600e58daa1952e448ebffd52) Signed-off-by: Libor
         Zoubek <lzoubek>
         Conflicts:
         modules/plugins/jboss-as-7/src/main/java/org/rhq/modules/plugins/jbossas7/DatasourceComponent.java

Comment 18 Simeon Pinder 2014-10-21 20:24:00 UTC
Moving to ON_QA as available to test with the latest brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=394734

Comment 19 Filip Brychta 2014-10-23 12:29:05 UTC
This bz is verified on
Version :	
3.3.0.ER05
Build Number :	
92b6d6a:2cdb528

but it probably caused this regression bz1156023