Bug 1379834 - EAP version identifier change results in loss/reset of plug-in configuration
Summary: EAP version identifier change results in loss/reset of plug-in configuration
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Inventory, Agent
Version: JON 3.3.7
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: CR01
: JON 3.3.8
Assignee: Ruben Vargas Palma
QA Contact: Hayk Hovsepyan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-27 19:54 UTC by Larry O'Leary
Modified: 2020-03-11 15:16 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1387299 (view as bug list)
Environment:
Last Closed: 2017-02-16 18:45:24 UTC
Type: Bug


Attachments (Terms of Use)
Excerpt from JBoss ON agent debug log showing property reset (28.43 KB, text/plain)
2016-09-27 19:54 UTC, Larry O'Leary
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1387299 0 urgent CLOSED EAP version identifier change results in loss/reset of plug-in configuration 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 2623591 0 None None None 2016-09-27 20:20:48 UTC
Red Hat Product Errata RHEA-2017:0285 0 normal SHIPPED_LIVE Red Hat JBoss Operations Network 3.3.8 bug fix update 2017-02-16 23:44:22 UTC

Internal Links: 1387299

Description Larry O'Leary 2016-09-27 19:54:16 UTC
Created attachment 1205307 [details]
Excerpt from JBoss ON agent debug log showing property reset

Description of problem:
When a patch/update is applied to an EAP server that is in inventory, it results in the loss of connection settings. For example, the management user, password, secure connection settings, etc. are restored to their post-import values. The result is that JBoss ON is no longer able to monitor/manage these resources until an administrator manually intervenes and restores the previous configuration.

Version-Release number of selected component (if applicable):
3.3.7

How reproducible:
Always

Steps to Reproduce:
1.  Install, configure, and start JBoss ON 3.3.6 system.
2.  Install EAP 6.4.0.GA server into its default base directory such as `jboss-eap-6.4`.
3.  Configure and start EAP 6.4.0 standalone-full server.
4.  Import EAP 6.4.0 server into JBoss ON inventory.
5.  From _Connection Settings_ page for newly imported EAP resource, set _user_ and _password_ to necessary values and _Truststore_ to `/data/fake-trust` or some other non-existent value.
6.  Verify EAP resource's availability is reported as _UP_.
7.  Shutdown EAP standalone server from outside of JBoss ON.
8.  Verify EAP resource's availability is reported as _DOWN_.
9.  Apply EAP 6.4 update-08 using `bin/jboss-cli.sh "patch apply path/to/jboss-eap-6.4.8-patch.zip"` command.
10. Start EAP 6.4.8 standalone-full server using the same command as before.
11. Once EAP server is started, execute JBoss ON discovery scan from platform resource.
12. Wait for JBoss ON discovery scan and inventory sync to complete. This could take a couple of minutes.

Actual results:
Connection settings user property value is changed to admin
Connection settings password property value is changed to "Unset"
Connection settings truststore property value is changed to "Unset"
From logs (attached agent-excerpt.log), it appears that this issue may also reset all other connection settings including host, port, trust passwords, etc.
The EAP server is no longer managed/monitored and is reported as DOWN and agent debug log reveals:

DEBUG [ResourceContainer.invoker.daemon-1] (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] [hostConfig: /data/jboss-eap-6.4/standalone/configuration/standalone-full.xml]: exception while checking availability
org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials.
	at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512)
	at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:465)
	at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:869)
	at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:860)
	at org.rhq.modules.plugins.jbossas7.BaseServerComponent.getAvailability(BaseServerComponent.java:133)
	at org.rhq.modules.plugins.jbossas7.BaseServerComponent.start(BaseServerComponent.java:114)
	at org.rhq.modules.plugins.jbossas7.StandaloneASComponent.start(StandaloneASComponent.java:91)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:759)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)


Expected results:
The EAP server should be reported as UP just like it was originally without its connection settings all getting reset.

Additional info:
Perhaps this is by design but this behavior makes using the management plug-in unusable considering that EAP has been doing inplace upgrades and updates since EAP 6.1. 

This was only tested with EAP 6. I am guessing that this same issue affects EAP 7 resources as well.

InventoryManager.mergeExistingResourcePluginConfiguration seems to be behind this. Perhaps the config diff process is backwards? If configuration is not the default, then we should not touch it. If it is the default, then the default discovery logic should be applied?

Comment 3 Larry O'Leary 2016-11-04 19:30:37 UTC
After further review of this issue, it has been made very clear that this is not a plug-in issue but rather a plug-in container InventoryManager issue. Because InventoryManager lives within the agent's libraries, any fix for this issue will most likely require a maintenance release of JBoss ON and its agent binary.

I am therefore re-assignment the component and target for inclusion in 3.3.8.

Comment 4 Richard Robinson 2016-11-07 15:14:41 UTC
What is the estimated date of release for JON 3.3.8?

Comment 6 Ruben Vargas Palma 2017-01-13 18:12:16 UTC
commit c260f0ce244e6008bf9e60d752980719f9504268
Author: Ruben Vargas <ruben.vp8510>
Date:   Sat Oct 22 19:29:17 2016 -0500

    On a resource update, merge only plug in read only property values.

Comment 9 Hayk Hovsepyan 2017-01-24 11:23:40 UTC
Hi Larry,

I was able to reproduce this BZ partly.
The only difference between what I encounter and what is described in steps, is in actual results: in my case, connection settings user property value is changed to "rhqadmin" instead of "admin" which was described in steps.
I had configured it to another super user.
And yes, all connection settings were reset after EAP6 patch install.

So my question is: What was the steps that make "admin" user to default one to reset connection settings user into it?
Or is the name of default user key point in this BZ, as I reproduced that it really reset settings to default?


And I see that bug is fixed in JON 3.3.8

Thanks,
Hayk

Comment 10 Stian Lund 2017-01-24 13:29:36 UTC
Hayk; 
the user property is reset to 'rhqadmin' by the system-user 'admin' so what you're seeing is the result we also get.

Also waiting for 3.3.8 here, I think pretty soon now :)

Comment 11 Larry O'Leary 2017-01-24 15:35:23 UTC
Hayk, 

The user I have in connection settings is "admin" due to the default provisioning used for EAP. My guess is that in your case, you either have used the (very bad) "Install RHQ user" operation, configured your EAP server's management user to be "rhqadmin", or you have not configured a management user at all for your EAP server.

In any event, as long as it switched back to what it was at initial discovery time, you have successfully reproduced the issue.

To answer your questions directly:

> So my question is: What was the steps that make "admin" user to default one to reset connection settings user into it?

Before my EAP instance was discovered, I added a management user to EAP with the user id of admin.

> Or is the name of default user key point in this BZ, as I reproduced that it really reset settings to default?

Exactly.

Comment 12 Hayk Hovsepyan 2017-01-24 16:19:57 UTC
Verified on revision ER01 build of JON 3.3.8.
Also, have reproduced that problem on EAP7 and verified the fix on it.

Comment 13 Filip Brychta 2017-02-03 17:05:46 UTC
Moving back to assigned because this fix introduce following NPE in agent.log:
2017-02-02 10:00:11,561 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Error in auto discovery
java.lang.NullPointerException
        at org.rhq.core.pc.inventory.InventoryManager.mergeExistingResourcePluginConfiguration(InventoryManager.java:1952)
        at org.rhq.core.pc.inventory.InventoryManager.mergeExistingResource(InventoryManager.java:1888)
        at org.rhq.core.pc.inventory.InventoryManager.mergeResourceFromDiscovery(InventoryManager.java:1794)
        at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.pluginDiscovery(AutoDiscoveryExecutor.java:189)
        at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:96)
        at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:83)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Reproduction steps:
- install JON 3.3.0 with all plugin packs
- upgrade to JON 3.3.8

Exception will be visible in log of co-located agent.

Comment 14 Ruben Vargas Palma 2017-02-03 17:45:46 UTC
commit 2b98508cfc5238433f61fb0e7313a059910880fd
Merge: a96f180 9eebe59
Author: Josejulio Martínez <finwemartinez>
Date:   Fri Feb 3 11:10:57 2017 -0600

    Merge pull request #292 from rubenvp8510/npe-inventory-merged
    
    Prevents NPE on mergeExistingResourcePluginConfiguration when property definition doesn't exist anymore.

commit 9eebe59c027908068211dbd481ce4944c35894fb
Author: Ruben Vargas <ruben.vp8510>
Date:   Fri Feb 3 10:29:00 2017 -0600

    Prevents NPE on mergeExistingResourcePluginConfiguration when property definition doesn't exist anymore.

Comment 16 Simeon Pinder 2017-02-08 06:03:24 UTC
Moving to ON_QA as available for test with the latest build:

JON 3.3.8 CR01 artifacts are available for test from here:
http://download.eng.bos.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/114/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip
 *Note: jon-server-patch-3.3.0.GA.zip maps to CR01 build of
 jon-server-3.3.0.GA-update-08.zip.

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=537179

Comment 17 Hayk Hovsepyan 2017-02-09 15:21:43 UTC
Verified on JON 3.3.8 CR01.
No NPE in agent log, issue is fixed.

Comment 18 errata-xmlrpc 2017-02-16 18:45:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2017-0285.html


Note You need to log in before you can comment on or make changes to this bug.