Hide Forgot
Created attachment 1205307 [details] Excerpt from JBoss ON agent debug log showing property reset Description of problem: When a patch/update is applied to an EAP server that is in inventory, it results in the loss of connection settings. For example, the management user, password, secure connection settings, etc. are restored to their post-import values. The result is that JBoss ON is no longer able to monitor/manage these resources until an administrator manually intervenes and restores the previous configuration. Version-Release number of selected component (if applicable): 3.3.7 How reproducible: Always Steps to Reproduce: 1. Install, configure, and start JBoss ON 3.3.6 system. 2. Install EAP 6.4.0.GA server into its default base directory such as `jboss-eap-6.4`. 3. Configure and start EAP 6.4.0 standalone-full server. 4. Import EAP 6.4.0 server into JBoss ON inventory. 5. From _Connection Settings_ page for newly imported EAP resource, set _user_ and _password_ to necessary values and _Truststore_ to `/data/fake-trust` or some other non-existent value. 6. Verify EAP resource's availability is reported as _UP_. 7. Shutdown EAP standalone server from outside of JBoss ON. 8. Verify EAP resource's availability is reported as _DOWN_. 9. Apply EAP 6.4 update-08 using `bin/jboss-cli.sh "patch apply path/to/jboss-eap-6.4.8-patch.zip"` command. 10. Start EAP 6.4.8 standalone-full server using the same command as before. 11. Once EAP server is started, execute JBoss ON discovery scan from platform resource. 12. Wait for JBoss ON discovery scan and inventory sync to complete. This could take a couple of minutes. Actual results: Connection settings user property value is changed to admin Connection settings password property value is changed to "Unset" Connection settings truststore property value is changed to "Unset" From logs (attached agent-excerpt.log), it appears that this issue may also reset all other connection settings including host, port, trust passwords, etc. The EAP server is no longer managed/monitored and is reported as DOWN and agent debug log reveals: DEBUG [ResourceContainer.invoker.daemon-1] (rhq.modules.plugins.jbossas7.BaseServerComponent)- ResourceType[id=0, name=JBossAS7 Standalone Server, plugin=JBossAS7, category=Server] [hostConfig: /data/jboss-eap-6.4/standalone/configuration/standalone-full.xml]: exception while checking availability org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Credentials for plugin to connect to AS7 management interface are invalid - update Connection Settings with valid credentials. at org.rhq.modules.plugins.jbossas7.ASConnection.executeRaw(ASConnection.java:304) at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:512) at org.rhq.modules.plugins.jbossas7.ASConnection.execute(ASConnection.java:465) at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:869) at org.rhq.modules.plugins.jbossas7.BaseComponent.readAttribute(BaseComponent.java:860) at org.rhq.modules.plugins.jbossas7.BaseServerComponent.getAvailability(BaseServerComponent.java:133) at org.rhq.modules.plugins.jbossas7.BaseServerComponent.start(BaseServerComponent.java:114) at org.rhq.modules.plugins.jbossas7.StandaloneASComponent.start(StandaloneASComponent.java:91) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:759) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Expected results: The EAP server should be reported as UP just like it was originally without its connection settings all getting reset. Additional info: Perhaps this is by design but this behavior makes using the management plug-in unusable considering that EAP has been doing inplace upgrades and updates since EAP 6.1. This was only tested with EAP 6. I am guessing that this same issue affects EAP 7 resources as well. InventoryManager.mergeExistingResourcePluginConfiguration seems to be behind this. Perhaps the config diff process is backwards? If configuration is not the default, then we should not touch it. If it is the default, then the default discovery logic should be applied?
After further review of this issue, it has been made very clear that this is not a plug-in issue but rather a plug-in container InventoryManager issue. Because InventoryManager lives within the agent's libraries, any fix for this issue will most likely require a maintenance release of JBoss ON and its agent binary. I am therefore re-assignment the component and target for inclusion in 3.3.8.
What is the estimated date of release for JON 3.3.8?
commit c260f0ce244e6008bf9e60d752980719f9504268 Author: Ruben Vargas <ruben.vp8510> Date: Sat Oct 22 19:29:17 2016 -0500 On a resource update, merge only plug in read only property values.
Moving to ON_QA as available for test with build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=534002 http://download.eng.bos.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/110/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip maps to ER01 build of JON 3.3.8.
Hi Larry, I was able to reproduce this BZ partly. The only difference between what I encounter and what is described in steps, is in actual results: in my case, connection settings user property value is changed to "rhqadmin" instead of "admin" which was described in steps. I had configured it to another super user. And yes, all connection settings were reset after EAP6 patch install. So my question is: What was the steps that make "admin" user to default one to reset connection settings user into it? Or is the name of default user key point in this BZ, as I reproduced that it really reset settings to default? And I see that bug is fixed in JON 3.3.8 Thanks, Hayk
Hayk; the user property is reset to 'rhqadmin' by the system-user 'admin' so what you're seeing is the result we also get. Also waiting for 3.3.8 here, I think pretty soon now :)
Hayk, The user I have in connection settings is "admin" due to the default provisioning used for EAP. My guess is that in your case, you either have used the (very bad) "Install RHQ user" operation, configured your EAP server's management user to be "rhqadmin", or you have not configured a management user at all for your EAP server. In any event, as long as it switched back to what it was at initial discovery time, you have successfully reproduced the issue. To answer your questions directly: > So my question is: What was the steps that make "admin" user to default one to reset connection settings user into it? Before my EAP instance was discovered, I added a management user to EAP with the user id of admin. > Or is the name of default user key point in this BZ, as I reproduced that it really reset settings to default? Exactly.
Verified on revision ER01 build of JON 3.3.8. Also, have reproduced that problem on EAP7 and verified the fix on it.
Moving back to assigned because this fix introduce following NPE in agent.log: 2017-02-02 10:00:11,561 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Error in auto discovery java.lang.NullPointerException at org.rhq.core.pc.inventory.InventoryManager.mergeExistingResourcePluginConfiguration(InventoryManager.java:1952) at org.rhq.core.pc.inventory.InventoryManager.mergeExistingResource(InventoryManager.java:1888) at org.rhq.core.pc.inventory.InventoryManager.mergeResourceFromDiscovery(InventoryManager.java:1794) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.pluginDiscovery(AutoDiscoveryExecutor.java:189) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:96) at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:83) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Reproduction steps: - install JON 3.3.0 with all plugin packs - upgrade to JON 3.3.8 Exception will be visible in log of co-located agent.
commit 2b98508cfc5238433f61fb0e7313a059910880fd Merge: a96f180 9eebe59 Author: Josejulio Martínez <finwemartinez> Date: Fri Feb 3 11:10:57 2017 -0600 Merge pull request #292 from rubenvp8510/npe-inventory-merged Prevents NPE on mergeExistingResourcePluginConfiguration when property definition doesn't exist anymore. commit 9eebe59c027908068211dbd481ce4944c35894fb Author: Ruben Vargas <ruben.vp8510> Date: Fri Feb 3 10:29:00 2017 -0600 Prevents NPE on mergeExistingResourcePluginConfiguration when property definition doesn't exist anymore.
Moving to ON_QA as available for test with the latest build: JON 3.3.8 CR01 artifacts are available for test from here: http://download.eng.bos.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/114/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip *Note: jon-server-patch-3.3.0.GA.zip maps to CR01 build of jon-server-3.3.0.GA-update-08.zip. https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=537179
Verified on JON 3.3.8 CR01. No NPE in agent log, issue is fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2017-0285.html