Red Hat Bugzilla – Bug 813967
Unable to upgrade agent plugins after a server upgrade
Last modified: 2015-02-01 18:27:52 EST
+++ This bug was initially created as a clone of Bug #812493 +++
Description of problem: After upgrading JON server from 3.0.0.GA to 3.0.1, uploading 3.0.1 versioned agent plugins results in server detecting them, then deleting them as obsolete. The end result is that the 3.0.0.GA plugin still seems to be used rather than the 3.0.1 plugin that was uploaded.
Version-Release number of selected component (if applicable): Upgrading from JON 3.0.0.GA to 3.0.1. Agent successfully auto-updated from 4.2.0.JON300.GA to 4.2.0.JON.3.0.1.GA.
How reproducible: Always
Steps to Reproduce:
1. Given a running JON 3.0.0.GA installation, follow instructions to upgrade server (shut down, unpack new version, start up, go to install page in browser and run install).
2. Given running 4.2.0.JON300.GA agent, with auto-update enabled, allow agent to auto-update after server is successfully upgraded.
3. Once server and agent have been successfully updated, download new 3.0.1 versions of plugins and unpack.
4. Upload 3.0.1 version of tomcat plugin, then click "Scan for Updates" so the server will pick up the new plugin.
5. Upload 3.0.1 version of EAP 5 plugin, then click "Scan for Updates" so the server will pick up the new plugin.
5. You should see something like the following output in the server log (note that the server detects the new plugin, copies it to the appropriate directory, then proceeds to delete it instead of deleting the old version):
19:14:34,291 INFO [ContainerBase] org.rhq.enterprise.gui.coregui.CoreGUI PluginFileUploadServlet: file was uploaded: jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar
19:14:34,294 INFO [PluginFileUploadServlet] A new plugin [jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar] has been uploaded to [/tmp/upload__32441646_136ade22675__7ffb_00000003.tmp]
19:14:34,300 INFO [PluginFileUploadServlet] A new plugin has been deployed [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/plugins/jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar]. A scan is required now in order to register it.
19:14:56,440 INFO [PluginDeploymentScanner] Found plugin jar at [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/plugins/jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar] and placed it at [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar]
19:14:57,774 INFO [AgentPluginScanner] Deleted an obsolete agent plugin file: /home/dvb/jboss/server/jon/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/jopr-tomcat-plugin-4.2.0.JON.3.0.1.GA.jar
19:21:41,070 INFO [ContainerBase] org.rhq.enterprise.gui.coregui.CoreGUI PluginFileUploadServlet: file was uploaded: jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar
19:21:41,078 INFO [PluginFileUploadServlet] A new plugin [jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar] has been uploaded to [/tmp/upload__32441646_136ade22675__7ffb_00000007.tmp]
19:21:41,086 INFO [PluginFileUploadServlet] A new plugin has been deployed [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/plugins/jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar]. A scan is required now in order to register it.
19:21:46,044 INFO [PluginDeploymentScanner] Found plugin jar at [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/plugins/jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar] and placed it at [/home/dvb/jboss/server/jon/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar]
19:21:47,456 INFO [AgentPluginScanner] Deleted an obsolete agent plugin file: /home/dvb/jboss/server/jon/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/jopr-jboss-as-5-plugin-4.2.0.JON.3.0.1.GA.jar
Actual results: Newly installed agent plugin is deleted after it is detected and copied to the appropriate location.
Expected results: Old version of the plugin is deleted as obsolete and newly installed agent plugin is preserved.
Additional info: A workaround would appear to be deleting the old plugin via the filesystem before uploading the new one.
--- Additional comment from email@example.com on 2012-04-17 10:19:06 EDT ---
documenting that i am encountering a BZ while attempting to repro this.
after step #1 ... i am uploading some JON 3.0 agent plug-ins. the UI is now allowing me to upload 1 plug-in ...but not a 2nd plug-in.
--- Additional comment from firstname.lastname@example.org on 2012-04-17 10:52:45 EDT ---
Attempting to repro ... attaching my server log.
--- Additional comment from email@example.com on 2012-04-17 10:53:26 EDT ---
Created attachment 578065 [details]
--- Additional comment from firstname.lastname@example.org on 2012-04-17 10:56:00 EDT ---
Copying a relevant snippet from my server log below
2012-04-17 10:50:47,898 INFO [org.rhq.enterprise.server.core.plugin.AgentPluginScanner] Deleted an obsolete agent plugin file: /root/rhq/BZ812493/JON301/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/jopr-jboss-as-plugin-4.2.0.JON.3.0.1.GA.jar
I believe this documents repro of the issue. I see the 3.01 plugin being identified and deleted as obsolete (incorrect).
--- Additional comment from email@example.com on 2012-04-17 11:22:27 EDT ---
attaching an image that shows Administration/Agent Plug-ins page ...
this image shows ...based on the timestamp only ... that the plugin update did not occur.
other than this ... there is no visual indication that the upgrade of the plug-in failed ... silent failure of the plugin upgrade?
--- Additional comment from firstname.lastname@example.org on 2012-04-17 11:23:06 EDT ---
Created attachment 578074 [details]
image of Administration/Agent Page
--- Additional comment from email@example.com on 2012-04-17 11:38:21 EDT ---
for reference, i am adding a link to the JON 3.01 upgrade documentation
for the purpose of comparing what is documented ... vs the repro steps above
I'm not sure whether the documented upgrade steps were executed, but if they were I think you would still hit the issue. I asked mazz to check what are plugin version comparator is returning comparing the 3.0.0 and 3.0.1 versions:
Failed tests: foo(org.apache.maven.artifact.versioning.ComparableVersionTest): 1 should be before 2: 1=[4.2.0.JON300.GA]; 2=[4.2.0.JON.3.0.1.GA]
So it looks like adding the periods to the 3.0.1 version string made it appear lexiographically before the 3.0.0 version.
We'll need to investigate a fix to 3.0.1 to address this.
It appears the new version string format is bad and results in RHQ thinking the new version is actually older.
Our old version: 4.2.0.JON300.GA
Our new version: 4.2.0.JON.3.0.1.GA
Those added "." characters in the new version format is causing the version check algorithm to think the 3.0.1 version is older than the 3.0.0 version.
Here's the problem - our version checking algorithm never expected a version inside a version. Here we have 4.2.0 and that is the version we parse (that is the major.minor.patch). But then we have this JON string that follows which is in the place of where "GA" or "RC" or something similar is supposed to go (based on our version syntax rules we use). Following "JON" we have ANOTHER apparent version string that we are somehow magically expecting RHQ to parse and understand as a major.minor.patch version string. That's not what is happening. We parse major.minor.patch that we see in the beginning - if something follows, we don't parse it as a SECOND version string - we look at it as the GA or RC or similar marking. (thus JON300.GA is compared to JON.3.0.1.GA - we do NOT see this as 3.0.0 compared to 3.0.1).
My proposal to work around this is to simply hardcode a check for "4.2.0.JON.3.0.1.GA" and in-memory change it to 4.2.0.JON301.GA to match our older version format. We should then release future versions by continuing our version format.
Either that or change "4.2.0.JON300.GA" to "4.2.0.JON.3.0.0.GA" in memory and continue with our new version format.
We need to hardcode the fix to make this easily patched WITHOUT screwing up third-party/community plugins that are following our current version syntax rules. We then need to ensure our next release(s) use version strings that are compared as we expect them.
The code for this (and a unit test) is ComparableVersion[Test]
git commit to master b83f264
after some back and forth, in the end, this commit simply introduces a hack that converts the old version string to the better, new one. This limits the impact and therefore has less risk. Was thinking about introducing a customization feature where we can add bad versions to some XML file so if this happens again we can just update the XML with the bad versions. But that would require storing that file on all servers in the HA cloud and the question is where to put the file. In the end, didn't waste more time on this by implementing a more complicated solution - we just modify ComparableVersion so it hardcodes the bad version and converts it to the good. There is a unit test method that checks that it works.
I tested JON 3.0.0 -> JON 3.0.1 upgrade - specifically looking at RHQ_PLUGIN table and watched the version column get upgraded as the new plugins got deployed and updated.
moving back to on_dev - after discussing with ccrouch, we are going to change
the version strings in the future back to what they were before - no dots in
the JON version string. So in the hack, we'll convert the JON.3.0.1.GA to
(In reply to comment #4)
> moving back to on_dev - after discussing with ccrouch, we are going to change
> the version strings in the future back to what they were before - no dots in
> the JON version string. So in the hack, we'll convert the JON.3.0.1.GA to
git commit to master 32137ba
this also adds a set of unit tests that check versions from past, current and several future versions of RHQ and JON.
git commit to master 56eb5ef - i messed up the version string in the test for 3.0.2. this fixes it. we want to match exactly what we will be using
Bulk closing of BZs that have no target version set, but which are ON_QA for more than a year and thus are in production for a long time.