Bug 830865 - Doc: Describe how to go to upgrade to 3.1GA if tech-preview AS7 plugin was used with jon3.0.0 or 3.0.1
Summary: Doc: Describe how to go to upgrade to 3.1GA if tech-preview AS7 plugin was us...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: Documentation
Version: 4.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: JON 3.1.0,RHQ 4.4.0
Assignee: Deon Ballard
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On: 818673 818919
Blocks: as7-plugin 801638
TreeView+ depends on / blocked
 
Reported: 2012-06-11 14:30 UTC by Charles Crouch
Modified: 2015-02-01 23:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 818673
Environment:
Last Closed: 2012-08-10 19:14:52 UTC
Embargoed:


Attachments (Terms of Use)

Description Charles Crouch 2012-06-11 14:30:01 UTC
+++ This bug was initially created as a clone of Bug #818673 +++

There are two reasons to do this:

1) since the release of the preview version of the AS7 plugin for JON, there have been changes made to the plugin descriptor that would cause our metadata update code to hang during an upgrade from the preview version of the plugin to the current version. So for customers upgrading from (JON 3.0.x + preview AS7 plugin) to (JON 3.1), their JON Server would hang during install due to the metadata update bug.

The metadata update issue was reported by Heiko in #rhq a month or so ago. I will be creating a separate BZ for that issue. The proposed workaround for the JON 3.1 upgrade issue at the time was to document in the release notes that the preview version of the AS7 plugin must be purged via the GUI, prior to upgrading to JON 3.0. The problem with this workaround is that many customers do not closely read the release notes. Any customer who does not will end up with a hung installer and have no idea what's going on. 

Renaming the plugin solves this, since the new AS7 plugin will be seen as a completely different plugin, and the metadata update bug will be completely avoided. Customers can then at their leisure go back and either 1) purge the old  AS7 plugin, or 2) disable the old AS7 plugin (they might do the latter if they wanted to keep the old Resources in inventory to hold on to some metric data or other historical data).

2) "jboss-as-7" isn't consistent with the names of all of the other core RHQ plugins, which are CapitalizedCamelCase (e.g. JBossAS5, Tomcat, Apache).


1) is the main reason to move ahead with this, though 2) is a nice additional benefit.

--- Additional comment from ian.springer on 2012-05-03 20:45:17 EDT ---

Created attachment 581992 [details]
a patch that implements this

To try this out in your 4.4.0-SNAPSHOT dev-container, you'll need to first purge your existing jboss-as-7 plugin - this is only because the filename of this plugin is identical to the new plugin (i.e. rhq-jboss-as-7-plugin-4.4.0-SNAPSHOT.jar). This purge will not be necessary when upgrading from RHQ 4.3 or JON 3.0.x, since the plugin filenames will not be the same.

--- Additional comment from ccrouch on 2012-05-03 22:36:15 EDT ---

Heiko, 
This BZ needs your blessing.
Ian,Heiko
We need to be as confident as possible this isn't going to destabilize the testsuite prior to pushing

--- Additional comment from lkrejci on 2012-05-04 07:11:11 EDT ---

Note that this change will break any code that depends on the plugin being called the old name.

We should search through the codebase of usages of the string "jboss-as-7" (not just java code but also any js scripts, etc) and make sure everything is aware of this change.

Notably bug 801638 needs revisited after this BZ has been implemented.

--- Additional comment from hrupp on 2012-05-04 08:21:51 EDT ---

The diff itself looks ok - due to the current issues with the test suite (Bug 818919) I was not able to test it.

--- Additional comment from hrupp on 2012-05-04 10:15:50 EDT ---

My biggest concern were the <c:option-source> expressions, but it looks like those have all been addressed. 

Pushed to master as version 7f3307e

--- Additional comment from fbrychta on 2012-05-30 11:20:48 EDT ---

Removed NEEDINFO flag after discussion with Ian

--- Additional comment from fbrychta on 2012-06-01 07:42:21 EDT ---

I followed this scenario:
1- jon3.0.1.GA + jon-plugin-pack-tech-preview-3.0.1.GA installed and running, eap6 running and imported to jon inventory
2- upgrade to jon3.1.0.RC1 + jon-plugin-pack-epp-3.1.0.CR1 according to jon upgrade manual
3- go to admistration and disable the old jBossAS7 plugin (the one from jon-plugin-pack-tech-preview-3.0.1.GA)
4- delete this plugin from jon administration
5- purge this plugin from jon administration (http://rhq-project.org/display/JOPR2/Agent+Plugin+Administration#AgentPluginAdministration-PurgingPlugins)


Results:
The old plugin was removed successfully (jar file from jon-server-3.1.0.CR1/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins, relevant row from rhq_plugin table, all references from gui)
The new plugin is working, eap6 was successfully imported to jon inventory

Found problems:
1- after step 4, the agent was shutdown and had to be started manually
2- errors and warns in agent.log

2012-06-01 11:19:28,051 WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.plugin-not-on-server}The plugin [plugins/rhq-script-plugin-4.2.0.JON.3.0.1.GA.jar] does not exist on the Server - renaming it to [rhq-script-plugin-4.2.0.JON.3.0.1.GA.jar.REJECTED] so it will not get deployed by the Plugin Container.
2012-06-01 11:19:28,051 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.plugin-rename-failed}Failed to rename illegitimate plugin [plugins/rhq-script-plugin-4.2.0.JON.3.0.1.GA.jar] to [rhq-script-plugin-4.2.0.JON.3.0.1.GA.jar.REJECTED].

Previous should be probably harmless??

2012-06-01 10:23:57,848 WARN  [ResourceDiscoveryComponent.invoker.daemon-4] (rhq.modules.plugins.jbossas7.ASConnection)- We got a 500 with the following response body back: {"outcome" : "failed", "failure-description" : "JBAS014739: No handler for read-resource at address [(\"subsystem\" => \"jaxr\")]", "rolled-back" : true}

2012-06-01 11:24:40,942 WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- An error occurred while checking for an updated Resource configuration for Resource[id=10002, uuid=9cf33767-33e5-4516-a43f-9ce8469f7773, type={jboss-as-7}JBossAS7 Standalone Server, key=/home/jon/jboss-eap-6.0, name=EAP /home/jon/jboss-eap-6.0, parent=dhcp-27-145.brq.redhat.com, version=EAP 6.0].
java.lang.Exception: java.lang.NoSuchMethodError: org.rhq.core.domain.configuration.definition.PropertyDefinitionMap.getPropertyDefinitions()Ljava/util/Map;
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:639)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.NoSuchMethodError: org.rhq.core.domain.configuration.definition.PropertyDefinitionMap.getPropertyDefinitions()Ljava/util/Map;
        at org.rhq.modules.plugins.jbossas7.ConfigurationLoadDelegate.loadHandlePropertyMap(ConfigurationLoadDelegate.java:264)
        at org.rhq.modules.plugins.jbossas7.ConfigurationLoadDelegate.loadHandleProperties(ConfigurationLoadDelegate.java:186)
.
.

2012-06-01 11:24:40,991 WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Plugin Error: Invalid Webservices resource configuration returned by jboss-as-7 plugin - Required property 'modify-soap-address' has a null value in Configuration[id=0].

2012-06-01 12:28:25,824 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Plugin Error: Failed to stop component for [Resource[id=10111, uuid=fdb24432-5f36-46f5-8705-57b3ec2ea4f2, type={JBossAS7}VHost, key=subsystem=web,virtual-server=default-host, name=default-host, parent=web]].
2012-06-01 12:28:26,262 WARN  [InventoryManager.discovery-1] (rhq.core.pc.plugin.PluginClassLoader)- Failed to purge embedded jars directory. Cause: java.io.IOException: Failed to delete file or directory: data/tmp/rhq-jboss-as-7-plugin-4.4.0.JON310CR1.jar6095728926369384907.classloader

this should be harmless as well, i checked data/tmp and it was empty

rhq server log doesn't contain any errors related to jBossPlugin

I executed this scenario twice with the same result. All logs from both runs attached

--- Additional comment from fbrychta on 2012-06-01 07:43:16 EDT ---

Created attachment 588422 [details]
logs for both runs

--- Additional comment from hrupp on 2012-06-04 10:59:54 EDT ---

Filip,
can you re-try with 301 installed and the old plugin.

Then delete + purge the old plugin.

Then shutdown the server and do the upgrade to 3.1 ?

--- Additional comment from fbrychta on 2012-06-05 09:54:12 EDT ---

Retested on this scenario:
1- jon3.0.1.GA + jon-plugin-pack-tech-preview-3.0.1.GA installed and running, eap6 running and imported to jon inventory, the rhq agent is running on the background
2- disable the jBossAS7 plugin
3- detete the jBossAS7 plugin  (the agent was stopped after this step so i started it manually )
4- purge the jBossAS7 plugin
5- upgrade to jon3.1.0.RC2 + jon-plugin-pack-eap-3.1.0.CR2
6- import eap6

Found problems:
1- the agent was stopped after step 3 - I tried to delete the jBossAS7 plugin from jon3.1.0.RC2 + jon-plugin-pack-eap-3.1.0.CR2 and the agent was stopped as well. I will create a separate BZ for this problem.
2- errors and warns in agent.log
2012-06-05 14:38:36,775 WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.plugin-not-on-server}The plugin [plugins/rhq-agent-plugin-4.2.0.JON.3.0.1.GA.jar] does not exist on the Server - renaming it to [rhq-agent-plugin-4.2.0.JON.3.0.1.GA.jar.REJECTED] so it will not get deployed by the Plugin Container.
2012-06-05 14:38:36,775 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.PluginUpdate)- {PluginUpdate.plugin-rename-failed}Failed to rename illegitimate plugin [plugins/rhq-agent-plugin-4.2.0.JON.3.0.1.GA.jar] to [rhq-agent-plugin-4.2.0.JON.3.0.1.GA.jar.REJECTED]. 

Should be harmless.

2012-06-05 14:40:58,925 WARN  [ResourceDiscoveryComponent.invoker.daemon-2] (rhq.modules.plugins.jbossas7.ASConnection)- We got a 500 with the following response body back: {"outcome" : "failed", "failure-description" : "JBAS014739: No handler for read-resource at address [(\"subsystem\" => \"jaxr\")]", "rolled-back" : true}

This is not related to upgrade, i found this warning even after clean installation of jon3.1.0.RC2 + jon-plugin-pack-eap-3.1.0.CR2 


Other warns from scenario from previous comment wasn't found. So the only problem for this scenario is that the agent was stopped and had to be started manually.

--- Additional comment from snegrea on 2012-06-05 12:31:12 EDT ---

The first two errors/warnings reported in comment #10 are not related to the AS7 plugin; they are more of a generic upgrade problem. Please create a separate BZ for them. However, I do not see them as critical. JON was upgrade and those errors are related to an old plugin that should not be there. From the log messages it's obvious that the plugin will not be added to the plugin container anyway.

The last warning from comment #10 is related to the AS7 plugin. But in this case, it is normal code execution. If you started the server with a profile that does not have the jaxr subsystem, then discovery for that subsystem fails. That warning is from the discovery process saying that the jaxr subsystem was not discovered (note "ResourceDiscoveryComponent" in the warning message); which is normal.

--- Additional comment from fbrychta on 2012-06-06 05:20:52 EDT ---

Separated BZs created:
https://bugzilla.redhat.com/show_bug.cgi?id=829227
https://bugzilla.redhat.com/show_bug.cgi?id=828938

Comment 1 Charles Crouch 2012-06-11 14:47:05 UTC
This BZ is around documenting the preferred upgrade steps for customers who were using the tech preview as7 plugin in jon300 or jon301.  Due to the tech preview nature of the older as7 plugin there is no upgrade path to the GA version. The new GA plugin can run alongside the older plugin but will result in two sets of EAP instances in inventory (one for each plugin). The following steps outline the easiest path to get customers who've used the tech preview plugin running with the GA version:

"
1- jon3.0.1.GA + jon-plugin-pack-tech-preview-3.0.1.GA installed and running, eap6 running and imported to jon inventory, the rhq agent is running on the background
2- disable the jBossAS7 plugin
3- detete the jBossAS7 plugin  (the agent was stopped after this step so i started it manually )
4- purge the jBossAS7 plugin
5- upgrade to jon3.1.0.RC2 + jon-plugin-pack-eap-3.1.0.CR2
6- import eap6
"

Comment 2 Deon Ballard 2012-06-11 15:57:47 UTC
Changing compontent to docs and owner to me.

Comment 3 Deon Ballard 2012-06-28 17:37:17 UTC
I added a new, small section to the upgrade docs:
http://docs.redhat.com/docs/en-US/JBoss_Operations_Network/3.1/html/Installation_Guide/eap6-upgrade.html


Note You need to log in before you can comment on or make changes to this bug.