Bug 747925

Summary: derived plugin types not updated when parent plugin's type is changed
Product: [Other] RHQ Project Reporter: Heiko W. Rupp <hrupp>
Component: Core ServerAssignee: John Mazzitelli <mazz>
Status: CLOSED CURRENTRELEASE QA Contact: Filip Brychta <fbrychta>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.0.0CC: fbrychta, hrupp, mazz
Target Milestone: ---   
Target Release: JON 3.1.0   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=789420
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-03 11:12:47 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 639398, 745494, 782579    

Description Heiko W. Rupp 2011-10-21 08:16:03 EDT
deploy an updated version of a plugin (that has active resources on it) with one added metric.

Check the rhq_measurement_def table that the new metric is there.
Go to the rhq_measurement_sched table and see that the schedules are not there. So they also do not
show up in the UI for the respective metrics.

May be fallout from BZ 74922
Comment 1 Robert Buck 2011-10-21 11:01:09 EDT
Bob's Notes:

JMX plugin, about open files. Updating all meta data in db.

* metric definition, dbstatement in javaclass
* for each resource of a type that is matching, create a schedule, put in schedules table in db, to be sent down to agent
* schedule did not show up, but db definition is there,
* so def is there but creation of schedule is not there

The inventorying code that I just added works properly but the previous code in the metric schedule update isn't quite right. I will provide a patch for review asap.
Comment 2 Heiko W. Rupp 2011-10-21 11:44:04 EDT
Looks like unrelated to 747922.
The plugin I was about to update is the JMX one 

         <metric displayName="Open file descriptors"
                 description="Number of open file descriptors by this process. This metric is not available on all operating systems (e.g. not on Windows)"/>

     <service name="Operating System" discovery="MBeanResourceDiscoveryComponent" class="MBeanResourceComponent"
Comment 3 Heiko W. Rupp 2011-10-21 12:03:49 EDT
[17:33:51] <pilhuhn> I just went through the call stack and it has nothing to do with quartz scheduling
[17:34:18] <pilhuhn> The type that ends up here org.rhq.enterprise.server.measurement.MeasurementScheduleManagerBean#createSchedulesForExistingResources is wrong
[17:34:29] <pilhuhn> this is a new type and not the old one to be updated.
[17:34:49] <pilhuhn> meaning that the new type has no resources on it and thus no schedules can be updated

This issue seems to do with the fact that the JMX plugin runs inside other plugins like AS4-plugin and Tomcat plugin.

So the code seems to only update the "raw jmx plugin related types" (e.g. when directly importing a jmx server), but not the ones where it runs embedded. (Not confirmed, I did not find the "manual add" button
Comment 4 John Mazzitelli 2011-11-04 14:59:28 EDT
just for the record, this does add the schedule properly for non-extended types. I just added a metric to the RHQAgent type and it worked. So it probably is related to the fact that one type might be an extension of another.
Comment 5 John Mazzitelli 2011-11-04 15:22:46 EDT
this is definitely related to plugin extensions. Its easy to see:

1) import an RHQ Agent resource - it has child types that extend the JMX plugin
2) add a dummy metric to the JMX plugin's Operation System type (<metric property="dummy1" />) and deploy the new jmx plugin
3) Notice the JMX type has the new metric, but the one under the RHQ Agent does not.

When debugging, this is where the call to:

   scheduleMgr.createSchedulesForExistingResources(existingType, newDefinition);

is made:

MeasurementMetadataManagerBean.updateMetadata(ResourceType, ResourceType) lines 67 and 111 (there are two calls here depending on if the type had metrics already or not)

We need to somehow get the server to make multiple calls here - one for each derived type. So this already makes the call for the JMX type but it does not do anything for the derived types (the one that belongs to the RHQ Agent type).

In fact, this is worse that what the original description implies. Because not only does it not add the schedule to the existing resource it ALSO doesn't add the definition to that type either.

For example, the RHQ Agent has a child type RHQ Agent JVM which has the child type Operating System (this is merely a copy of the JMX plugin's types). Taht RHQ Agent child type does NOT even have the new metric definition. So its not just existing resources that don't have the new metric, but existing types that derived from the original type that was changed don't get the definitions.
Comment 6 John Mazzitelli 2011-11-04 15:24:50 EDT
we need to make sure this kind of bug doesn't exist in other places where we upgrade metadata. For example, what happens if I add a new plugin configuration property to a JMX resource type? Will the derived types (from, say the RHQ Agent type) also get that new configuration property? I suspect that that might also be broken. Need to test that.
Comment 7 John Mazzitelli 2011-11-04 15:34:18 EDT
we also need to make sure this isn't broken on REMOVAL of metadata. For example, if we remove a metric (or configuration property or anything else), do the copies of that type also get updated?
Comment 8 John Mazzitelli 2011-11-04 15:35:36 EDT
and of course we ALSO need to make sure this isn't broken on UPDATE of existing metadata. For example, if we modify a metric (or configuration property or anything else), do
the copies of that type also get updated?
Comment 9 John Mazzitelli 2011-11-04 15:56:14 EDT
OK, confirmed this is really bad.

In the JMX plugin's "Operating System" type, I added a plugin configuration property, I changed a metric description, I removed a metric and I added a metric. The JMX type itself (the parent type) was updated fine. The extended types (using the plugin "embedded" extension model - see http://rhq-project.org/display/RHQ/AMPS-Plugin+Extensions ) were not modified at all.

Looks like the metadata update code does not take into account the plugin extensions feature. If a type is changed, we need to look at all plugins and see if any types in any plugins extend the modified plugin and change those types.

This has the feel of a major refactoring effort to fix this. I can't think of any code that we have that looks at OTHER plugins when one plugin is updated. When a plugin is updated, AFAIK, we only check and update that plugin's type metadata. Now, we have to look at the end plugin dependency graph, determine what types, if any, extend the changed plugin and fix all their types if need be.
Comment 10 John Mazzitelli 2011-11-04 16:19:44 EDT
while I didn't test this, this also will probably mean any changes to operations also are not propogated. add/remove/modify operations won't get down to child types.

ANY metadata (operations, metrics, plugin/resource config, etc) won't be propogated. we simply do not look at extended types.
Comment 11 John Mazzitelli 2011-11-04 17:14:21 EDT
this seems to affect on the types in plugins


all extending one type - JMX plugin's JMX Server type.
Comment 12 John Mazzitelli 2011-11-04 19:53:34 EDT
bug #741331 changed some metrics from summary to detail. We need to revert that back because when this bug IS fixed, we'll want the solution to upgrade its metadata. So we will leave it the way it was last GA release.

We must remember to put the jmx plugin's rhq-plugin.xml back the way we really want it after this bug is fixed. Just revert back to the previous git commit. I'll provide the git hashes in another comment once I revert the jmx plugin's rhq-plugin.xml and commit it.
Comment 13 John Mazzitelli 2011-11-04 19:58:40 EDT
git master: cb8ad66fa52b6078d39a3beebaf92ee2cfc2788a

when this bug is fixed, we want to revert this file back to the way it was in git commit 6e953998dd354da8ae726bf7f9bcd8772d5c7842
Comment 14 John Mazzitelli 2011-11-04 20:00:40 EDT
git commit to release_3.x branch: 570a73c42ab563aaa2be114038b0d79601d4032e
Comment 15 Charles Crouch 2011-11-09 10:32:34 EST
Lowering priority as this is not a regression and has a high chance of 
introducing issue if implemented at this stage of the release. Should be 
addressed post release
Comment 16 Charles Crouch 2012-02-07 10:18:09 EST
Upping priority of this for JON3.1.0. Its an important fix that needs to be addressed
Comment 17 Charles Crouch 2012-02-15 08:05:01 EST
Mazz, I'd like to start the investigation of this one. Probably makes sense for this to be in its own feature branch, since I'm assuming its going to be a fairly hairy change
Comment 18 John Mazzitelli 2012-02-17 15:29:48 EST
work on this is being done in the "bug/747925" branch:

Comment 19 John Mazzitelli 2012-02-24 19:55:02 EST
i think starting at git commit 4dcc098db6a1dc828122b74afefc3f25fa3c7dfb in bug/747925 branch, this is working. I still want to do more testing and possibly write more unit tests to call this one done, though.
Comment 20 John Mazzitelli 2012-02-28 16:30:14 EST
branch bug/747925 has everything in. tests are passing.
Comment 21 John Mazzitelli 2012-03-06 11:07:20 EST
(In reply to comment #13)
> when this bug is fixed, we want to revert this file back to the way it was in
> git commit 6e953998dd354da8ae726bf7f9bcd8772d5c7842

In the bug/747925 branch, I did this; see commit 75d3d8a13f1cffdce3c18218e0e12d26533094f4
Comment 22 John Mazzitelli 2012-03-09 12:03:59 EST
i merged bug/747925 branch into master - git commit c5199bc8dd55c20a799eac2bf2c7531495c7e85d
Comment 23 Filip Brychta 2012-06-06 08:14:07 EDT
Verified on 3.1.0.CR3. I updated Generic JMX plugin and checked relevant changes on rhq-agent resource.
Comment 24 Heiko W. Rupp 2013-09-03 11:12:47 EDT
Bulk closing of old issues in VERIFIED state.