Bob's Notes:

JMX plugin, about open files. Updating all meta data in db.

* metric definition, dbstatement in javaclass
* for each resource of a type that is matching, create a schedule, put in schedules table in db, to be sent down to agent
* schedule did not show up, but db definition is there,
* so def is there but creation of schedule is not there

The inventorying code that I just added works properly but the previous code in the metric schedule update isn't quite right. I will provide a patch for review asap.

Looks like unrelated to 747922.
The plugin I was about to update is the JMX one 

Adding
         <metric displayName="Open file descriptors"
                 defaultOn="false"
                 category="utilization"
                 property="OpenFileDescriptorCount"
                 description="Number of open file descriptors by this process. This metric is not available on all operating systems (e.g. not on Windows)"/>

to
     <service name="Operating System" discovery="MBeanResourceDiscoveryComponent" class="MBeanResourceComponent"

[17:33:51] <pilhuhn> I just went through the call stack and it has nothing to do with quartz scheduling
[17:34:18] <pilhuhn> The type that ends up here org.rhq.enterprise.server.measurement.MeasurementScheduleManagerBean#createSchedulesForExistingResources is wrong
[17:34:29] <pilhuhn> this is a new type and not the old one to be updated.
[17:34:49] <pilhuhn> meaning that the new type has no resources on it and thus no schedules can be updated

This issue seems to do with the fact that the JMX plugin runs inside other plugins like AS4-plugin and Tomcat plugin.

So the code seems to only update the "raw jmx plugin related types" (e.g. when directly importing a jmx server), but not the ones where it runs embedded. (Not confirmed, I did not find the "manual add" button

just for the record, this does add the schedule properly for non-extended types. I just added a metric to the RHQAgent type and it worked. So it probably is related to the fact that one type might be an extension of another.

this is definitely related to plugin extensions. Its easy to see:

1) import an RHQ Agent resource - it has child types that extend the JMX plugin
2) add a dummy metric to the JMX plugin's Operation System type (<metric property="dummy1" />) and deploy the new jmx plugin
3) Notice the JMX type has the new metric, but the one under the RHQ Agent does not.

When debugging, this is where the call to:

   scheduleMgr.createSchedulesForExistingResources(existingType, newDefinition);

is made:

MeasurementMetadataManagerBean.updateMetadata(ResourceType, ResourceType) lines 67 and 111 (there are two calls here depending on if the type had metrics already or not)

We need to somehow get the server to make multiple calls here - one for each derived type. So this already makes the call for the JMX type but it does not do anything for the derived types (the one that belongs to the RHQ Agent type).

In fact, this is worse that what the original description implies. Because not only does it not add the schedule to the existing resource it ALSO doesn't add the definition to that type either.

For example, the RHQ Agent has a child type RHQ Agent JVM which has the child type Operating System (this is merely a copy of the JMX plugin's types). Taht RHQ Agent child type does NOT even have the new metric definition. So its not just existing resources that don't have the new metric, but existing types that derived from the original type that was changed don't get the definitions.

we need to make sure this kind of bug doesn't exist in other places where we upgrade metadata. For example, what happens if I add a new plugin configuration property to a JMX resource type? Will the derived types (from, say the RHQ Agent type) also get that new configuration property? I suspect that that might also be broken. Need to test that.

we also need to make sure this isn't broken on REMOVAL of metadata. For example, if we remove a metric (or configuration property or anything else), do the copies of that type also get updated?

and of course we ALSO need to make sure this isn't broken on UPDATE of existing metadata. For example, if we modify a metric (or configuration property or anything else), do
the copies of that type also get updated?

OK, confirmed this is really bad.

In the JMX plugin's "Operating System" type, I added a plugin configuration property, I changed a metric description, I removed a metric and I added a metric. The JMX type itself (the parent type) was updated fine. The extended types (using the plugin "embedded" extension model - see http://rhq-project.org/display/RHQ/AMPS-Plugin+Extensions ) were not modified at all.

Looks like the metadata update code does not take into account the plugin extensions feature. If a type is changed, we need to look at all plugins and see if any types in any plugins extend the modified plugin and change those types.

This has the feel of a major refactoring effort to fix this. I can't think of any code that we have that looks at OTHER plugins when one plugin is updated. When a plugin is updated, AFAIK, we only check and update that plugin's type metadata. Now, we have to look at the end plugin dependency graph, determine what types, if any, extend the changed plugin and fix all their types if need be.

while I didn't test this, this also will probably mean any changes to operations also are not propogated. add/remove/modify operations won't get down to child types.

ANY metadata (operations, metrics, plugin/resource config, etc) won't be propogated. we simply do not look at extended types.

this seems to affect on the types in plugins

jboss-as
jboss-as-5
jboss-as-7
rhq-agent
tomcat

all extending one type - JMX plugin's JMX Server type.

bug #741331 changed some metrics from summary to detail. We need to revert that back because when this bug IS fixed, we'll want the solution to upgrade its metadata. So we will leave it the way it was last GA release.

We must remember to put the jmx plugin's rhq-plugin.xml back the way we really want it after this bug is fixed. Just revert back to the previous git commit. I'll provide the git hashes in another comment once I revert the jmx plugin's rhq-plugin.xml and commit it.

git master: cb8ad66fa52b6078d39a3beebaf92ee2cfc2788a

when this bug is fixed, we want to revert this file back to the way it was in git commit 6e953998dd354da8ae726bf7f9bcd8772d5c7842

git commit to release_3.x branch: 570a73c42ab563aaa2be114038b0d79601d4032e

Lowering priority as this is not a regression and has a high chance of 
introducing issue if implemented at this stage of the release. Should be 
addressed post release

Upping priority of this for JON3.1.0. Its an important fix that needs to be addressed

Mazz, I'd like to start the investigation of this one. Probably makes sense for this to be in its own feature branch, since I'm assuming its going to be a fairly hairy change

work on this is being done in the "bug/747925" branch:

http://git.fedorahosted.org/git/?p=rhq/rhq.git;h=refs/heads/bug/747925

i think starting at git commit 4dcc098db6a1dc828122b74afefc3f25fa3c7dfb in bug/747925 branch, this is working. I still want to do more testing and possibly write more unit tests to call this one done, though.

branch bug/747925 has everything in. tests are passing.

(In reply to comment #13)
> when this bug is fixed, we want to revert this file back to the way it was in
> git commit 6e953998dd354da8ae726bf7f9bcd8772d5c7842

In the bug/747925 branch, I did this; see commit 75d3d8a13f1cffdce3c18218e0e12d26533094f4

i merged bug/747925 branch into master - git commit c5199bc8dd55c20a799eac2bf2c7531495c7e85d

Verified on 3.1.0.CR3. I updated Generic JMX plugin and checked relevant changes on rhq-agent resource.

Bulk closing of old issues in VERIFIED state.