Bug 551056 - agent created multiple classloaders for a single resource when starting clean
Summary: agent created multiple classloaders for a single resource when starting clean
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: Plugin Container
Version: 1.4
Hardware: All
OS: All
high
high
Target Milestone: ---
: ---
Assignee: John Mazzitelli
QA Contact: Sunil Kondkar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-12-28 18:35 UTC by John Mazzitelli
Modified: 2010-08-12 16:44 UTC (History)
0 users

Fixed In Version: 2.4
Clone Of:
Environment:
Last Closed: 2010-08-12 16:44:37 UTC
Embargoed:


Attachments (Terms of Use)
patch to avoid using uuid as a unique resource key in classloader manager (9.94 KB, patch)
2009-12-28 20:12 UTC, John Mazzitelli
no flags Details | Diff

Description John Mazzitelli 2009-12-28 18:35:47 UTC
Description of problem:

Using classLoader="instance" in plugin descriptors cause problems when starting an agent clean when there are resources in inventory that need to be synced to the agent. In this case, multiple resource classloaders are created for a single resource and things go haywire - culminating in things like "ClassCastException" for what seems like correct casting.

How reproducible:

Commit to inventory a resource that needs an instance classlaoder. Get the agent fully synced and ensure it works and things are fine. Then, shutdown the agent and restart it using --cleanconfig. You may notice (unsure if it happens all the time, probably depends on the code) that you will get odd class errors like ClassCastException for what looks like a safe cast.

Analysis:

InventoryManager.mergeResourceFromDiscovery is getting called before the agent syncs with the server. In this case, there are no resource containers in the inventory manager for known, committed resources in inventory. Therefore, when that method calls "findMatchingChildResource" to see if the resource exists (via "match" method), nothing will be found and it will create a resource classloader for it, associating that CL with the discovered "uuid" which is NOT going to be the UUID of the resource in inventory. The discovery component therefore will load in some classes in that classloader.

Then after the sync happens, the resource container is associated with the "real" resource with the "real" UUID. The resource component getting called for the first time requires a new classloader because the UUID is different (ClassLoaderManager keys on UUID) and creates a new CL. The component tries to use that classloader but is operating on classes loaded by the first CL that was created and bad things happen.

I don't know how to fix this yet - I think the ClassLoaderManager needs to do some tricks to know that a previous classloader should be reused. We can't use UUID as the key - there has to be some other identifier that is the same for both newly discovered resources and resources that are in inventory. Some relationship with resourceKey and parent resourceKey perhaps??

Comment 1 John Mazzitelli 2009-12-28 20:12:40 UTC
Created attachment 380671 [details]
patch to avoid using uuid as a unique resource key in classloader manager

see attached cl.patch for a proposed patch that no longer uses UUID in the classloader manager - instead it uses a "Canonical" key object that uses resource key/type/plugin, parent key/type/plugin as the canonical resource ID. I ran a quick test and it at least fixes the problem I had in my particular use-case. I think this should be a good general solution, but more testing needs to be done.

Comment 2 John Mazzitelli 2009-12-28 21:25:07 UTC
preliminary testing shows that this is OK now with the patch. I'll be checking it in with some additional code to the inventory manager and agent plugin so it can report more details on classloaders. this can be moved along to qa... though QA will probably need to wait for me to checkin the byteman plugin and jdbc trace plugin before they can test it (I don't know how to replicate the problem without those two).

Comment 3 John Mazzitelli 2010-01-07 14:58:05 UTC
this has been checked into master branch

Comment 4 Corey Welton 2010-03-24 03:02:04 UTC
qa -> skondkar

Comment 5 John Mazzitelli 2010-03-24 13:02:58 UTC
In the reproducible steps, when I say, "Commit to inventory a resource that needs an instance classloader", this means any resource whose descriptor has the classLoader="instance" attribute, like this:

    <server name="some-resource-name" classLoader="instance" ...>

Today I think one of the few plugins that use instance classloaders is the JBossAS 5 plugin - so you need to inventory a JBossAS 5 server to test this. I see this in the jboss-as-5 plugin:

    <server name="JBossAS Server"
            classLoader="instance" ...

Comment 6 Sunil Kondkar 2010-04-09 10:31:43 UTC
Verified on maser build#165 (Revision: 1d862cf744ee88a5024414996b2adb9bffd52eaa)

Installed JBossAS 5 server on agent and after it's inventory and availability , shutdown the agent and restarted it using --cleanconfig. 
Repeated shutdown and restart with --cleanconfig few times.

Observed that the agent.log did not display ClassCastException.

Comment 7 Corey Welton 2010-08-12 16:44:37 UTC
Mass-closure of verified bugs against JON.


Note You need to log in before you can comment on or make changes to this bug.