This JIRA is the reason why we say in the docs:

"Running an embedded Agent in production is not recommended and, therefore, it is not currently supported. There are some known issues with managing certain resources with the embedded agent."

Running debugging, I can see the EJB3 class:

org/jboss/ejb3/statistics/InvocationStatistics.class

found in the agent's data/tmp/

I executed this to find that path:

	 (java.net.URL) jar:file:/C:/mazz/source/rhq/trunk/modules/enterprise/agent/target/rhq-agent-1.2.0-SNAPSHOT/data/tmp/jbossall-client62975.jar!/org/jboss/ejb3/statistics/InvocationStatistics.class

Thread.currentThread().getContextClassLoader().getResource("org/jboss/ejb3/statistics/InvocationStatistics.class")

(called from within EJB3BeanComponent.getInvocationStatistics()).

When I look at the embedded agent's data/tmp directory, all I have are directories for each plugin, I do not see any files like "jbossall-client62975.jar" in there.  Wondering why the plugin container was unable to write the tmp files there for the plugins.

when in standalone:

        Thread.currentThread().setContextClassLoader(getEmsBean().getClass().getClassLoader());

sets the classloader to an EMS child-first classloader (i.e. getEmsBean().getClass().getClassLoader() returns an EMS child-first CL).

but when in embedded agent, that returns a bland URLClassLoader.  Unsure what that happens.

Here's the results of my investigation, look closely and you can see why this problem is occurring.

===================
EMBEDDED AGENT:

Put a break point at JBossASServerComponent, line 1129.

This was logged:

02:16:58,750 INFO  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://127.0.0.1:2099] with install path [C:\mazz\source\rhq\trunk\dev-container\jbossas]...

The EMS connection settings are:

EmsConnection [Internal]
	Initial Context = com.sun.jndi.rmi.registry.RegistryContextFactory
	JNDI Name = null
	Server URL = jnp://127.0.0.1:2099
	Principle = 
	Credentials = 
	 Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss
	 Advanced Property [jnp.timeout] = 30000
	 Advanced Property [jnp.disableDiscovery] = true
	 Advanced Property [jnp.sotimeout] = 15000
	 Control Property [mc4j.ems.CopyJarsToTemp] = true
	 Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\dev-container\jbossas\server\default\data\embedded-agent\tmp

Other variables are:

jbossHomeDir="C:\\mazz\\source\\rhq\\trunk\\dev-container\\jbossas" (id=9628)	
connectionTypeDescriptorClass="org.mc4j.ems.connection.support.metadata.InternalVMTypeDescriptor" (id=9631)	

After I step over line 1129, no files get created in that jar temp dir, but step one more line and it does get connected successfully.

==================
STANDALONE AGENT:

Same breakpoint on JBossASServerComponent, line 1129.

This was logged:

2008-10-21 02:20:51,000 INFO  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://127.0.0.1:2099] with install path [C:\mazz\source\rhq\trunk\dev-container\jbossas]...

The EMS connection settings are:

EmsConnection [JBoss]
	Initial Context = org.jnp.interfaces.NamingContextFactory
	JNDI Name = jmx/rmi/RMIAdaptor
	Server URL = jnp://127.0.0.1:2099
	Principle = null
	Credentials = null
	 Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss
	 Advanced Property [jnp.timeout] = 30000
	 Advanced Property [jnp.disableDiscovery] = true
	 Advanced Property [java.naming.factory.url.pkgs] = org.jboss.naming:org.jnp.interfaces
	 Advanced Property [jnp.sotimeout] = 15000
	 Control Property [mc4j.ems.CopyJarsToTemp] = true
	 Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\modules\enterprise\agent\target\rhq-agent-1.2.0-SNAPSHOT\data\tmp

Other variables are:

jbossHomeDir	"C:\\mazz\\source\\rhq\\trunk\\dev-container\\jbossas" (id=6426)	
connectionTypeDescriptorClass	"org.mc4j.ems.connection.support.metadata.JBossConnectionTypeDescriptor" (id=6434)	

After I step over line 1129, I see these files get placed in tmp dir and one more step over 1130 and it connects successfully:

concurrent11826.jar
hibernate311835.jar
jboss-j2ee11830.jar
jboss-jmx11821.jar
jboss-jsr77-client11827.jar
jboss-management11831.jar
jboss-system11822.jar
jboss-transaction11828.jar
jboss11825.jar
jbossall-client11823.jar
jbosssx-client11833.jar
jbosssx11832.jar
jnp-client11829.jar
log4j11824.jar
xercesImpl11834.jar

So you can see what is happening by looking at what EMS uses to connect to the JBossAS server.

First, when running embedded agent, it is monitoring the RHQ Server's JBossAS server - both are running in the same VM.  The EMS connection that the embedded agent's JBossAS plugin uses is:

EmsConnection [Internal]
Initial Context = com.sun.jndi.rmi.registry.RegistryContextFactory

This does NOT require any JBoss client libraries to connect to the MBeanServer - it uses an internal client.

In the standalone agent case, the agent is running in one VM but the RHQ Server's JBossAS server is in another VM.  The EMS connection the standalone agent's JBossAS plugin uses in this:

EmsConnection [JBoss]
Initial Context = org.jnp.interfaces.NamingContextFactory

This DOES require the JBoss JNP client.

Here's the problem: even though we don't need the JBoss JNP client library to connect to the JBossAS in the embedded agent case, we STILL need some JBoss client libraries to access some remote EJB features. For example, we need the jar that contains the org.jboss.ejb3.statistics.InvocationStatistics class (see EJB3BeanComponent.getInvocationStatistics() where it needs that class, otherwise, "getEmsBean().getAttribute("InvokeStats").refresh();" throws a  ClassNotFoundException).

In the standalone agent case, we find it in the "jbossall-client###.jar" (where ### is just random number used to make the file unique when writing it to the tmp dir). But that jbossas-client.jar is not extracted by EMS when using the internal mechanism to connect to the JBossAS MBeanServer. It is only extracted by EMS when it uses the JBoss-JNP-specific mechanism to connect to JBossAS server.

In short, we need a way to either:

a) get EMS to get the JBoss client jars in the connection classloader even if it uses the internal connection mechanism, because the client jars are needed by other parts of the JBossAS plugin.

or

b) for the JBossAS plugin to always use the  Jboss connection mechanism and to never use the internal mechanism (even if the agent is running in the same VM as the JBossAS server).

Need more info from Greg to figure out the solution to this problem.

assigning to greg for more info.

this may actually be fixable now that RHQ-1094 added the "additionalClassPathEntries" property in the jmx plugin.

Perhaps we can have the jboss-as plugin expose this plugin config itself and pass it along to the EMS connection. This could allow the user to tell the agent where it can find the EJB jars that are relevant for the server.

JBossASDiscoveryComponent.discoverJBossPcIsEmbeddedIn:

// Set the connection type (used by JMX plugin to connect to the MBean server).
pluginConfiguration.put(new PropertySimple(JMXDiscoveryComponent.CONNECTION_TYPE, InternalVMTypeDescriptor.class.getName()));

I think this is to support the embedded console that was supposed to ship in JBossAS 4. But this also affects the embedded agent. Because this assumes all client classes are in the plugin's classloader - which isn't the case when running in embedded agent.

rev1059 refactors some code in JBossASDiscoveryComponent so there is an isEmbeddedInServer method. I still can't get it to work fully.

when I uncomment the code in the latest rev1059 of the discovery component:

                if (isEmbeddedInServer(configDir)) {
                    return null; // abort - we are embedded, but we're inside the enterprise Server
                }

 I get this:

java.lang.ClassCastException: javax.management.ObjectName cannot be cast to javax.management.ObjectName

in the server component at :

org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection():

                this.connection.loadSynchronous(false); // this loads all the MBeans

Pushed to 1.4

This bug was previously known as http://jira.rhq-project.org/browse/RHQ-1001
This bug is incorporated by RHQ-1094

Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug

making sure we're not missing any bugs in rhq_triage

mazz -- will this work with our new classloading strategy?

Hello, 

I am very interested on this topic.

I saw this issue was included in RHQ-1094 which is closed. 

Does that mean this issue is already fixed in the release 4.0.1 ?

Thanks,
Costel

(In reply to comment #16)
> I saw this issue was included in RHQ-1094 which is closed. 
> 
> Does that mean this issue is already fixed in the release 4.0.1 ?

bug #534283 (formerly known as RHQ-1094) adds a property that allows the JMX plugin to be given additional jar files to be placed in classpath. I was guessing that we could use this to fix this BZ. Nothing has been done in this area - feel free to try to get this to work. If you can, post a patch and we'll test it and include it if it fixes the problem.

I added the jbossall-client.jar in the embedded-agent/lib folder
[...rhq-server-4.0.1/jbossas/server/default/deploy/rhq-agent.sar/META-INF/embedded-agent/lib/]

It seems that the mentioned exception:

(org.rhq.plugins.jbossas.EJB3BeanComponent)- Failed to retrieve EJB3 invocation stats - perhaps JBoss EJB3 impl version is less than RC9 Patch 1.
org.mc4j.ems.connection.EmsException: Could not load attribute value, target bean threw exception org.jboss.ejb3.statistics.InvocationStatistics
[...]

doesn't appear anymore.
Do you think this could be a fix for this issue?

(In reply to comment #18)
> I added the jbossall-client.jar in the embedded-agent/lib folder
> [...rhq-server-4.0.1/jbossas/server/default/deploy/rhq-agent.sar/META-INF/embedded-agent/lib/]
> 
> It seems that the mentioned exception:
> 
> (org.rhq.plugins.jbossas.EJB3BeanComponent)- Failed to retrieve EJB3 invocation
> stats - perhaps JBoss EJB3 impl version is less than RC9 Patch 1.
> org.mc4j.ems.connection.EmsException: Could not load attribute value, target
> bean threw exception org.jboss.ejb3.statistics.InvocationStatistics
> [...]
> 
> doesn't appear anymore.
> Do you think this could be a fix for this issue?

The problem that that would introduce is the embedded agent will no longer be able to monitor any other JBoss AS instances that require a different version of the JBoss client classes (since you've now put the JBoss client classes directly into top classloader of the agent - this means they will load first before any plugin-classloaders will).

If you do not plan on having your embedded agent monitor other versions of JBossAS (other than the version which is the same as the RHQ Server - which IIRC is JBossAS 4.2), then I think this could be OK. Of course, this would require some testing to make sure nothing else breaks because of this (e.g. are there JMX classes in that jbossall-client? If so, this means you will probably get class loading exceptions if you try to monitor other JMX-based resources).

Actually I need the agent embedded into the monitored JBossAS and not into the RHQ Server, but I think the way of doing this is similar. 
So there is no need to monitor other JBossAS with different version.

What do think about this idea of embedding the agent into the monitored JBossAS ? This would be easier for the IT admins to manage.

(In reply to comment #20)
> Actually I need the agent embedded into the monitored JBossAS and not into the
> RHQ Server, but I think the way of doing this is similar. 
> So there is no need to monitor other JBossAS with different version.
> 
> What do think about this idea of embedding the agent into the monitored JBossAS
> ? This would be easier for the IT admins to manage.

The agent is designed to be embedded (more technically, it is the PLUGIN CONTAINER that is designed to be embedded) in other JVMs. This is actually how our admin console was implemented in the JBoss EAP 5 app servers - the admin console IS the embedded plugin container with a custom UI wrapped around it.

The embedded *agent* is a sar that is deployed inside the RHQ Server's JBossAS instance. I suppose you could take that sar and embed it in your own JBossAS instance (you would do this to avoid having to run a separate agent process on your box to monitor that JBossAS instance). I don't know what roadblocks you would hit doing this - the embedded agent sar wasn't specifically packages to be a generic sar for deployment in any old JBossAS server - it was specifically created for deployment in the RHQ Server. But, it wouldn't surprise me if it worked doing what you describe.

If you do this, write up a blog or something and share it with the RHQ community on the rhq-users mailing list - I'm sure some people would find this useful.

embedded agent has been discontinued