Bug 534180 - (RHQ-1001) embedded agent cannot access EJB services via JMX
embedded agent cannot access EJB services via JMX
Status: CLOSED WONTFIX
Product: RHQ Project
Classification: Other
Component: Agent (Show other bugs)
1.1
All All
low Severity medium (vote)
: ---
: ---
Assigned To: RHQ Project Maintainer
Mike Foley
http://jira.rhq-project.org/browse/RH...
: SubBug
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-10-20 21:11 EDT by John Mazzitelli
Modified: 2013-02-27 11:25 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-27 11:25:24 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description John Mazzitelli 2008-10-20 21:11:00 EDT
Enable the embedded agent and notice that the embedded agent fails to get any metric data or execute any operations on any EJB SLSB services.  If I try to execute the operation to get method stats, the operation fails and this is the stack:

java.lang.Exception: Failed to retrieve EJB3 invocation stats - perhaps JBoss EJB3 impl version is less than RC9 Patch 1.
	at org.rhq.plugins.jbossas.EJB3BeanComponent.getInvocationStatistics(EJB3BeanComponent.java:205)
	at org.rhq.plugins.jbossas.EJB3BeanComponent.invokeOperation(EJB3BeanComponent.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:450)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:619)
Caused by: org.mc4j.ems.connection.EmsException: Could not load attribute value, target bean threw exception org.jboss.ejb3.statistics.InvocationStatistics
	at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:212)
	at org.rhq.plugins.jbossas.EJB3BeanComponent.getInvocationStatistics(EJB3BeanComponent.java:199)
	... 11 more
Caused by: java.lang.ClassNotFoundException: org.jboss.ejb3.statistics.InvocationStatistics
	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
	at org.jboss.mx.server.InvocationContext.loadClass(InvocationContext.java:317)
	at org.jboss.mx.server.InvocationContext.getAttributeTypeClass(InvocationContext.java:169)
	at org.jboss.mx.server.Invocation.getAttributeTypeClass(Invocation.java:108)
	at org.jboss.mx.interceptor.ModelMBeanAttributeInterceptor.invoke(ModelMBeanAttributeInterceptor.java:70)
	at org.jboss.mx.interceptor.PersistenceInterceptor.invoke(PersistenceInterceptor.java:76)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
	at org.jboss.mx.server.AbstractMBeanInvoker.getAttribute(AbstractMBeanInvoker.java:362)
	at org.jboss.mx.server.MBeanServerImpl.getAttribute(MBeanServerImpl.java:556)
	at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:199)
	... 12 more
Comment 1 John Mazzitelli 2008-10-20 21:13:32 EDT
This JIRA is the reason why we say in the docs:

"Running an embedded Agent in production is not recommended and, therefore, it is not currently supported. There are some known issues with managing certain resources with the embedded agent."
Comment 2 John Mazzitelli 2008-10-21 01:13:25 EDT
Running debugging, I can see the EJB3 class:

org/jboss/ejb3/statistics/InvocationStatistics.class

found in the agent's data/tmp/

I executed this to find that path:

	 (java.net.URL) jar:file:/C:/mazz/source/rhq/trunk/modules/enterprise/agent/target/rhq-agent-1.2.0-SNAPSHOT/data/tmp/jbossall-client62975.jar!/org/jboss/ejb3/statistics/InvocationStatistics.class

Thread.currentThread().getContextClassLoader().getResource("org/jboss/ejb3/statistics/InvocationStatistics.class")

(called from within EJB3BeanComponent.getInvocationStatistics()).

When I look at the embedded agent's data/tmp directory, all I have are directories for each plugin, I do not see any files like "jbossall-client62975.jar" in there.  Wondering why the plugin container was unable to write the tmp files there for the plugins.
Comment 3 John Mazzitelli 2008-10-21 01:24:13 EDT
when in standalone:

        Thread.currentThread().setContextClassLoader(getEmsBean().getClass().getClassLoader());

sets the classloader to an EMS child-first classloader (i.e. getEmsBean().getClass().getClassLoader() returns an EMS child-first CL).

but when in embedded agent, that returns a bland URLClassLoader.  Unsure what that happens.
Comment 4 John Mazzitelli 2008-10-21 02:32:08 EDT
Here's the results of my investigation, look closely and you can see why this problem is occurring.

===================
EMBEDDED AGENT:

Put a break point at JBossASServerComponent, line 1129.

This was logged:

02:16:58,750 INFO  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://127.0.0.1:2099] with install path [C:\mazz\source\rhq\trunk\dev-container\jbossas]...

The EMS connection settings are:

EmsConnection [Internal]
	Initial Context = com.sun.jndi.rmi.registry.RegistryContextFactory
	JNDI Name = null
	Server URL = jnp://127.0.0.1:2099
	Principle = 
	Credentials = 
	 Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss
	 Advanced Property [jnp.timeout] = 30000
	 Advanced Property [jnp.disableDiscovery] = true
	 Advanced Property [jnp.sotimeout] = 15000
	 Control Property [mc4j.ems.CopyJarsToTemp] = true
	 Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\dev-container\jbossas\server\default\data\embedded-agent\tmp

Other variables are:

jbossHomeDir="C:\\mazz\\source\\rhq\\trunk\\dev-container\\jbossas" (id=9628)	
connectionTypeDescriptorClass="org.mc4j.ems.connection.support.metadata.InternalVMTypeDescriptor" (id=9631)	

After I step over line 1129, no files get created in that jar temp dir, but step one more line and it does get connected successfully.

==================
STANDALONE AGENT:

Same breakpoint on JBossASServerComponent, line 1129.

This was logged:

2008-10-21 02:20:51,000 INFO  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas.JBossASServerComponent)- Loading JBoss connection [jnp://127.0.0.1:2099] with install path [C:\mazz\source\rhq\trunk\dev-container\jbossas]...

The EMS connection settings are:

EmsConnection [JBoss]
	Initial Context = org.jnp.interfaces.NamingContextFactory
	JNDI Name = jmx/rmi/RMIAdaptor
	Server URL = jnp://127.0.0.1:2099
	Principle = null
	Credentials = null
	 Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss
	 Advanced Property [jnp.timeout] = 30000
	 Advanced Property [jnp.disableDiscovery] = true
	 Advanced Property [java.naming.factory.url.pkgs] = org.jboss.naming:org.jnp.interfaces
	 Advanced Property [jnp.sotimeout] = 15000
	 Control Property [mc4j.ems.CopyJarsToTemp] = true
	 Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\modules\enterprise\agent\target\rhq-agent-1.2.0-SNAPSHOT\data\tmp

Other variables are:

jbossHomeDir	"C:\\mazz\\source\\rhq\\trunk\\dev-container\\jbossas" (id=6426)	
connectionTypeDescriptorClass	"org.mc4j.ems.connection.support.metadata.JBossConnectionTypeDescriptor" (id=6434)	

After I step over line 1129, I see these files get placed in tmp dir and one more step over 1130 and it connects successfully:

concurrent11826.jar
hibernate311835.jar
jboss-j2ee11830.jar
jboss-jmx11821.jar
jboss-jsr77-client11827.jar
jboss-management11831.jar
jboss-system11822.jar
jboss-transaction11828.jar
jboss11825.jar
jbossall-client11823.jar
jbosssx-client11833.jar
jbosssx11832.jar
jnp-client11829.jar
log4j11824.jar
xercesImpl11834.jar

Comment 5 John Mazzitelli 2008-10-21 02:45:13 EDT
So you can see what is happening by looking at what EMS uses to connect to the JBossAS server.

First, when running embedded agent, it is monitoring the RHQ Server's JBossAS server - both are running in the same VM.  The EMS connection that the embedded agent's JBossAS plugin uses is:

EmsConnection [Internal]
Initial Context = com.sun.jndi.rmi.registry.RegistryContextFactory

This does NOT require any JBoss client libraries to connect to the MBeanServer - it uses an internal client.

In the standalone agent case, the agent is running in one VM but the RHQ Server's JBossAS server is in another VM.  The EMS connection the standalone agent's JBossAS plugin uses in this:

EmsConnection [JBoss]
Initial Context = org.jnp.interfaces.NamingContextFactory

This DOES require the JBoss JNP client.

Here's the problem: even though we don't need the JBoss JNP client library to connect to the JBossAS in the embedded agent case, we STILL need some JBoss client libraries to access some remote EJB features. For example, we need the jar that contains the org.jboss.ejb3.statistics.InvocationStatistics class (see EJB3BeanComponent.getInvocationStatistics() where it needs that class, otherwise, "getEmsBean().getAttribute("InvokeStats").refresh();" throws a  ClassNotFoundException).

In the standalone agent case, we find it in the "jbossall-client###.jar" (where ### is just random number used to make the file unique when writing it to the tmp dir). But that jbossas-client.jar is not extracted by EMS when using the internal mechanism to connect to the JBossAS MBeanServer. It is only extracted by EMS when it uses the JBoss-JNP-specific mechanism to connect to JBossAS server.

In short, we need a way to either:

a) get EMS to get the JBoss client jars in the connection classloader even if it uses the internal connection mechanism, because the client jars are needed by other parts of the JBossAS plugin.

or

b) for the JBossAS plugin to always use the  Jboss connection mechanism and to never use the internal mechanism (even if the agent is running in the same VM as the JBossAS server).

Need more info from Greg to figure out the solution to this problem.
Comment 6 John Mazzitelli 2008-10-21 02:45:27 EDT
assigning to greg for more info.
Comment 7 John Mazzitelli 2008-11-20 13:40:42 EST
this may actually be fixable now that RHQ-1094 added the "additionalClassPathEntries" property in the jmx plugin.

Perhaps we can have the jboss-as plugin expose this plugin config itself and pass it along to the EMS connection. This could allow the user to tell the agent where it can find the EJB jars that are relevant for the server.
Comment 8 John Mazzitelli 2009-08-10 01:22:51 EDT
JBossASDiscoveryComponent.discoverJBossPcIsEmbeddedIn:

// Set the connection type (used by JMX plugin to connect to the MBean server).
pluginConfiguration.put(new PropertySimple(JMXDiscoveryComponent.CONNECTION_TYPE, InternalVMTypeDescriptor.class.getName()));

I think this is to support the embedded console that was supposed to ship in JBossAS 4. But this also affects the embedded agent. Because this assumes all client classes are in the plugin's classloader - which isn't the case when running in embedded agent.
Comment 9 John Mazzitelli 2009-08-10 03:48:30 EDT
rev1059 refactors some code in JBossASDiscoveryComponent so there is an isEmbeddedInServer method. I still can't get it to work fully.
Comment 10 John Mazzitelli 2009-08-10 03:53:05 EDT
when I uncomment the code in the latest rev1059 of the discovery component:

                if (isEmbeddedInServer(configDir)) {
                    return null; // abort - we are embedded, but we're inside the enterprise Server
                }

 I get this:

java.lang.ClassCastException: javax.management.ObjectName cannot be cast to javax.management.ObjectName

in the server component at :

org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection():

                this.connection.loadSynchronous(false); // this loads all the MBeans
Comment 11 Corey Welton 2009-08-26 15:00:07 EDT
Pushed to 1.4
Comment 12 Red Hat Bugzilla 2009-11-10 15:21:34 EST
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-1001
This bug is incorporated by RHQ-1094
Comment 13 wes hayutin 2010-02-16 11:52:05 EST
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug
Comment 14 wes hayutin 2010-02-16 11:58:05 EST
making sure we're not missing any bugs in rhq_triage
Comment 15 Corey Welton 2010-12-23 09:22:36 EST
mazz -- will this work with our new classloading strategy?
Comment 16 Costel C 2011-08-26 11:41:41 EDT
Hello, 

I am very interested on this topic.

I saw this issue was included in RHQ-1094 which is closed. 

Does that mean this issue is already fixed in the release 4.0.1 ?

Thanks,
Costel
Comment 17 John Mazzitelli 2011-08-26 11:55:48 EDT
(In reply to comment #16)
> I saw this issue was included in RHQ-1094 which is closed. 
> 
> Does that mean this issue is already fixed in the release 4.0.1 ?

bug #534283 (formerly known as RHQ-1094) adds a property that allows the JMX plugin to be given additional jar files to be placed in classpath. I was guessing that we could use this to fix this BZ. Nothing has been done in this area - feel free to try to get this to work. If you can, post a patch and we'll test it and include it if it fixes the problem.
Comment 18 Costel C 2011-08-30 09:16:47 EDT
I added the jbossall-client.jar in the embedded-agent/lib folder
[...rhq-server-4.0.1/jbossas/server/default/deploy/rhq-agent.sar/META-INF/embedded-agent/lib/]

It seems that the mentioned exception:

(org.rhq.plugins.jbossas.EJB3BeanComponent)- Failed to retrieve EJB3 invocation stats - perhaps JBoss EJB3 impl version is less than RC9 Patch 1.
org.mc4j.ems.connection.EmsException: Could not load attribute value, target bean threw exception org.jboss.ejb3.statistics.InvocationStatistics
[...]

doesn't appear anymore.
Do you think this could be a fix for this issue?
Comment 19 John Mazzitelli 2011-08-30 09:25:54 EDT
(In reply to comment #18)
> I added the jbossall-client.jar in the embedded-agent/lib folder
> [...rhq-server-4.0.1/jbossas/server/default/deploy/rhq-agent.sar/META-INF/embedded-agent/lib/]
> 
> It seems that the mentioned exception:
> 
> (org.rhq.plugins.jbossas.EJB3BeanComponent)- Failed to retrieve EJB3 invocation
> stats - perhaps JBoss EJB3 impl version is less than RC9 Patch 1.
> org.mc4j.ems.connection.EmsException: Could not load attribute value, target
> bean threw exception org.jboss.ejb3.statistics.InvocationStatistics
> [...]
> 
> doesn't appear anymore.
> Do you think this could be a fix for this issue?

The problem that that would introduce is the embedded agent will no longer be able to monitor any other JBoss AS instances that require a different version of the JBoss client classes (since you've now put the JBoss client classes directly into top classloader of the agent - this means they will load first before any plugin-classloaders will).

If you do not plan on having your embedded agent monitor other versions of JBossAS (other than the version which is the same as the RHQ Server - which IIRC is JBossAS 4.2), then I think this could be OK. Of course, this would require some testing to make sure nothing else breaks because of this (e.g. are there JMX classes in that jbossall-client? If so, this means you will probably get class loading exceptions if you try to monitor other JMX-based resources).
Comment 20 Costel C 2011-08-30 09:56:10 EDT
Actually I need the agent embedded into the monitored JBossAS and not into the RHQ Server, but I think the way of doing this is similar. 
So there is no need to monitor other JBossAS with different version.

What do think about this idea of embedding the agent into the monitored JBossAS ? This would be easier for the IT admins to manage.
Comment 21 John Mazzitelli 2011-08-30 10:17:45 EDT
(In reply to comment #20)
> Actually I need the agent embedded into the monitored JBossAS and not into the
> RHQ Server, but I think the way of doing this is similar. 
> So there is no need to monitor other JBossAS with different version.
> 
> What do think about this idea of embedding the agent into the monitored JBossAS
> ? This would be easier for the IT admins to manage.

The agent is designed to be embedded (more technically, it is the PLUGIN CONTAINER that is designed to be embedded) in other JVMs. This is actually how our admin console was implemented in the JBoss EAP 5 app servers - the admin console IS the embedded plugin container with a custom UI wrapped around it.

The embedded *agent* is a sar that is deployed inside the RHQ Server's JBossAS instance. I suppose you could take that sar and embed it in your own JBossAS instance (you would do this to avoid having to run a separate agent process on your box to monitor that JBossAS instance). I don't know what roadblocks you would hit doing this - the embedded agent sar wasn't specifically packages to be a generic sar for deployment in any old JBossAS server - it was specifically created for deployment in the RHQ Server. But, it wouldn't surprise me if it worked doing what you describe.

If you do this, write up a blog or something and share it with the RHQ community on the rhq-users mailing list - I'm sure some people would find this useful.
Comment 22 John Mazzitelli 2013-02-27 11:25:24 EST
embedded agent has been discontinued

Note You need to log in before you can comment on or make changes to this bug.