Created attachment 498102 [details] A LocalDiscovery JMX plugin Description of problem: the current Generic JMX agent plugin is not able to discover JVM processes not using a jmx remote configuration Version-Release number of selected component (if applicable): RHQ v 4.x, JON 2.4.x and JON 3.0 Beta How reproducible: N/A Actual results: Inability of the agent to discover Java processes that do not have a jmx remote port open Expected results: Managing jmx remote ports in large, enterprise systems is particularly desirable. JON/RHQ should be able to attach to the MBean server of any local JVM process running under the same account as the agent JVM, without the need to expose the MBean server in a jmx remoting port. Additional info: I have attached a plugin that in our environment, runs alongside the standard Generic JMX plug, that uses the Java 6 Attach API to search for any JVM running under the agent's account. It provides the same, basic, JVM statistics the current Generic JMX plugin does. We would have simply extended the existing Generic JMX plugin instead of writing a separate complete plugin, except that the EMSConnection ConnectionProvider and ResourceContext of the JMXServerComponent class are "private" instead of "protected". This plugin also contains support for a standard, non-container JVM stop/start/restart/status that uses a common startup script. It should be broken out into a separate plugin. The code is thoroughly documented. Ideally, the LocalJMXDiscoveryComponent and LocalJMXServerComponent capability should be rolled into the Standard Generic JMX plugin that ships with the product. It could first check on any jmx-remoting configurations and use that, and then search for any JVMs not using that by using the Attach API (JDK6+).
Managing jmx ports in large enterprise environments is UNdesirable.... :)
Soon to come, we have extracted the Stand Alone JVM stop/start support from the JVM Discovery plugin and put it in a separate plugin. We have also enhanced it somewhat making it more flexible as it should now discover any associated opeartional script with any Java process that contains the correct templated signature in the expected standard relative location.
A fix for this has been checked into master - here are the commits: http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=d06bde3 http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=4e25940 http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=c56e22e http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=f83f9b9 http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=4673ea4 The main commit with the meat of the fix is the last one (4673ea4). The updated Javadoc for JMXDiscoveryComponent summarizes the changes to discovery: * This component will discover JVM processes that appear to be long-running (i.e. "servers"). Specifically, it will * discover java processes that: * <ul> * <li>have enabled JMX Remoting (JSR-160) via com.sun.management.jmxremote* system properties on their command lines, or</li> * <li>are Sun/Oracle-compatible java processes accessible via the com.sun.tools.attach API AND specify the * org.rhq.resourceKey system property on their command lines (e.g. -Dorg.rhq.resourceKey=FOO); the attach API uses IPC * under the covers, so for a process to be accessible, it either must be running as the same user as the RHQ Agent, * or the Agent must be running as root<li> * </ul> * Some other java processes that do not meet these criteria can be manually added if they expose JMX remotely in * another supported form (WebLogic, WebSphere, etc.).
I accidentally changed the line endings in rhq-agent.bat from CRLF to LF in commit 4673ea4. http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=459fa87 changes them back to CRLF.
Created attachment 557282 [details] Add osx & default builds to pom.xml
Created attachment 557283 [details] Replaces tools.jar with variable from top level pom.xml per other attachment
This broke OSX builds (more). The OSX 1.6 JDK doesn't have tools.jar, instead all of the tools.jar classes are in classes.jar I've attached two patches that fix the OSX build. No promises that it doesn't break builds for another OS, I don't have another machine to test with. The first to /pom.xml adds two profiles 1) a default_build that's activeByDefault and 2) an osx_build whose activation is <os><family>mac dependent. It then sets a property toolsjar = tools.jar or classes.jar The second patch is for /modules/plugins/jmx/pom.xml it replaces the use of ${java.home}... with ${toolsjar}
Thanks, Alan. http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=8125902 adds support for building on the Mac OS X / Apple JDK, as well as for the new JVM Attach feature when running the Agent on OS X.
side note: in order for Eclipse to successfully compile the RHQ project, we have to tell it where to find tools.jar, which is what this new JMX plugin feature requires. I document how to do this here: http://www.rhq-project.org/display/RHQ/Eclipse+Tips#EclipseTips-Building
*** Bug 786255 has been marked as a duplicate of this bug. ***
In my experience...The discovery seems like a good thing but if a Java process is discovered and attached by other means (such as a custom plugin) then you see a lot of noise in the logs. It would be nice if: * You can skip discovery of processes already managed by other plugins. Maybe create a 'process registry' that maps which processes are managed by a specific plugin. Currently there is some hard coding going on here. * The messages are kind of confusing and noisy ("cannot be discovered"; even though the process already WAS discovered) I'm not sure if this feature is entirely worthwhile since it is quite easy to create a plugin for a Java process anyway. Changing a startup script to indicate a resource name for a process might actually be just as easy as creating a plugin, possibly.
The jmx server discovery uses a configurable list of exclusion strings. If a java process' command line contains one of these strings, it is ignored (not discovered). The default value is {"org.jboss.Main", "catalina.startup.Bootstrap"}. This can be overridden by setting the rhq.jmxplugin.process-filters System property when starting the Agent, e.g.: RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Drhq.jmxplugin.process-filters=org.jboss.Main,catalina.startup.Bootstrap,com.xyz.FooMain" Note, the Agent java process is also always excluded, since it's typically already discovered by the rhq-agent plugin. However, I will probably refactor that to use the same mechanism used for everything else. I've gotten rid of the "cannot be discovered" log messages. I agree they are misleading in the case where some other plugin has discovered the java process. Thanks for the feedback. I am working on doc for this feature now and will post a link to them here when they're ready for review.
Mazz, any thoughts on this one?
I don't think any more work is planned for this. Also, I think the recent work in Bug 988735 also applies.