Bug 703557

Summary: [jmx plugin] auto-discover JVMs not exposed via JMX Remoting
Product: [Other] RHQ Project Reporter: David Webster <dawebster>
Component: PluginsAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2CC: asantos, ccrouch, genman, hrupp, jshaughn, mazz, rsoares
Target Milestone: ---Keywords: FutureFeature
Target Release: RHQ 4.3.0   
Hardware: All   
OS: All   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=799532
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-12 17:51:27 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 760116    
Attachments:
Description Flags
A LocalDiscovery JMX plugin
none
Add osx & default builds to pom.xml
asantos: review+
Replaces tools.jar with variable from top level pom.xml per other attachment asantos: review+

Description David Webster 2011-05-10 12:54:12 EDT
Created attachment 498102 [details]
A LocalDiscovery JMX plugin

Description of problem: the current Generic JMX agent plugin is not able to discover JVM processes not using a jmx remote configuration  


Version-Release number of selected component (if applicable): RHQ v 4.x, JON 2.4.x and JON 3.0 Beta  


How reproducible:  N/A


Actual results:  Inability of the agent to discover Java processes that do not have a jmx remote port open


Expected results:  Managing jmx remote ports in large, enterprise systems is particularly desirable.  JON/RHQ should be able to attach to the MBean server of any local JVM process running under the same account as the agent JVM, without the need to expose the MBean server in a jmx remoting port.


Additional info:  I have attached a plugin that in our environment, runs alongside the standard Generic JMX plug, that uses the Java 6 Attach API to search for any JVM running under the agent's account.  It provides the same, basic, JVM statistics the current Generic JMX plugin does.

We would have simply extended the existing Generic JMX plugin instead of writing a separate complete plugin, except that the EMSConnection ConnectionProvider and ResourceContext of the JMXServerComponent class are "private" instead of "protected".

This plugin also contains support for a standard, non-container JVM stop/start/restart/status that uses a common startup script.  It should be broken out into a separate plugin.  The code is thoroughly documented.

Ideally, the LocalJMXDiscoveryComponent and LocalJMXServerComponent capability should be rolled into the Standard Generic JMX plugin that ships with the product.  It could first check on any jmx-remoting configurations and use that, and then search for any JVMs not using that by using the Attach API (JDK6+).
Comment 1 David Webster 2011-05-10 12:55:21 EDT
Managing jmx ports in large enterprise environments is UNdesirable.... :)
Comment 3 David Webster 2011-06-03 17:13:26 EDT
Soon to come, we have extracted the Stand Alone JVM stop/start support from the JVM Discovery plugin and put it in a separate plugin.  We have also enhanced it somewhat making it more flexible as it should now discover any associated opeartional script with any Java process that contains the correct templated signature in the expected standard relative location.
Comment 4 Ian Springer 2012-01-23 14:31:48 EST
A fix for this has been checked into master - here are the commits:

http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=d06bde3
http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=4e25940
http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=c56e22e
http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=f83f9b9
http://git.fedorahosted.org/git?p=rhq/rhq.git;a=commit;h=4673ea4

The main commit with the meat of the fix is the last one (4673ea4).

The updated Javadoc for JMXDiscoveryComponent summarizes the changes to discovery:

 * This component will discover JVM processes that appear to be long-running (i.e. "servers"). Specifically, it will
 * discover java processes that:
 * <ul>
 *  <li>have enabled JMX Remoting (JSR-160) via com.sun.management.jmxremote* system properties on their command lines, or</li>
 *  <li>are Sun/Oracle-compatible java processes accessible via the com.sun.tools.attach API AND specify the
 *      org.rhq.resourceKey system property on their command lines (e.g. -Dorg.rhq.resourceKey=FOO); the attach API uses IPC
 *      under the covers, so for a process to be accessible, it either must be running as the same user as the RHQ Agent,
 *      or the Agent must be running as root<li>
 * </ul>
 * Some other java processes that do not meet these criteria can be manually added if they expose JMX remotely in
 * another supported form (WebLogic, WebSphere, etc.).
Comment 5 Ian Springer 2012-01-23 15:22:18 EST
I accidentally changed the line endings in rhq-agent.bat from CRLF to LF in commit 4673ea4. http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=459fa87 changes them back to CRLF.
Comment 6 Alan Santos 2012-01-24 13:22:03 EST
Created attachment 557282 [details]
Add osx & default builds to pom.xml
Comment 7 Alan Santos 2012-01-24 13:23:00 EST
Created attachment 557283 [details]
Replaces tools.jar with variable from top level pom.xml per other attachment
Comment 8 Alan Santos 2012-01-24 13:23:21 EST
This broke OSX builds (more). The OSX 1.6 JDK doesn't have tools.jar, instead all of the tools.jar classes are in classes.jar

I've attached two patches that fix the OSX build. No promises that it doesn't break builds for another OS, I don't have another machine to test with.

The first to /pom.xml adds two profiles 1) a default_build that's activeByDefault and 2) an osx_build whose activation is <os><family>mac dependent. It then sets a property toolsjar = tools.jar or classes.jar

The second patch is for /modules/plugins/jmx/pom.xml it replaces the use of ${java.home}... with ${toolsjar}
Comment 9 Ian Springer 2012-01-24 16:39:06 EST
Thanks, Alan. http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=8125902 adds support for building on the Mac OS X / Apple JDK, as well as for the new JVM Attach feature when running the Agent on OS X.
Comment 10 John Mazzitelli 2012-01-30 10:15:04 EST
side note: in order for Eclipse to successfully compile the RHQ project, we have to tell it where to find tools.jar, which is what this new JMX plugin feature requires. I document how to do this here: http://www.rhq-project.org/display/RHQ/Eclipse+Tips#EclipseTips-Building
Comment 11 Elias Ross 2012-02-06 13:33:07 EST
*** Bug 786255 has been marked as a duplicate of this bug. ***
Comment 12 Elias Ross 2012-03-01 16:46:40 EST
In my experience...The discovery seems like a good thing but if a Java process is discovered and attached by other means (such as a custom plugin) then you see a lot of noise in the logs.

It would be nice if:
* You can skip discovery of processes already managed by other plugins. Maybe create a 'process registry' that maps which processes are managed by a specific plugin. Currently there is some hard coding going on here.
* The messages are kind of confusing and noisy ("cannot be discovered"; even though the process already WAS discovered)

I'm not sure if this feature is entirely worthwhile since it is quite easy to create a plugin for a Java process anyway. Changing a startup script to indicate a resource name for a process might actually be just as easy as creating a plugin, possibly.
Comment 13 Ian Springer 2012-03-02 16:50:32 EST
The jmx server discovery uses a configurable list of exclusion strings. If a java process' command line contains one of these strings, it is ignored (not discovered). The default value is {"org.jboss.Main", "catalina.startup.Bootstrap"}. This can be overridden by setting the rhq.jmxplugin.process-filters System property when starting the Agent, e.g.:

RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Drhq.jmxplugin.process-filters=org.jboss.Main,catalina.startup.Bootstrap,com.xyz.FooMain"

Note, the Agent java process is also always excluded, since it's typically already discovered by the rhq-agent plugin. However, I will probably refactor that to use the same mechanism used for everything else.

I've gotten rid of the "cannot be discovered" log messages. I agree they are misleading in the case where some other plugin has discovered the java process.

Thanks for the feedback. 

I am working on doc for this feature now and will post a link to them here when they're ready for review.
Comment 14 Jay Shaughnessy 2013-02-26 17:30:18 EST
Mazz, any thoughts on this one?
Comment 15 Jay Shaughnessy 2013-09-12 17:51:27 EDT
I don't think any more work is planned for this.  Also, I think the recent work in Bug 988735 also applies.