Bug 799532 - JMX plugin discovery causes Java processes to output stack traces
Summary: JMX plugin discovery causes Java processes to output stack traces
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: Plugin Container
Version: 4.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: JON 3.1.0
Assignee: Viet Nguyen
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks: jon310-sprint11, rhq44-sprint11
TreeView+ depends on / blocked
 
Reported: 2012-03-02 21:52 UTC by Elias Ross
Modified: 2013-09-03 15:14 UTC (History)
2 users (show)

Fixed In Version: 4.4
Clone Of:
Environment:
Last Closed: 2013-09-03 15:14:33 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 703557 0 unspecified CLOSED [jmx plugin] auto-discover JVMs not exposed via JMX Remoting 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 819116 0 unspecified CLOSED Attach API only works with RHQ_AGENT_JAVA_HOME 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 824509 0 high CLOSED (solaris only) agent only detects jmx-enabled processes own by agent user 2021-02-22 00:41:40 UTC

Internal Links: 703557 819116 824509

Description Elias Ross 2012-03-02 21:52:56 UTC
Description of problem:

RHQ agent running JMX plugin (4.3.0-SNAPSHOT) seems to be sending kill -QUIT to all system Java processes.

If Java is sending its own output to STDOUT, then the shell sends it to a file, then that file becomes corrupted.


Version-Release number of selected component (if applicable):

4.3.0


How reproducible:

Seemingly hard to reproduce. Doesn't seem to happen for all Java processes, oddly enough.

Seems to happen on Linux JDK 1.6.0_30 and _16.

Steps to Reproduce:
1. One way is to run the RHQ command line client as 'rhq' user.
2. Then run agent discovery through the agent command line as root user.

Additional info:

I'm guessing this happens through the attach API, although I am not clear what call does this.

This is from the OpenJDK

src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java

        // Find the socket file. If not found then we attempt to start the
        // attach mechanism in the target VM by sending it a QUIT signal.
        // Then we attempt to find the socket file again.
        path = findSocketFile(pid);
        if (path == null) {
            File f = createAttachFile(pid);
            try {
                // On LinuxThreads each thread is a process and we don't have the
                // pid of the VMThread which has SIGQUIT unblocked. To workaround
                // this we get the pid of the "manager thread" that is created
                // by the first call to pthread_create. This is parent of all
                // threads (except the initial thread).
                if (isLinuxThreads) {
                    int mpid;
                    try {
                        mpid = getLinuxThreadsManager(pid);
                    } catch (IOException x) {
                        throw new AttachNotSupportedException(x.getMessage());
                    }
                    assert(mpid >= 1);
                    sendQuitToChildrenOf(mpid);
                } else {
                    sendQuitTo(pid);
                }


At the very least, there should be a way to disable this discovery mechanism or not attempt to attach to processes not explicitly requested.

Comment 1 Mike Foley 2012-03-05 16:42:44 UTC
per triage (crouch, loleary, foley).  JON 3.1 timeframe.  assign to ips.

Comment 2 Ian Springer 2012-03-06 18:33:04 UTC
Fixed in master:

http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=1c1e9b9

It turns out the Java attach API can only discover java processes running as the same user. When running as root, it will send a SIGQUIT as part of an attempt to attach to a java process running as another user, but it ultimately fails to attach anyway. So I've updated the JMX Server discovery code so it no longer even tries to attach to java processes running as users other than the user the Agent is running as.

Comment 3 Viet Nguyen 2012-04-30 15:47:55 UTC
Ran agent as root and was able to add non-root jmx processes. will verify with brew build.
JON  3.0.1.GA
agent 4.2.0.JON.3.0.1.GA

Comment 4 Viet Nguyen 2012-05-04 22:11:18 UTC
verified in 3.1.0.GA
-agent can discover any jmx enabled processes owned by others
-agent only discover same owner processes with -Dorg.rhq.resourceKey

see also https://bugzilla.redhat.com/show_bug.cgi?id=819116

Comment 5 Heiko W. Rupp 2013-09-03 15:14:33 UTC
Bulk closing of old issues in VERIFIED state.


Note You need to log in before you can comment on or make changes to this bug.