Bug 799532

Summary: JMX plugin discovery causes Java processes to output stack traces
Product: [Other] RHQ Project Reporter: Elias Ross <genman>
Component: Plugin ContainerAssignee: Viet Nguyen <vnguyen>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: high    
Version: 4.3CC: hrupp, vnguyen
Target Milestone: ---   
Target Release: JON 3.1.0   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=703557
https://bugzilla.redhat.com/show_bug.cgi?id=819116
https://bugzilla.redhat.com/show_bug.cgi?id=824509
Whiteboard:
Fixed In Version: 4.4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-03 11:14:33 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 782579    

Description Elias Ross 2012-03-02 16:52:56 EST
Description of problem:

RHQ agent running JMX plugin (4.3.0-SNAPSHOT) seems to be sending kill -QUIT to all system Java processes.

If Java is sending its own output to STDOUT, then the shell sends it to a file, then that file becomes corrupted.


Version-Release number of selected component (if applicable):

4.3.0


How reproducible:

Seemingly hard to reproduce. Doesn't seem to happen for all Java processes, oddly enough.

Seems to happen on Linux JDK 1.6.0_30 and _16.

Steps to Reproduce:
1. One way is to run the RHQ command line client as 'rhq' user.
2. Then run agent discovery through the agent command line as root user.

Additional info:

I'm guessing this happens through the attach API, although I am not clear what call does this.

This is from the OpenJDK

src/solaris/classes/sun/tools/attach/LinuxVirtualMachine.java

        // Find the socket file. If not found then we attempt to start the
        // attach mechanism in the target VM by sending it a QUIT signal.
        // Then we attempt to find the socket file again.
        path = findSocketFile(pid);
        if (path == null) {
            File f = createAttachFile(pid);
            try {
                // On LinuxThreads each thread is a process and we don't have the
                // pid of the VMThread which has SIGQUIT unblocked. To workaround
                // this we get the pid of the "manager thread" that is created
                // by the first call to pthread_create. This is parent of all
                // threads (except the initial thread).
                if (isLinuxThreads) {
                    int mpid;
                    try {
                        mpid = getLinuxThreadsManager(pid);
                    } catch (IOException x) {
                        throw new AttachNotSupportedException(x.getMessage());
                    }
                    assert(mpid >= 1);
                    sendQuitToChildrenOf(mpid);
                } else {
                    sendQuitTo(pid);
                }


At the very least, there should be a way to disable this discovery mechanism or not attempt to attach to processes not explicitly requested.
Comment 1 Mike Foley 2012-03-05 11:42:44 EST
per triage (crouch, loleary, foley).  JON 3.1 timeframe.  assign to ips.
Comment 2 Ian Springer 2012-03-06 13:33:04 EST
Fixed in master:

http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=1c1e9b9

It turns out the Java attach API can only discover java processes running as the same user. When running as root, it will send a SIGQUIT as part of an attempt to attach to a java process running as another user, but it ultimately fails to attach anyway. So I've updated the JMX Server discovery code so it no longer even tries to attach to java processes running as users other than the user the Agent is running as.
Comment 3 Viet Nguyen 2012-04-30 11:47:55 EDT
Ran agent as root and was able to add non-root jmx processes. will verify with brew build.
JON  3.0.1.GA
agent 4.2.0.JON.3.0.1.GA
Comment 4 Viet Nguyen 2012-05-04 18:11:18 EDT
verified in 3.1.0.GA
-agent can discover any jmx enabled processes owned by others
-agent only discover same owner processes with -Dorg.rhq.resourceKey

see also https://bugzilla.redhat.com/show_bug.cgi?id=819116
Comment 5 Heiko W. Rupp 2013-09-03 11:14:33 EDT
Bulk closing of old issues in VERIFIED state.