Created attachment 395625 [details]
Attached: Full agent.log

Started to happen after I exposed JMX by editing startup.sh:

JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.port=12123"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"
export JAVA_OPTS

When shutting down, agent throws InterruptedIOException:

---------------------------------------------------------
Failed to execute prompt command [discovery -f]. Cause: java.lang.NullPointerException:null
> ^CShutting down...
log4j:ERROR Failed to flush writer,
java.io.InterruptedIOException
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:336)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(StreamEncoder.java:404)
        at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(StreamEncoder.java:408)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:152)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:213)
        at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:57)
        at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:315)
        at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:236)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.info(Category.java:663)
        at mazz.i18n.log4j.Log4jLogMsg.info(Log4jLogMsg.java:638)
        at mazz.i18n.log4j.Log4jLogger.info(Log4jLogger.java:270)
        at org.rhq.enterprise.agent.PrimaryServerSwitchoverThread.run(PrimaryServerSwitchoverThread.java:132)
The agent will wait for [0] threads to die
Shutdown complete - agent will now exit.
[ozizka@dev13 agent]$
---------------------------------------------------------

Note that the JMX works (jconsole shows the metrics).

Comment #3 has been documented at https://bugzilla.redhat.com/show_bug.cgi?id=535575

Mazz, is this same as referenced bug in comment #5?

(In reply to comment #6)
> Mazz, is this same as referenced bug in comment #5?

no, this NPE seems to come from the discovery -f command, not during shutdown.

This seems related to https://bugzilla.redhat.com/show_bug.cgi?id=564514

Similarly, this needs to be reproduced to see if it is still a problem.

The logs posted in comment #2 are fairly clean and I cannot find any NPE. The big exception posted in comment #3 after sending the interrupt key combo should have been resolved by the BZ posted in comment #5.

So, the only thing to still research is the NPE reported before the shutdown. That NPE could have been caused by any plugin, not just Tomcat, since discovery -f executes the discovery process for all the plugins installed. 

I see no attempts to isolate the error (given that JON server, agent and Tomcat were on the same machine). Also, important information such as: JRE version and side effects after the NPE are missing from the description. 

I will attempt to replicate the environment and retest this issue with just Tomcat server and JON agent on the same machine.

The JDK was Sun JDK, likely 1.6.latest - since dev13 is a QA machine whose JDKs are kept up-to-date.

I could not replicate the reported problem. With or without the JMX settings there are no NPE reported in the agent logs or console. 

Please retest.