Bug 833059

(In reply to comment #1)
> Steps to Reproduce:
> 1. Build latest APR and jni wrappers
> 2. Configure a Tomcat 7 instance to use the AprLifecycleListener (no need to
> enable SSL)
> 3. Enable TC remote JMX

Can you provide more details on your Tomcat configuration? Perhaps list what configuration files you modified (and what those modifications were) that changed a Tomcat 7 out-of-box that matches your configuration? You can also attach your modified Tomcat config files as examples as well. Anything so we can more accurately and repeatedly reproduce your configuration that shows the issue.

i see the problem when I installed Tomcat 7 on Windows (windows comes with the native libs already pre-built).

Here's my server.xml snippets that are relevent:

  # BZ change - SSLEngine was changed from on to "off"
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
...
    # BZ change - uncommented this thread pool executor for use in the 8081 connector
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
...
    # BZ change - the port 8080 was unchanged but the 8081 was added to use a shared executor
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

    <!-- A "Connector" using the shared thread pool-->
    <Connector executor="tomcatThreadPool"
               port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8444" />

    # BZ change - SSLEnabled="false" rather than "true" - no need to use SSL for this test
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="false"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />

I don't know if it matters that I used a thread pool executor. But these are lines that I tweeked in the server.xml before running Tomcat 7.

I see these MBeans:

Catalina:type=GlobalRequestProcessor,name="ajp-apr-8009"
Catalina:type=GlobalRequestProcessor,name="http-apr-8080"
...and others...

These are very non-typical MBean names. I can't recall ever seeing MBeans whose names have literal double-quotes in them. In any event, it is almost assuredly a problem in our parsing of the MBean names that causes the problem.

Yes, you are seeing the exact same thing I am seeing, ObjectName name params with embedded quotes in the name.  No idea why they did that?  It is pretty odd.  Your setup is very close to mine, and it exhibits the same behavior on Linux hosts with Linux compiled APR.

Wow, ok, this is really a bad problem now. Tomcat 7 appears to have changed the MBean name and this effects everything because now all of our ObjectNames need to have quotes in them - otherwise, things fail because we can't query the MBean. Things like availability checks will now fail (and they do, as I see on my box) because we are pinging MBeans with ObjectNames that don't have quotes in the name attribute.

I believe it affects only Tomcat Native (APR) configs.  If you are using NIO/BIO connectors, it works fine....

(In reply to comment #5)
> I believe it affects only Tomcat Native (APR) configs.  If you are using
> NIO/BIO connectors, it works fine....

David,

If I attach a patched Tomcat plugin, can you test it in your environment? (presumably you can test it in some test environment) I am not convinced I have all the different permutations on my box so I'm not sure if my local testing will be enough. I'd like for you to also try it to see if things work for you. If you can, try it with APR configs and non-APR configs (NIO/BIO connectors like you mentioned).

I'll try with Connectors that also use shared executors, since that was also something that newer Tomcats broke (but we had fixed that earlier - I need to make sure that isn't broke again).

Certainly.  Can try both.  I will try and get to it tonight....

I may not have a patch ready by tonight. The more I dig into this, the more and more code it looks like this is causing us to change.

Without APR configured (I comment out the APR listener in server.xml) I see these global request proc MBeans:

Catalina:type=GlobalRequestProcessor,name="ajp-bio-8009"
Catalina:type=GlobalRequestProcessor,name="http-bio-8443"
... and others...

Notice "-bio" (and from the APR enabled Tomcat) "-apr" in the object names now. This appears to be new in Tomcat 7.

Need to see what the new format is when you specify addresses in the Connector configuration. This is going to change the parsing routines and we have to make sure we don't break backwards compat. with earlier Tomcat versions.

When specifying "address" attribute on the Connectors in server.xml, MBean names are like:

Catalina:type=GlobalRequestProcessor,name="http-bio-192.168.1.10-8080"

When using APR connectors, the same kind of thing is used:

Catalina:type=GlobalRequestProcessor,name="ajp-apr-192.168.1.10-8009"
(this using AJP but the format of the ObjectName is the same)

When you have a "address" attribute, the connector MBean ObjectNames are different too:

Catalina:type=Connector,port=8009,address="192.168.1.10"

More quotes. Argh!

OK, right now I'm going to backpedal. After looking more closely, these object name changes are not trivial and is going to cause some careful patching in the Tomcat plugin code.

I wish the Tomcat team would understand that changing ObjectNames like this affect management tools that want to interact with their MBeans.

So, for right now, it is clear that the RHQ Tomcat plugin cannot fully support Tomcat 7. At least it looks like the Connectors are broken. I don't know what else is broken. But for this BZ, I'll look at trying to see if I can get Tomcat 7 connectors to work again without breaking the plugin from losing compatibility with Tomcat 5 and 6.

Even if the Tomcat developers were to fix it with the next upgrade, there would still be the previous 26 sub releases to deal.  Probably going to have to do some kind of version check and apply a TC7 specific parser.  A tedious change for sure.  Just let me know, and I can test it whenever....

Eventually I would like to see the TC plugin use the Java 6 attach API rather than require remote JMX.  Remote JMX gives large corporations like ours administrative overhead of managing securing securing 100's of open ports as we have an IA requirement that all Production open remote JMX ports be secured with unique credentials....

I'm going to close this issue as a duplicate to bug 707349 - I didn't realize that BZ existed. That other issue is a generic "get Tomcat 7 to work" BZ so we'll work in that one. I'll add notes to that BZ.

*** This bug has been marked as a duplicate of bug 707349 ***

Ok, I will follow your progress on that one.  We plan on rolling out Tomcat 7 to the Enterprise late 4th Qtr but most likely right after the New Year, so not a big rush from us on it.  We would be happy to test any new plugin when it's available, if you'd like.