Bug 833059 - Tomcat plugin unable to parse native APR configured connector mbeans for Tomcat 7 and latest release of APR and jni wrapper.
Tomcat plugin unable to parse native APR configured connector mbeans for Tomc...
Status: CLOSED DUPLICATE of bug 707349
Product: RHQ Project
Classification: Other
Component: Agent (Show other bugs)
4.4
Unspecified Unspecified
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: John Mazzitelli
Mike Foley
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-18 09:48 EDT by David Webster
Modified: 2012-06-19 10:13 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-19 09:38:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Agent log containing the error. (70.34 KB, application/octet-stream)
2012-06-18 09:48 EDT, David Webster
no flags Details

  None (edit)
Description David Webster 2012-06-18 09:48:57 EDT
Created attachment 592652 [details]
Agent log containing the error.

Description of problem: Problem with the Tomcat agent plugin's ability to parse the GlobalRequestProcess MBeans in Tomcat 7 using APR (Native) connections.  Specifically, the ability to extract the ports due to imbedded quotes in the ObjectName of the mbeans.  This leads to RHQ being unable to establish communication with the connectors and console always marking the connectors and all deployed web-modules as unavailable.


Version-Release number of selected component (if applicable):

RHQ (and delivered TC agent plugin) 4.4.0.  Tomcat 7.0.26.  APR 1.1.22 (I think)
RHEL 6 and CentOS 6.2

How reproducible:

On Demand


Steps to Reproduce:
1. Build latest APR and jni wrappers
2. Configure a Tomcat 7 instance to use the AprLifecycleListener (no need to enable SSL)
3. Enable TC remote JMX
4. Install and run an RHQ agent
5. Inventory the new platform and set up the needed Tomcat connections and enable the server to pull newly discovered components

Look for the following MBean parser errors (NumberFormatException):
2012-06-17 19:22:26,943 WARN  [ResourceDiscoveryComponent.invoker.daemon-3] (jboss.on.plugins.tomcat.TomcatConnectorDiscoveryComponent)- Failed to parse ObjectName for GlobalRequestProcessor: "http-apr-10001": java.lang.NumberFormatException: For input string: "10001""
2012-06-17 19:22:26,943 WARN  [ResourceDiscoveryComponent.invoker.daemon-3] (jboss.on.plugins.tomcat.TomcatConnectorDiscoveryComponent)- Failed to parse ObjectName for GlobalRequestProcessor: "ajp-apr-10003": java.lang.NumberFormatException: For input string: "10003""

followed by an inability to communicate:

2012-06-17 19:22:29,883 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot start component for Resource[id=10562, uuid=350abf2b-8006-451d-b7d8-d9429d97507f, type={Tomcat}Tomcat Connector, key=Catalina:port=10003,type=Connector, name=?-10003, parent=Tomcat (10001)] from synchronized merge due to invalid plugin config: Failed to start component for resource Resource[id=10562, uuid=350abf2b-8006-451d-b7d8-d9429d97507f, type={Tomcat}Tomcat Connector, key=Catalina:port=10003,type=Connector, name=?-10003, parent=Tomcat (10001)].
2012-06-17 19:22:29,888 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot start component for Resource[id=10561, uuid=ea708d64-bdf3-4733-86c7-49b4765eee2a, type={Tomcat}Tomcat Connector, key=Catalina:port=10001,type=Connector, name=?-10001, parent=Tomcat (10001)] from synchronized merge due to invalid plugin config: Failed to start component for resource Resource[id=10561, uuid=ea708d64-bdf3-4733-86c7-49b4765eee2a, type={Tomcat}Tomcat Connector, key=Catalina:port=10001,type=Connector, name=?-10001, parent=Tomcat (10001)].

Use JConsole to browse the mbeans and you will the odd naming they used....

  
Actual results:


Expected results:


Additional info:
Comment 1 John Mazzitelli 2012-06-18 13:14:28 EDT
(In reply to comment #1)
> Steps to Reproduce:
> 1. Build latest APR and jni wrappers
> 2. Configure a Tomcat 7 instance to use the AprLifecycleListener (no need to
> enable SSL)
> 3. Enable TC remote JMX

Can you provide more details on your Tomcat configuration? Perhaps list what configuration files you modified (and what those modifications were) that changed a Tomcat 7 out-of-box that matches your configuration? You can also attach your modified Tomcat config files as examples as well. Anything so we can more accurately and repeatedly reproduce your configuration that shows the issue.
Comment 2 John Mazzitelli 2012-06-18 14:59:49 EDT
i see the problem when I installed Tomcat 7 on Windows (windows comes with the native libs already pre-built).

Here's my server.xml snippets that are relevent:

  # BZ change - SSLEngine was changed from on to "off"
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
...
    # BZ change - uncommented this thread pool executor for use in the 8081 connector
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
...
    # BZ change - the port 8080 was unchanged but the 8081 was added to use a shared executor
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

    <!-- A "Connector" using the shared thread pool-->
    <Connector executor="tomcatThreadPool"
               port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8444" />

    # BZ change - SSLEnabled="false" rather than "true" - no need to use SSL for this test
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="false"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />

I don't know if it matters that I used a thread pool executor. But these are lines that I tweeked in the server.xml before running Tomcat 7.

I see these MBeans:

Catalina:type=GlobalRequestProcessor,name="ajp-apr-8009"
Catalina:type=GlobalRequestProcessor,name="http-apr-8080"
...and others...

These are very non-typical MBean names. I can't recall ever seeing MBeans whose names have literal double-quotes in them. In any event, it is almost assuredly a problem in our parsing of the MBean names that causes the problem.
Comment 3 David Webster 2012-06-18 15:21:49 EDT
Yes, you are seeing the exact same thing I am seeing, ObjectName name params with embedded quotes in the name.  No idea why they did that?  It is pretty odd.  Your setup is very close to mine, and it exhibits the same behavior on Linux hosts with Linux compiled APR.
Comment 4 John Mazzitelli 2012-06-18 16:00:21 EDT
Wow, ok, this is really a bad problem now. Tomcat 7 appears to have changed the MBean name and this effects everything because now all of our ObjectNames need to have quotes in them - otherwise, things fail because we can't query the MBean. Things like availability checks will now fail (and they do, as I see on my box) because we are pinging MBeans with ObjectNames that don't have quotes in the name attribute.
Comment 5 David Webster 2012-06-18 16:34:27 EDT
I believe it affects only Tomcat Native (APR) configs.  If you are using NIO/BIO connectors, it works fine....
Comment 6 John Mazzitelli 2012-06-18 16:38:20 EDT
(In reply to comment #5)
> I believe it affects only Tomcat Native (APR) configs.  If you are using
> NIO/BIO connectors, it works fine....

David,

If I attach a patched Tomcat plugin, can you test it in your environment? (presumably you can test it in some test environment) I am not convinced I have all the different permutations on my box so I'm not sure if my local testing will be enough. I'd like for you to also try it to see if things work for you. If you can, try it with APR configs and non-APR configs (NIO/BIO connectors like you mentioned).

I'll try with Connectors that also use shared executors, since that was also something that newer Tomcats broke (but we had fixed that earlier - I need to make sure that isn't broke again).
Comment 7 David Webster 2012-06-18 16:45:12 EDT
Certainly.  Can try both.  I will try and get to it tonight....
Comment 8 John Mazzitelli 2012-06-18 17:02:01 EDT
I may not have a patch ready by tonight. The more I dig into this, the more and more code it looks like this is causing us to change.
Comment 9 John Mazzitelli 2012-06-18 17:08:22 EDT
Without APR configured (I comment out the APR listener in server.xml) I see these global request proc MBeans:

Catalina:type=GlobalRequestProcessor,name="ajp-bio-8009"
Catalina:type=GlobalRequestProcessor,name="http-bio-8443"
... and others...

Notice "-bio" (and from the APR enabled Tomcat) "-apr" in the object names now. This appears to be new in Tomcat 7.

Need to see what the new format is when you specify addresses in the Connector configuration. This is going to change the parsing routines and we have to make sure we don't break backwards compat. with earlier Tomcat versions.
Comment 10 John Mazzitelli 2012-06-18 17:14:39 EDT
When specifying "address" attribute on the Connectors in server.xml, MBean names are like:

Catalina:type=GlobalRequestProcessor,name="http-bio-192.168.1.10-8080"

When using APR connectors, the same kind of thing is used:

Catalina:type=GlobalRequestProcessor,name="ajp-apr-192.168.1.10-8009"
(this using AJP but the format of the ObjectName is the same)
Comment 11 John Mazzitelli 2012-06-18 17:19:59 EDT
When you have a "address" attribute, the connector MBean ObjectNames are different too:

Catalina:type=Connector,port=8009,address="192.168.1.10"

More quotes. Argh!
Comment 12 John Mazzitelli 2012-06-18 17:33:46 EDT
OK, right now I'm going to backpedal. After looking more closely, these object name changes are not trivial and is going to cause some careful patching in the Tomcat plugin code.

I wish the Tomcat team would understand that changing ObjectNames like this affect management tools that want to interact with their MBeans.

So, for right now, it is clear that the RHQ Tomcat plugin cannot fully support Tomcat 7. At least it looks like the Connectors are broken. I don't know what else is broken. But for this BZ, I'll look at trying to see if I can get Tomcat 7 connectors to work again without breaking the plugin from losing compatibility with Tomcat 5 and 6.
Comment 13 David Webster 2012-06-18 20:47:21 EDT
Even if the Tomcat developers were to fix it with the next upgrade, there would still be the previous 26 sub releases to deal.  Probably going to have to do some kind of version check and apply a TC7 specific parser.  A tedious change for sure.  Just let me know, and I can test it whenever....

Eventually I would like to see the TC plugin use the Java 6 attach API rather than require remote JMX.  Remote JMX gives large corporations like ours administrative overhead of managing securing securing 100's of open ports as we have an IA requirement that all Production open remote JMX ports be secured with unique credentials....
Comment 14 John Mazzitelli 2012-06-19 09:38:04 EDT
I'm going to close this issue as a duplicate to bug 707349 - I didn't realize that BZ existed. That other issue is a generic "get Tomcat 7 to work" BZ so we'll work in that one. I'll add notes to that BZ.

*** This bug has been marked as a duplicate of bug 707349 ***
Comment 15 David Webster 2012-06-19 10:13:27 EDT
Ok, I will follow your progress on that one.  We plan on rolling out Tomcat 7 to the Enterprise late 4th Qtr but most likely right after the New Year, so not a big rush from us on it.  We would be happy to test any new plugin when it's available, if you'd like.

Note You need to log in before you can comment on or make changes to this bug.