Bug 956509

Doc Text added.

@David, can you please review the Doc Text content?

Hi Weinan,
There is a problem with both storeconfig-tc6 and tc7. Both use catalina-ha and transitive dependency, catalina-tribes. We exclude both viz JBPAPP-9557. I haven't yet had time to look the storeconfig source closely for a workaround.

@Mandar, the doc is fine for now but may need to be edited later.

From the pm meeting this is deferred from 2.0.1. I'll confirm in ews-pm.

This bug should be in known issues. It will not be fixed with EWS 2.0.1

It doesn't work properly.
Tested environment:
 - JON 3.1.2 server with EWS plugin hotfix + EWS 2.0.1 with storeconfig jars from comm10
 - RHEL5 and SunJDK 1.6

Tests steps:
start T6 - change virtual host property autoDeploy to false - restart T6
start T7 - change virtual host property autoDeploy to false - restart T7

server.xmls before and after changes attached.

Created attachment 790881 [details]
T6 server.xmls

Created attachment 790882 [details]
T7 server.xmls

T6 - server.xml is not well formed after storeconfig operation and missing some properties (address, etc.)
T7 - server.xml does not include Listener tags after storeconfig operation and also missing some tags 
     (<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase"/>)

for TC6 it looks like the server.xml is broken: only a part of it. Any thing in the agent log? Like an exception?

I tried T6 test again manually and it seems well formed but missing address attribute in connectors elements and some others attributes. Other missing attributes like appBase="webapps", unpackWARs="true", xmlValidation="false", xmlNamespaceAware="false" in Host element and connectionTimeout="20000" in HTTP connector elemnts should be defaults so they are not needed in server.xml
Missing address attribute for connectors is a problem.

I also tried T7 test manually and missing Listeners elements are still there.

Created attachment 790974 [details]
T6 server.xml manual testing

I'm not so sure the missing address attribute is that big of a problem.
From T6 documentation for the 'address' attribute:

   "For servers with more than one IP address, this attribute specifies which
    address will be used for listening on the specified port. By default, this
    port will be used on all IP addresses associated with the server."

    (http://tomcat.apache.org/tomcat-6.0-doc/config/http.html#Attributes)

I wonder if it was omitted on purpose since it can (probably) be considered unnecessary when the machine has only one IP address.

appBase="webapps" is definitely a default setting. Not sure about the rest. (Someone might need to dig in the code).

I don't know how the storeconfig is implemented, but I'd expect something along the lines of "save everything we can't get from default values to server.xml".

the missing address in the connector is a problem it needs to be fixed.

Findings: T6 not well formed server.xml is created after changing configuration of deployed application.
Steps to reproduce:
1. Deploy application
2. Change their configuration in JON UI, for example set reloadable attribute to NO
3. Check server.xml -> not well formed

There is also error in catalina.out after changing app configuration:
Aug 28, 2013 8:08:00 AM org.apache.catalina.storeconfig.StoreConfig store
SEVERE: java.lang.InstantiationException: org.apache.catalina.core.StandardEngine$AccessLogListener

i can reproduce with a simple jmx client.
(1) save a copy of the stock server.xml, 
(2) create a new virt host by invoking addChild on Catalina:type=Engine
(3) invoke storeConfig on Catalina:type=StoreConfig
(4) compare the original with the current

Missing Listeners is because that code is commented. I think there's some configuration problems as well.

StoreConfig tries to rewrite the server.xml, but the result is missing all comments, the file header, and element attributes.

i'm working on it.

I don't think you will be able to save the comments.
What element attributes are missing?

No. the comments aren't in the model. Just components and properties.

(1) The server/filters are not being written. This might be because the listeners are configured as 'TransientChild'in server-registry. This has my focus.

(2) The Server attributes port and shutdown are lost.
(3) maxThreads is lost in Connectors, but it does include attributes that are 
not in the base server.xml e.g., Connector:8080 - maxConnections, keepAliveTimeout; Connector:8009 - connectionTimeout, keepAliveTimout, maxConnections, packetSize, tomcatAuthentication. 
(4) storeconfig is missing a Realm for UserDatabaseRealm. storeconfig may not currently allow for a nested Realm.
(5) the connector for 8443 is lost. It's commented in the base server.xml.

storeconfig is an introspection based tool, which produces a bare bones server.xml out of the server's runtime state. To avoid producing a server.xml that would be more verbose than what a user would usually do, it uses introspected default values (other "default" values can be set from somewhere else without the user doing anything to its server.xml, but storeconfig can't see that and they will be saved), known listeners that will always be set, etc. But it can also never be foolproof, and will also never look like the original server.xml.

To examine possible issues and know if it's a bug / can't fix / normal, you need to post the original server.xml snippets (and storeconfiged server.xml snippet too if possible). And specify is this is with 6 or 7 too.

Created attachment 793762 [details]
TC& Out-of-the-box server.xml with coments removed

The comments have been removed.

Created attachment 793764 [details]
TC7 server.xml after storeConfig

A vhost is added: "testhost" and storeConfig() has been called

The result server.xml is broken as comment#23 alludes. 

The error is:
SEVERE: Cannot move orignal context output file at /var/lib/tomcat7/webapps/host-manager/META-INF/context.xml
Sep 04, 2013 1:15:54 PM org.apache.tomcat.util.digester.Digester fatalError
SEVERE: Parse Fatal Error at line 20 column 34: Attribute "connectionTimeout" was already specified for element "Connector".
org.xml.sax.SAXParseException; systemId: file:/usr/share/tomcat7/conf/server.xml; lineNumber: 20; columnNumber: 34; Attribute "connectionTimeout" was already specified for element "Connector".

The complaint is true for both Connector:port=8080 and 8009

I'm going to attach the full log.

Created attachment 793768 [details]
TC7 log showing errors created by StoreConfig

TC7 catalina.out showing errors from connector elements.

I found that new T8 RC1 on tomcat.apache.org includes catalina-storeconfig.jar.
May be some code from T8 will be helpful.

Yes, we integrated storeconfig in Tomcat trunk now.

Since the default server.xml works fine for me, maybe the tag used has a problem ? I did diffs and all, but maybe I messed up or something. I also checked all my uncommitted changes were the cluster removal (so I'm not testing saving the cluster config, obviously). Can you try building the JAR from sandbox/storeconfig7/trunk ? If there's a problem, creating a new tag would be easy.

Digging deeper - The Server/Listener-s need to be written. StoreConfigLifecycleListener is removed along with the other Listeners. 

Changing server-registry and commenting the TransientChild elements under the Listener Description seems to have no affect.

"The Server/Listener-s need to be written."
Yes, and this works fine for me. Did you try trunk ?

solved the Listener problem. Next: Connector/connectionTimeout is being defined twice.

*** Bug 583996 has been marked as a duplicate of this bug. ***

tomcat7-storeconfig-0.0.1.Alpha2-redhat-4.jar tested with EWS 2.0.1 T7 + JON 3.2.0.ER2 with SunJDK 1.6 on RHEL6
================================================================================================================
Findings:
----------
1. Listeners are mostly fixed, but there is still missing <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
2. There is a still missing attribute address in connector elements
3. There is a still missing Realm element <Realm className="org.apache.catalina.realm.UserDatabaseRealm" .....
<Realm className="org.apache.catalina.realm.LockOutRealm">
  <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
</Realm>
4. Persistency of changed deployed webapp configuration doesn't work


tomcat6-storeconfig-0.0.1.Alpha2-redhat-3.jar tested with EWS 2.0.1 T6 + JON 3.2.0.ER2 with SunJDK 1.6 on RHEL5
================================================================================================================
Findings:
----------
1. T6 not well formed server.xml is created after changing configuration of deployed application (comm#23).
2. There is a still missing attribute address in connector elements

I can work on it, but you need to clarify 4), saving context.xml (which is the webapp config) actually works fine for me in Tomcat.

ad 4)
I deployed webbapp without context.xml file. The problem is storeconfig didn't create it after JON UI webapp config change.

Ok, maybe this is about the webapps that do not have a context.xml, in that case I don't think it creates one (otherwise, it would always be creating one, since it's not going to know if there's anything to save). That's a limitation.

I think the storeconfig should always create context.xml.

Ok, I think I found a way. That's a lot of changes to port, and the branches are all different.

After porting, all looks ok now, except item 3) for Tomcat 7, which needs a new API addition to realm. As storeconfig is supposed to support many Tomcat 7s as an addon, this does not look like a realistic scenario, so nested realms (which are quite useless anyway) cannot be supported.

Should I now create new tags for storeconfig6 and storeconfig7 ?

I think yes

Ok, new tag: Alpha3

With https://brewweb.devel.redhat.com/buildinfo?buildID=298779
1) Not tested TC6 doesn't support it.
2) It seems no a problem in storeconfig but some in the tomcat plugin.
3) Not tested TC6 doesn't support it.
4) the file is created it looks OK.

Created attachment 810029 [details]
storeconfig lib for tc7

adding for ease of access.

Created attachment 810181 [details]
storeconfig lib for tc6

Created attachment 810182 [details]
storeconfig lib for tc6

tomcat7-storeconfig-0.0.1.Alpha3-redhat-2.jar tested with EWS 2.0.1 T7 + JON 3.2.0.ER3 with SunJDK 1.6 on RHEL6
==============================================
1. Listeners - FIXED
2. missing attribute address in connector elements - FIXED
3. Nested Realm doesn't support it - ok, this will go to known issues
4. Persistency of changed deployed webapp configuration - FIXED
5. FOUND - Persistence of connector configuration change doesn't work (I tried to change "Max Threads" attribute to 300)

tomcat6-storeconfig-0.0.1.Alpha3-redhat-8.jar tested with EWS 2.0.1 T6 + JON 3.2.0.ER3 with SunJDK 1.6 on RHEL6
==============================================
1. T6 not well formed server.xml is created after changing configuration of deployed application (comm#23) - FIXED
2. missing attribute address in connector elements - FIXED
3. FOUND - Persistency of changed deployed webapp configuration doesn't work

2) How did you test it? It doesn't work for me. (TC6 and TC7).

2) address attribute in connector elements - 
   - i set this attribute before starting tomcat in server.xml manually
   - start tomcat and discovery it to JON
   - change some other attribute for connector in JON UI
   - restart tomcat in JON UI
   - check the connector attributes

and whats your steps Jean-Frederic ?

Address:
 - start tomcat
 - discover and import it
 - set the address in the JON UI and press save
the server.xml is not changed.

https://git.fedorahosted.org/cgit/rhq/rhq.git/commit/?id=7548d1752c8d518f43cf8b8deb496b793c9b1866
fix in master that needs to be ported.

tomcat6-storeconfig-0.0.1.Alpha3-redhat-12.jar tested with EWS 2.1.0 ER2 T6 + JON 3.2.0.GA with SunJDK 1.6 on RHEL6
===============================================
Persistency of changed deployed webapp configuration doesn't work

tomcat7-storeconfig-0.0.1.Alpha3-redhat-3.jar tested with EWS 2.1.0 ER2 T7 + JON 3.2.0.GA with SunJDK 1.6 on RHEL6
===============================================
Persistence of connector configuration change doesn't work (I tried to change "Max Threads" attribute to 300)

That isn't storeconfig problem.
BTW: to test storeconfig you don't need JON but just jconsole.

That isn't storeconfig problem.
BTW: to test storeconfig you don't need JON but just jconsole.

do you have something in the agent.log file?

It works with master. So it is fix missing somewhere in rhq code...

Created attachment 911655 [details]
JON agent.log for T6 config app problem

Created attachment 911656 [details]
Catalina.out for T6 config app problem

Created attachment 911657 [details]
agent.log for T7 config virtual host problem

Created attachment 911658 [details]
Catalina.out for T7 config virtual host problem

https://git.fedorahosted.org/cgit/rhq/rhq.git/patch/?id=7548d1752c8d518f43cf8b8deb496b793c9b1866 is missing in the branch.

I don't have problems with the virtual host what are you testing?
BTW: Could you please limit the logs to the error or around the error?

Sorry, its my mistake in comm# 72 nad 73.
T7 has problem with - Persistence of connector configuration change doesn't work (I tried to change "Max Threads" attribute to 300)

Created attachment 911676 [details]
sample application for JON Tomcat plugin deploy/undeploy/config change testing

Larry, please verify comm#74 and add the patch for building new Tomcat plugin build.

Note http://svn.apache.org/viewvc?view=revision&revision=r1605724 might be back ported to provide patches for our branches.

Created attachment 912410 [details]
patch for mod_cluster "problem".

Prevent server.xml being destroyed by mod_cluster listener.

Created attachment 914483 [details]
Patch for Tomcat 6 context files creation

Adds the capability to save context config if the config file doesn't exist in storeconfig6.

pre-tested Remy's patch and its safe to include it in CR1

Verified with EWS 2.1.0 CR2 on RHEL - OracleJDK 1.6/1.7, OpenJDK 1.6/1.7, IBMJDK 1.6/1.7 
MS Windows, Solaris - OracleJDK 1.6/1.7

waiting for rpm tests...

O.K.

This fix, change, or update was tested and released in Web Server Plugin Pack Update-01 for JBoss ON 3.2 -- https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=32543