Bug 583996

Summary: EWS Tomcat: "Store Configuration" operation doesn't appear to do anything?
Product: [JBoss] JBoss Enterprise Web Server 2 Reporter: Corey Welton <cwelton>
Component: JON PluginAssignee: David Knox <dknox>
Status: CLOSED DUPLICATE QA Contact: Libor Fuka <lfuka>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: ccrouch, cwelton, dknox, jclere, jdoyle, jmartisk, jshaughn, lfuka, majoshi, mfoley, myarboro, pslavice, rhatlapa, rmaucher, rsvoboda, snegrea
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Other   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
In a running implementation of JBoss Enterprise Web Server's JBoss Operations Network (JON), selecting the <guimenu>Store Configuration</guimenu> option in the <guimenu>Operations</guimenu> menu does not update the <filename>server.xml</filename> file. This occurs because the <classname>StoreConfig</classname> MBean used to persist changes to <filename>server.xml</filename> was not included in the version of Tomcat that shipped with JBoss Enterprise Web Server. This is a known issue for JBoss Enterprise Web Server 2.0.1 and currently no workaround is available for this issue.
Story Points: ---
Clone Of: Environment:
Solaris / EWS 1.0.1
Last Closed: 2013-09-25 10:15:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 956509    
Bug Blocks: 683054, 625146    
Attachments:
Description Flags
jar for tomcat6
none
jar for tomcat7
none
Tomcat6 catalina.out after instalation of jar and testing none

Description Corey Welton 2010-04-20 13:13:22 UTC
Description of problem:
When I select the "Store Configuration" op, it says it executes and is successful, but I don't see anything in the agent.log on the platform, and the file doesn't appear to change.

Version-Release number of selected component (if applicable):


How reproducible:
Every time

Steps to Reproduce:
1.  Get EWS up, running and monitored (green) on a Solaris box
2.  Operations > Store Configuration
3.  Submit
  
Actual results:
Nothing shows up in server log; also, we never see the date change on the filename:

bash-2.05# ls -alF ./share/apache-tomcat-6.0.24/conf/server.xml
-rw-r--r--   1 root     tomcat      6616 Feb 11 10:52 ./share/apache-tomcat-6.0.24/conf/server.xml


Expected results:

Op takes place as expected.
Additional info:

Comment 1 Corey Welton 2010-04-20 13:28:38 UTC
From EWS log:

Apr 20, 2010 6:05:41 AM org.apache.catalina.core.StandardServer storeConfig
SEVERE: StoreConfig mbean not registeredCatalina:type=StoreConfig

Comment 2 Charles Crouch 2010-05-20 15:22:41 UTC
Should check the code/jshaughn to determine what this was meant to do.

Comment 3 Jay Shaughnessy 2010-05-27 13:33:02 UTC
So, the StoreConfig operation is basically intended to persist the mbean settings to server.xml.  You can change mbean settings all you like but the setting will last only until Tomcat is shut down.

Exposing StoreConfig as an operation, I think, was basically done because it could be done. Not for any specific reason, but rather for some unanticipated user need.

Even if we didn't expose it we would call it from the plugin internals. We invoke the storeConfig operation ourselves to persist config changes. So, when a user makes a change in our config gui we want it to be persisted.  So, we follow up the mbean setters with a call to storeConfig to ensure that the changes survive the next restart.

So, in general, a call to storeConfig may not have obvious consequences unless you know what you're looking for.

Now, for the error above, that does not look like a good thing. Certainly these calls have been being made successfully for previous versions. We'll need to understand why this mbean may not have been accessible.

Comment 4 John Sanda 2010-05-27 13:51:01 UTC
I have EWS 1.0.1 installed and running on Fedora 12, and I am seeing the same error that Corey reported in catalina.out. I will do some investigation to verify whether or not the StoreConfig mbean is in fact unavailable.

Comment 5 John Sanda 2010-05-27 14:04:19 UTC
https://issues.apache.org/bugzilla/show_bug.cgi?id=42708 indicates that the StoreConfig mbean was not initially shipped with tomcat 6.

Comment 6 John Sanda 2010-05-27 14:26:57 UTC
I just did a couple tests and it does not appear that server.xml is getting updated. 

Steps:
1. In the resource browser, drill down into the localhost tomcat virtual host.
2. Go to the configuration tab and click edit.
3. Change the values of the Auto Deploy and Unpack Wars properties.
4. Open <TOMCAT_HOME>/conf/server.xml and look at the <Host name="localhost"> element and see that the autoDeploy and unpackWAR attributes remain unchanged.

Comment 7 John Sanda 2010-05-27 17:36:52 UTC
Jay pointed out to me that the tomcat plugin is calling the Server mbean and not the StoreConfig mbean. I tested a call to the storeConfig() operation on the Server mean using jconsole, and server.xml was still not updated. I pulled down the tomcat source for versions 6.0.24 and 6.0.25 and verified that the storeConfig() operation does in fact ultimately attempt to delegate to the StoreConfig mbean. This of course fails since StoreConfig is no longer part of the tomcat distribution. The net effect of this is that resource config updates currently do not work for the tomcat vhost resource type.

Comment 8 Corey Welton 2010-09-21 01:48:12 UTC
jsanda, jshaughn - how are we going to fix this?

Comment 9 Jay Shaughnessy 2010-09-21 03:37:14 UTC
I don't think we're going to fix this. Rather TC is supposed to put this feature back in at some point, like 6.1.  I haven't checked recently for new releases...

Comment 10 Charles Crouch 2010-11-02 00:41:34 UTC
Push to JON3.0 to retest against a later version of EWS

Comment 11 Charles Crouch 2011-03-08 14:57:48 UTC
Associated with EWS1.0.2 so it can be tested to see if the version of Tomcat6 which ships with EWS1.0.2 actually supports this feature.

Comment 12 Ondřej Žižka 2011-03-16 01:38:24 UTC
Still the same behavior:

Nothing shows up in server log; also, we never see the date change on the file.

Comment 13 Ondřej Žižka 2011-03-16 01:51:50 UTC
EWS 1.0.2 / RHEL 5 x86_64 / Tomcat 6
JON 2.4.1

Comment 14 Ondřej Žižka 2011-03-16 21:16:22 UTC
Related? https://bugzilla.redhat.com/show_bug.cgi?id=688359

Comment 15 Stefan Negrea 2011-10-12 04:56:36 UTC
Tested with Tomcat 5.5.33 (distributed with EWS 1.0.2) and found no problem. The store configuration method is invoked correctly by RHQ and server.xml is updated every time this call is received by the Tomcat instance.

Comment 16 Stefan Negrea 2011-10-12 21:38:46 UTC
I was able to replicate the reported issue with Tomcat 6.0.32 (distributed with EWS 1.0.2).

storeConfig method of Server MBean makes an internal call to StoreConfig MBean to delegate the work. StoreConfig has not yet been ported to Tomcat 6.0.x. Here is the most recent Tomcat-dev discussion regarding this: http://mail-archives.apache.org/mod_mbox/tomcat-dev/201003.mbox/%3C4BA8EB2E.4040100@apache.org%3E .

The official Tomcat BZ regarding this: https://issues.apache.org/bugzilla/show_bug.cgi?id=42708


With regards to RHQ, the Tomcat plugin makes the correct call to storeConfig of Server MBean of the Tomcat instance. This is a non-issue on RHQ side.


The recommendation is to retest once the official Tomcat BZ is resolved.

Comment 17 Charles Crouch 2012-10-10 22:02:47 UTC
*** Bug 688359 has been marked as a duplicate of this bug. ***

Comment 18 Charles Crouch 2012-10-10 22:04:15 UTC
Marked 688359 as dupe, since its hitting the some issue, just when trying to configure Virtual Hosts

Comment 19 Jan Martiska 2012-10-11 13:27:49 UTC
*** Bug 864966 has been marked as a duplicate of this bug. ***

Comment 20 Charles Crouch 2012-10-11 19:08:58 UTC
Unassigning from Stefan

Comment 21 Charles Crouch 2012-10-11 19:13:16 UTC
Moved to the JON product BZ to make tracking it easier.

Comment 23 Jean-frederic Clere 2012-10-23 12:55:57 UTC
The work has been started in http://svn.apache.org/repos/asf/tomcat/sandbox/storeconfig6/

Comment 24 Rémy Maucherat 2012-11-15 12:59:02 UTC
Storeconfig for Tomcat 6: http://svn.apache.org/repos/asf/tomcat/sandbox/storeconfig6/
Storeconfig for Tomcat 7: http://svn.apache.org/repos/asf/tomcat/sandbox/storeconfig7/

The two JARs need to be built, added to the Tomcat libs, and the StoreConfig listener added to the Server. The storeconfig MBean will then be available.

However, the Tomcat configuration is very freeform, so storeconfig takes a lot of guesses and it will never be 100% correct in all cases. If things go wrong, it does a good job of saving copies of the previous configuration files.

Comment 25 John Doyle 2012-11-20 16:11:40 UTC
On the EWS call today we discussed that we needed to deliver a new jar for Tomcat to address this defect.  The options were to deliver with the plugin or deliver with EWS.  In order to avoid a new release of EWS, I would prefer to deliver with the plugin, but I'm not sure how that would happen.  

Is there a way to deliver this jar with the plugin such that the user does not have to install it, or is there a manual installation step involved?

Can we provide the jar as a bundle that the user can install in tomcat with JON?

What are our options?

~john

Comment 26 Rémy Maucherat 2012-11-20 16:18:24 UTC
The installation process for the user is:
- place the JAR in the Tomcat lib folder
- add a Listener to the Server element in server.xml, with className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"

Comment 27 John Doyle 2012-11-20 21:04:49 UTC
OK, so delivering it as a bundle is not enough because there is also a config change needed.  So we've to a chicken and egg problem.

I think the installation process you describe is OK for an initial fix.  I would like to explore if there is a better user experience that we can offer.  If this is the best we can do in the long term than I think it is probably best to add this jar to EWS in a future release.

Comment 28 Libor Fuka 2012-12-07 13:52:43 UTC
When and from where can we download JARs for testing with Tomcats ?

Comment 29 Libor Fuka 2012-12-07 14:01:33 UTC
When and from where can we download JARs for testing with Tomcats ? 
We need the JARs at 18.12.2012 at the latest. Thx.

Comment 30 Jean-frederic Clere 2012-12-12 17:28:19 UTC
Created attachment 662539 [details]
jar for tomcat6

Comment 31 Jean-frederic Clere 2012-12-12 18:09:32 UTC
Created attachment 662554 [details]
jar for tomcat7

Comment 32 Libor Fuka 2012-12-13 08:18:27 UTC
I tested with Tomcat6 from EWS 2.0 GA and catalina.out still shows the problem:
13.12.2012 9:16:57 org.apache.tomcat.util.modeler.BaseModelMBean invoke
SEVERE: Exception invoking method storeConfig
javax.management.InstanceNotFoundException: Catalina:type=StoreConfig
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:833)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
	at org.apache.catalina.core.StandardServer.storeConfig(StandardServer.java:641)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
	at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:447)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
	at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
	at sun.rmi.transport.Transport$1.run(Transport.java:159)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

Comment 33 Libor Fuka 2012-12-13 08:25:05 UTC
I added listener from comment 26 and there is another error after starting Tomcat6:

INFO: Starting Coyote AJP/1.3 on ajp-8009
13.12.2012 9:22:28 org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
13.12.2012 9:22:28 org.apache.tomcat.util.digester.Digester startElement
SEVERE: Begin event threw error
java.lang.NoClassDefFoundError: org/apache/catalina/ha/CatalinaCluster
	at org.apache.catalina.storeconfig.StoreRegistry.<clinit>(StoreRegistry.java:62)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at java.lang.Class.newInstance0(Class.java:355)
	at java.lang.Class.newInstance(Class.java:308)
	at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:206)
	at org.apache.tomcat.util.digester.Rule.begin(Rule.java:153)
	at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1356)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:501)
	at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(XMLDTDValidator.java:767)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1363)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$ContentDriver.scanRootElementHook(XMLDocumentScannerImpl.java:1318)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3104)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:922)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
	at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1661)
	at org.apache.catalina.storeconfig.StoreLoader.load(StoreLoader.java:242)
	at org.apache.catalina.storeconfig.StoreConfigLifecycleListener.createMBean(StoreConfigLifecycleListener.java:75)
	at org.apache.catalina.storeconfig.StoreConfigLifecycleListener.lifecycleEvent(StoreConfigLifecycleListener.java:58)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:759)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.ClassNotFoundException: org.apache.catalina.ha.CatalinaCluster
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 36 more
13.12.2012 9:22:28 org.apache.catalina.storeconfig.StoreLoader load
SEVERE: java.lang.NoClassDefFoundError: org/apache/catalina/ha/CatalinaCluster
13.12.2012 9:22:28 org.apache.catalina.startup.Catalina start
INFO: Server startup in 305 ms

Comment 34 Libor Fuka 2012-12-13 08:28:45 UTC
I also tested with JON again - operation chnage Tomcat configuration and there is also another error in catalina.out.
Attaching complete Tomcat6 catalina.out.

Comment 35 Libor Fuka 2012-12-13 08:29:52 UTC
Created attachment 662778 [details]
Tomcat6 catalina.out after instalation of jar and testing

Comment 36 Charles Crouch 2012-12-14 16:32:37 UTC
To be honest I think the best option here is to patch EWS so customers don't need to do anything when running the latest version of EWS.

I'm hesitant to put anything which is not a server or agent plugin in the plugin packs since it will likely confuse customers and lead to errors.

Another options is just to put the jar in the product-connectors folder in JON, which is available as a separate download, alongside the agent, from the JON admin pages. This currently contains binaries needed to configure monitoring of apache, and response time monitoring of EAP. That doesn't really solve the install issue which Remy talks about, customers still need to manually drop the jar in the right place and update their configuration file

(In reply to comment #27)
> OK, so delivering it as a bundle is not enough because there is also a
> config change needed.  So we've to a chicken and egg problem.
> 
> I think the installation process you describe is OK for an initial fix.  I
> would like to explore if there is a better user experience that we can
> offer.  If this is the best we can do in the long term than I think it is
> probably best to add this jar to EWS in a future release.

Comment 37 Libor Fuka 2013-01-10 10:15:59 UTC
Affects EWS 1.0.2 Tomcat6 and EWS 2.0.0 Tomcat6 & Tomcat7

Comment 39 Jean-frederic Clere 2013-04-16 15:56:10 UTC
the jar will be in EWS and the listener will be added to server.xml

Comment 40 Michal Haško 2013-04-17 11:19:15 UTC
*** Bug 901050 has been marked as a duplicate of this bug. ***

Comment 41 Mandar Joshi 2013-05-14 06:23:09 UTC
Added DocText.

@Wei Nan Li, can you please review the Doc Text content?

Comment 42 Weinan Li 2013-05-14 07:49:17 UTC
Hi Mandar, Dave is still working on adding storeconfig into EWS 2.0.1, and not finished yet. Here's the relative tickets:

https://engineering.redhat.com/rt/Ticket/Display.html?id=199363

For EWS1/2.0.0, we won't have this new package. It will be included in upcoming 2.0.1

Comment 43 Weinan Li 2013-05-14 07:49:42 UTC
Assign to Dave as he's working on this one currently.

Comment 44 Mandar Joshi 2013-05-27 12:19:29 UTC
@David, can you please review the Doc Text content?

Comment 45 Jean-frederic Clere 2013-05-31 06:38:01 UTC
"In a running implementation of JBoss Enterprise Web Server on Solaris"
Why Solaris it should be:
When .... in JON (JBoss Operations Network).

Comment 46 Misha H. Ali 2013-05-31 07:04:02 UTC
Is it correct now, Jean-Frederic?

Comment 47 Jean-frederic Clere 2013-05-31 07:43:45 UTC
"to the agent.log file or" remove that please.

In fact there is something in the agent.log but I don't remember what :-(

Comment 48 Misha H. Ali 2013-06-01 05:08:10 UTC
Thank you, Jean-Frederic. Consolidating into a single release note now. If anyone can spot anything else that may be incorrect, please point it out.

Comment 49 Libor Fuka 2013-06-03 06:18:47 UTC
This bug should be in known issues. It will not be fixed with EWS 2.0.1

Comment 50 Jean-frederic Clere 2013-06-12 08:44:57 UTC
The bug will be fixed in CR2 (June 18th).

Comment 51 Libor Fuka 2013-06-12 11:14:34 UTC
Misha please move it to bug fixes in 2.0.1 release notes.

Comment 52 Misha H. Ali 2013-06-12 11:31:59 UTC
Updated docs_text field and issue type to reflect that this is a resolved issue.

Comment 57 Libor Fuka 2013-09-25 10:15:48 UTC

*** This bug has been marked as a duplicate of bug 956509 ***