Bug 595792

Summary: Have agent-server comm start earlier
Product: [Other] RHQ Project Reporter: Heiko W. Rupp <hrupp>
Component: Communications SubsystemAssignee: John Mazzitelli <mazz>
Status: CLOSED WONTFIX QA Contact: Corey Welton <cwelton>
Severity: medium Docs Contact:
Priority: low    
Version: 3.0.0Keywords: FutureFeature
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-24 13:24:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Heiko W. Rupp 2010-05-25 15:57:45 UTC
With the new self-updating agents, there is no more need to shut agents down at server upgrade time.
Downside is that now the agents (re-) connect to the server "much too early" leaving nasty exceptions in the server log as soon as tomcat has enabled the remoting-servlet URL:

2010-05-25 16:21:51,969 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
java.lang.reflect.UndeclaredThrowableException
	at $Proxy433.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
	at java.lang.Thread.run(Thread.java:619)
Caused by: javax.management.InstanceNotFoundException: jboss.remoting:service=invoker,transport=servlet is not registered.
	at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:523)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
	... 26 more



They only go away when the agent-server-comm is initialized:

2010-05-25 16:25:57,714 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]] initializer: Starting the server-agent communications services

Comment 1 Corey Welton 2010-09-24 12:38:59 UTC
mazz - what do you think of the suggestion?

Comment 2 John Mazzitelli 2010-09-24 13:14:47 UTC
This would require a reshuffling of the startup routines in StartupServlet. In past experience, changing the ordering of when the comm layer starts is VERY dangerous and can have many side effects.  Several times in the past we tried changing the ordering of the StartupServlet and its come back to bite us because of the side effects that we didn't realize would happen.

Therefore, unless absolutely necessary, we should not be changing when the comm layer starts up (or any of the ordering of things in StartupServlet). If the only problem here is an exception message in the logs, that does not rise to the level to where we should change this.

Comment 3 John Mazzitelli 2011-03-24 13:24:10 UTC
will not be shuffling the startup routines in StartupServlet. Alot has gone into getting the StartupServlet "just right" and agent-server comm is getting started in the proper place. If we move it up earlier, other bad things will happen. See the comments in StartupServlet for more info on why we should not adjust this stuff unless absolutely necessary (and even in that case, it still may not be possible to re-shuffle the startup code without introducing other bugs).