Description of problem: When starting JBoss ON Server, the following ERROR message is logged: 14:48:27,910 INFO [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker]] (http-/10.33.0.12:7443-5) ServerInvokerServlet: invokerObjectNameQuery=jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* 14:48:27,911 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] (http-/10.33.0.12:7443-5) JBWEB000235: Allocate exception for servlet ServerInvokerServlet: javax.servlet.ServletException: Could not find the remoting servlet invoker: jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* - need to wait for remoting to be initialized later at org.rhq.enterprise.communications.servlet.ServerInvokerServlet.getInvokerFromInvokerName(ServerInvokerServlet.java:78) [classes:] at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.init(ServerInvokerServlet.java:78) [jboss-remoting-2.5.4.SP4.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_25] at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) [rt.jar:1.7.0_25] at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:155) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:108) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1188) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:950) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:151) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Version-Release number of selected component (if applicable): JBoss ON 3.2.0 Beta How reproducible: Sometimes - when the agent is still running and is trying to connect to the JBoss ON Server before the server is actually up. Steps to Reproduce: 1. 2. 3. Actual results: Above ERROR messages are logged in the server.log file until the following message is logged in the server.log file: INFO [org.rhq.enterprise.server.core.StartupBean] (pool-7-thread-1) Starting the server-agent communications services... Expected results: No ERROR messages in the server.log file. Additional info: Remote servlet is enabled too early in the server start up procedure so the running agents can try to contact server before the server is actually up. We should change this so that remote servlet is enabled later so that we don't get ERROR messages in the log file.
master commit c124f467eec21a367e493b5b9d7981657484be14 Author: Jay Shaughnessy <jshaughn> Date: Tue Dec 3 13:38:09 2013 -0500 This is not a simple issue to resolve because it is not RHQ that logs the ERROR level exception. It is logged by JBoss. Our code is required to throw a ServletException indicating that we can't find the servlet server invoker (because it's not yet initialized). We could hide ERRORs for this JBoss package via log configuration, but then we may suppress actual, relevant runtime problems. The commit updates the message included in our ServletException to be very explicit about this not being an actual problem during server startup and initialization. Hopefully this will be sufficient. If this proves insufficient we should likely suppress the logging or, if we absolutely must log it, direct it to another file (not really desirable).
To be clear, the message Jay changed it to is: Could not find the remoting servlet invoker: jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* DURING SERVER STARTUP AND INITIALIZATION THIS IS NOT AN ERROR AND CAN BE IGNORED. This may be a problem if occurring during normal server runtime.
As updated in the RHQ BZ, the messaging change is already in. I suggest this be closed.
Not sure that's an option. Also, correction: The messaging is not in the release/3.2.x branch. Shall it be cherry-picked from master for 3.2.1?
fixed in master -- see https://bugzilla.redhat.com/show_bug.cgi?id=1035723#c5
Jay, please c-p it over.
release/jon3.2.x commit df0caa5144d9f0eceb01e3bee529da620da1ace2 Author: Jay Shaughnessy <jshaughn> Date: Mon Jun 23 10:14:54 2014 -0400 Cherry-Pick master 809d86f43c966e6a527e32cddabf5b1c4ce56ab4 Signed-off-by: Jay Shaughnessy <jshaughn>
Moving to ON_QA as available for test in latest build: http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.2.GA/6-28-2014/
Verified against 3.2.2 DR3. Verification steps: ./rhqctl stop --storage --agent ./rhqctl stop --server ./rhqctl start --storage --agent ; ./rhqctl start --server validate whether stack trace is present in server.log
This has been verified and released in Red Hat JBoss Operations Network 3.2 Update 02 (3.2.2) available from the Red Hat Customer Portal[1]. [1]: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=31783
removing needs info flag