Bug 679510

Summary: Race condition in PluginDeploymentScanner during RHQ server shutdown
Product: [Other] RHQ Project Reporter: Lukas Krejci <lkrejci>
Component: Core ServerAssignee: RHQ Project Maintainer <rhq-maint>
Status: NEW --- QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.0.0   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Lukas Krejci 2011-02-22 14:01:08 EST
Description of problem:

While the RHQ Server was shutting down, the following appeared in the log:

19:49:06,266 ERROR [PluginDeploymentScanner] Scan failed. Cause: java.lang.RuntimeException:Failed to lookup local interface to EJB class org.rhq.enterprise.server.resource.metadata.PluginManagerBean, localJNDI=[rhq/PluginManagerBean/local] -> javax.naming.NameNotFoundException:local not bound

followed by a number of:

19:49:17,068 INFO  [WebappClassLoader] Illegal access: this web application instance has been stopped already.  Could not load org.apache.log4j.Level.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
        at org.apache.commons.logging.impl.Log4jProxy$1.run(Log4jProxy.java:68)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.commons.logging.impl.Log4jProxy.threadContextClassLoader(Log4jProxy.java:90)
        at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:78)
        at sun.reflect.GeneratedConstructorAccessor11.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:605)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:333)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:307)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:645)
        at org.rhq.enterprise.server.system.SystemManagerBean.<init>(SystemManagerBean.java:95)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.jboss.ejb3.EJBContainer.construct(EJBContainer.java:517)
        at org.jboss.ejb3.AbstractPool.create(AbstractPool.java:66)
        at org.jboss.ejb3.InfinitePool.get(InfinitePool.java:49)
        at org.jboss.ejb3.ThreadlocalPool.create(ThreadlocalPool.java:50)
        at org.jboss.ejb3.ThreadlocalPool.get(ThreadlocalPool.java:90)
        at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:54)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:108)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.stateless.StatelessContainer.callTimeout(StatelessContainer.java:175)
        at org.jboss.ejb.txtimer.TimerImpl$TimerTaskImpl.run(TimerImpl.java:561)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)

and mixed with another instances of the PluginDeploymentScanner ERRORs.

Even after 5 minutes, the server still haven't finished shutting down.

How reproducible:
rarely

Steps to Reproduce:
1. Start the RHQ server
2. Shut it down
  
Actual results:
The above in the server logs, the server never completes the shutdown.

Expected results:
the RHQ server shuts down normally.