Description of problem: Availability check for RHQ Server can thrown 'NotOpenException', as show below: java.lang.reflect.UndeclaredThrowableException at $Proxy74.isRegistered(Unknown Source) at org.mc4j.ems.impl.jmx.connection.bean.DMBean.isRegistered(DMBean.java:188) at org.rhq.plugins.jmx.MBeanResourceComponent.isMBeanAvailable(MBeanResourceComponent.java:242) at org.rhq.plugins.jmx.MBeanResourceComponent.getAvailability(MBeanResourceComponent.java:229) at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.jboss.remoting3.NotOpenException: Writes closed at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:180) at org.jboss.remotingjmx.protocol.v2.ClientConnection$TheConnection.isRegistered(ClientConnection.java:741) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59) ... 13 more This may be related on this: 989879 JBoss Enterprise Application Platform 6 EJB jpai NEW NotOpenException: Cannot open new channel because close was initiated on server shutdown Version-Release number of selected component (if applicable): RHQ 4.8 How reproducible: Unclear, but it seems likely to happen when the server is closed. But it seems also to happen when the server is in fact running and perhaps has a stale connection. Expected results: The check availability method should not ordinarily throw exceptions. Additional info: It isn't clear if the agent will recover from this condition or not.
I see this as well: 2013-08-16 08:29:59,874 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [RHQ Serve r Plugin Subsystem] Resources - failed after 0 ms. java.lang.Exception: Discovery component invocation failed. at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:298) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy90.queryNames(Unknown Source) at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:307) at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:332) at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:149) at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:98) at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:85) at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:294) ... 5 more Caused by: org.jboss.remoting3.NotOpenException: Writes closed at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:180) at org.jboss.remotingjmx.protocol.v2.ClientConnection$TheConnection.queryNames(ClientConnection.java:699) at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59) ... 15 more
this is replicated by doing the following: 1) import the RHQ Server resource 2) make sure everything is in inventory and gree 3) shutdown the server (but do not shutdown the agent) 4) immediately restart the server Eventually, the error happens to the RHQ Server service resources.
I think this is either the cause of the problem, or its related: https://issues.jboss.org/browse/AS7-4508
In MBeanResourceComponent in the JMX plugin: private boolean isMBeanAvailable() { EmsBean emsBean = getEmsBean(); boolean isAvailable = emsBean.isRegistered(); that isRegistered() call is what throws the exception. We need the JBossAS7 plugin to somehow capture that exception and clean out is old connection and rebuild the connection. I think that should work, though I'm not sure reading the AS7 JIRAs.
I tried to fix this by re-building the connection on error. However, now I'm seeing this, as reported in BZ against JBoss EAP - bug #900595
closing this as dup to bug #900595 since its all affecting the same thing. *** This bug has been marked as a duplicate of bug 1003797 ***