Bug 997601

Summary: Avail check for RHQ Server component (Measurement Subsystem, e.g.) can throw NotOpenException
Product: [Other] RHQ Project Reporter: Elias Ross <genman>
Component: PluginsAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED DUPLICATE QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.8CC: hrupp, mazz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-07 18:38:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Elias Ross 2013-08-15 18:02:46 UTC
Description of problem:

Availability check for RHQ Server can thrown 'NotOpenException', as show below:

java.lang.reflect.UndeclaredThrowableException
	at $Proxy74.isRegistered(Unknown Source)
	at org.mc4j.ems.impl.jmx.connection.bean.DMBean.isRegistered(DMBean.java:188)
	at org.rhq.plugins.jmx.MBeanResourceComponent.isMBeanAvailable(MBeanResourceComponent.java:242)
	at org.rhq.plugins.jmx.MBeanResourceComponent.getAvailability(MBeanResourceComponent.java:229)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.jboss.remoting3.NotOpenException: Writes closed
	at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297)
	at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:180)
	at org.jboss.remotingjmx.protocol.v2.ClientConnection$TheConnection.isRegistered(ClientConnection.java:741)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59)
	... 13 more

This may be related on this:

989879	JBoss Enterprise Application Platform 6	EJB	 jpai	NEW		NotOpenException: Cannot open new channel because close was initiated on server shutdown


Version-Release number of selected component (if applicable):

RHQ 4.8

How reproducible:

  Unclear, but it seems likely to happen when the server is closed. But it seems also to happen when the server is in fact running and perhaps has a stale connection.


Expected results:

   The check availability method should not ordinarily throw exceptions.


Additional info:

   It isn't clear if the agent will recover from this condition or not.

Comment 1 Heiko W. Rupp 2013-08-16 07:17:39 UTC
I see this as well:

2013-08-16 08:29:59,874 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [RHQ Serve
r Plugin Subsystem] Resources - failed after 0 ms.
java.lang.Exception: Discovery component invocation failed.
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:298)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.reflect.UndeclaredThrowableException
        at com.sun.proxy.$Proxy90.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:307)
        at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:332)
        at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:149)
        at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:98)
        at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:85)
        at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:294)
        ... 5 more
Caused by: org.jboss.remoting3.NotOpenException: Writes closed
        at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297)
        at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:180)
        at org.jboss.remotingjmx.protocol.v2.ClientConnection$TheConnection.queryNames(ClientConnection.java:699)
        at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59)
        ... 15 more

Comment 2 John Mazzitelli 2013-10-02 20:37:34 UTC
this is replicated by doing the following:

1) import the RHQ Server resource
2) make sure everything is in inventory and gree
3) shutdown the server (but do not shutdown the agent)
4) immediately restart the server

Eventually, the error happens to the RHQ Server service resources.

Comment 3 John Mazzitelli 2013-10-02 20:40:47 UTC
I think this is either the cause of the problem, or its related:

https://issues.jboss.org/browse/AS7-4508

Comment 4 John Mazzitelli 2013-10-03 20:42:16 UTC
In MBeanResourceComponent in the JMX plugin:

    private boolean isMBeanAvailable() {
        EmsBean emsBean = getEmsBean();
        boolean isAvailable = emsBean.isRegistered();

that isRegistered() call is what throws the exception. We need the JBossAS7 plugin to somehow capture that exception and clean out is old connection and rebuild the connection. I think that should work, though I'm not sure reading the AS7 JIRAs.

Comment 5 John Mazzitelli 2013-10-07 18:31:32 UTC
I tried to fix this by re-building the connection on error. However, now I'm seeing this, as reported in BZ against JBoss EAP - bug #900595

Comment 6 John Mazzitelli 2013-10-07 18:38:26 UTC
closing this as dup to bug #900595 since its all affecting the same thing.

*** This bug has been marked as a duplicate of bug 1003797 ***