Description of problem: In JON 2.4 it appears that when a queue is missing or in an invalid state all metric collection is interrupted. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
We need more info from GSS if we're going to be able reproduce this, but it should still be investigated.
Deploy a message queue in AS, have JON discover it, now remove it from the file system. JON still tries to gather metrics and availability for the queue. When the failure occurs it results in metric collection failing for all metrics due to an uncaught exception bubbling up to the metric sub-system.
For JBas4 plugin, the code is in org.rhq.plugins.jbossas.AbstractMessagingComponent#getValues the code within for (MeasurementScheduleRequest request : requests) { } should be put into a try { .. do the existing work ..} catch (Exception e} { report } block Looks like AS5 plugin has such a try-catch block already.
It is problem in AS5 plugin.
2010-12-13 15:02:48,853 ERROR [ResourceContainer.invoker.daemon-13] (org.rhq.plugins.jbossas5.ApplicationServerComponent) java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57) at $Proxy81.getComponentsForType(Unknown Source) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136) at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.lang.InterruptedException at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:823) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167) at org.jboss.remoting.Client.invoke(Client.java:1917) at org.jboss.remoting.Client.invoke(Client.java:768) at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java) ... 16 more Caused by: java.lang.InterruptedException at EDU.oswego.cs.dl.util.concurrent.Semaphore.attempt(Semaphore.java:120) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1124) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167) at org.jboss.remoting.Client.invoke(Client.java:1917) at org.jboss.remoting.Client.invoke(Client.java:768) at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java) at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57) at $Proxy81.getComponentsForType(Unknown Source) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153) at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136) at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72) ... 22 more
commit a5f87a1e536094352beeaa8360da2a7624d9022a release-3branch
In some rare cases connection to profile service becomes defunct. The whole AS server is than reported as unavailable because agent is using the old defunct connection to profile service and all operations on profile service fails. To reproduce the state I have deployed about 20 defunct message queues and 5 working queues and randomly deleted some of them. During the tests there should be high load on processor and memory. When the connection will become defunct you will find in agent log INFO Connection has become defunct. Reconnecting...
Verified on JON-2.4.1 build #f52 (build number: 10966:3cea406) Steps: 1) Install JON-2.4.1 and discover JBossAS-5.1 server 2) Deploy a message queue in AS5 (create more then 25 message queues) 3) Wait for some time so JON will discover it 4) Remove some message queues from the file system Observation: Removed message queues shows unavailable in JON, other message queues shows available and collecting metrics. There isn't any error/exception in server or agent log.
Bookkeeping - closing bug - fixed in recent release.