Bug 638216 - When a queue is missing or in an invalid state, metric collection is interrupted
When a queue is missing or in an invalid state, metric collection is interrupted
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Plugins (Show other bugs)
3.0.0
All All
low Severity medium (vote)
: ---
: ---
Assigned To: Filip Drabek
Corey Welton
:
Depends On:
Blocks: 725852 jon241-bugs
  Show dependency treegraph
 
Reported: 2010-09-28 10:17 EDT by dsteigne
Modified: 2011-07-26 16:45 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-23 21:15:12 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description dsteigne 2010-09-28 10:17:56 EDT
Description of problem:

In JON 2.4 it appears that when a queue is missing or in an invalid state all metric collection is interrupted.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Charles Crouch 2010-11-16 16:10:49 EST
We need more info from GSS if we're going to be able reproduce this, but it should still be investigated.
Comment 2 dsteigne 2010-12-02 09:41:33 EST
Deploy a message queue in AS, have JON discover it, now remove it from the file system. JON still tries to gather metrics and availability for the queue. When the failure occurs it results in metric collection failing for all metrics due to an uncaught exception bubbling up to the metric sub-system.
Comment 4 Heiko W. Rupp 2010-12-09 14:21:53 EST
For JBas4 plugin,
the code is in org.rhq.plugins.jbossas.AbstractMessagingComponent#getValues
the code within 

 for (MeasurementScheduleRequest request : requests) {

}

should be put into a 
try { .. do the existing work ..} 
catch (Exception e} {
   report
}

block

Looks like AS5 plugin has such a try-catch block already.
Comment 5 Filip Drabek 2010-12-09 16:01:14 EST
It is problem in AS5 plugin.
Comment 6 Filip Drabek 2010-12-13 11:04:02 EST
2010-12-13 15:02:48,853 ERROR [ResourceContainer.invoker.daemon-13] (org.rhq.plugins.jbossas5.ApplicationServerComponent)
java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57)
	at $Proxy81.getComponentsForType(Unknown Source)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:823)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
	at org.jboss.remoting.Client.invoke(Client.java:1917)
	at org.jboss.remoting.Client.invoke(Client.java:768)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java)
	... 16 more
Caused by: java.lang.InterruptedException
	at EDU.oswego.cs.dl.util.concurrent.Semaphore.attempt(Semaphore.java:120)
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1124)
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
	at org.jboss.remoting.Client.invoke(Client.java:1917)
	at org.jboss.remoting.Client.invoke(Client.java:768)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java)
	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57)
	at $Proxy81.getComponentsForType(Unknown Source)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
	... 22 more
Comment 7 Filip Drabek 2010-12-17 08:30:55 EST
commit a5f87a1e536094352beeaa8360da2a7624d9022a release-3branch
Comment 8 Filip Drabek 2010-12-20 05:34:36 EST
In some rare cases connection to profile service becomes defunct. The whole AS server is than reported as unavailable because agent is using the old defunct connection to profile service and all operations on profile service fails.
To reproduce the state I have deployed about 20 defunct message queues and 5 working queues and randomly deleted some of them. 
During the tests there should be high load on processor and memory.
When the connection will become defunct you will find in agent log INFO Connection has become defunct. Reconnecting...
Comment 9 Rajan Timaniya 2010-12-21 06:55:52 EST
Verified on JON-2.4.1 build #f52 (build number: 10966:3cea406)

Steps:
1) Install JON-2.4.1 and discover JBossAS-5.1 server
2) Deploy a message queue in AS5 (create more then 25 message queues)
3) Wait for some time so JON will discover it
4) Remove some message queues from the file system

Observation:
Removed message queues shows unavailable in JON, other message queues shows available and collecting metrics. There isn't any error/exception in server or agent log.
Comment 10 Corey Welton 2011-05-23 21:15:12 EDT
Bookkeeping - closing bug - fixed in recent release.
Comment 11 Corey Welton 2011-05-23 21:15:20 EDT
Bookkeeping - closing bug - fixed in recent release.
Comment 12 Corey Welton 2011-05-23 21:15:22 EDT
Bookkeeping - closing bug - fixed in recent release.
Comment 13 Corey Welton 2011-05-23 21:15:23 EDT
Bookkeeping - closing bug - fixed in recent release.

Note You need to log in before you can comment on or make changes to this bug.