Bug 638216

Summary:	When a queue is missing or in an invalid state, metric collection is interrupted
Product:	[Other] RHQ Project	Reporter:	dsteigne
Component:	Plugins	Assignee:	Filip Drabek <fdrabek>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Corey Welton <cwelton>
Severity:	medium	Docs Contact:
Priority:	low
Version:	3.0.0	CC:	fdrabek, hrupp, rtimaniy
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-05-24 01:15:12 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	725852, 616081

Description dsteigne 2010-09-28 14:17:56 UTC

Description of problem:

In JON 2.4 it appears that when a queue is missing or in an invalid state all metric collection is interrupted.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Charles Crouch 2010-11-16 21:10:49 UTC

We need more info from GSS if we're going to be able reproduce this, but it should still be investigated.

Comment 2 dsteigne 2010-12-02 14:41:33 UTC

Deploy a message queue in AS, have JON discover it, now remove it from the file system. JON still tries to gather metrics and availability for the queue. When the failure occurs it results in metric collection failing for all metrics due to an uncaught exception bubbling up to the metric sub-system.

Comment 4 Heiko W. Rupp 2010-12-09 19:21:53 UTC

For JBas4 plugin,
the code is in org.rhq.plugins.jbossas.AbstractMessagingComponent#getValues
the code within 

 for (MeasurementScheduleRequest request : requests) {

}

should be put into a 
try { .. do the existing work ..} 
catch (Exception e} {
   report
}

block

Looks like AS5 plugin has such a try-catch block already.

Comment 5 Filip Drabek 2010-12-09 21:01:14 UTC

It is problem in AS5 plugin.

Comment 6 Filip Drabek 2010-12-13 16:04:02 UTC

2010-12-13 15:02:48,853 ERROR [ResourceContainer.invoker.daemon-13] (org.rhq.plugins.jbossas5.ApplicationServerComponent)
java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57)
	at $Proxy81.getComponentsForType(Unknown Source)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:823)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
	at org.jboss.remoting.Client.invoke(Client.java:1917)
	at org.jboss.remoting.Client.invoke(Client.java:768)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java)
	... 16 more
Caused by: java.lang.InterruptedException
	at EDU.oswego.cs.dl.util.concurrent.Semaphore.attempt(Semaphore.java:120)
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1124)
	at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:167)
	at org.jboss.remoting.Client.invoke(Client.java:1917)
	at org.jboss.remoting.Client.invoke(Client.java:768)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aop.generatedproxies.AOPProxy$1.getComponentsForType(AOPProxy$1.java)
	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.plugins.jbossas5.connection.JaasAuthenticationInvocationHandler.invoke(JaasAuthenticationInvocationHandler.java:57)
	at $Proxy81.getComponentsForType(Unknown Source)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailabilityNow(ApplicationServerComponent.java:153)
	at org.rhq.plugins.jbossas5.ApplicationServerComponent.getAvailability(ApplicationServerComponent.java:136)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
	... 22 more

Comment 7 Filip Drabek 2010-12-17 13:30:55 UTC

commit a5f87a1e536094352beeaa8360da2a7624d9022a release-3branch

Comment 8 Filip Drabek 2010-12-20 10:34:36 UTC

In some rare cases connection to profile service becomes defunct. The whole AS server is than reported as unavailable because agent is using the old defunct connection to profile service and all operations on profile service fails.
To reproduce the state I have deployed about 20 defunct message queues and 5 working queues and randomly deleted some of them. 
During the tests there should be high load on processor and memory.
When the connection will become defunct you will find in agent log INFO Connection has become defunct. Reconnecting...

Comment 9 Rajan Timaniya 2010-12-21 11:55:52 UTC

Verified on JON-2.4.1 build #f52 (build number: 10966:3cea406)

Steps:
1) Install JON-2.4.1 and discover JBossAS-5.1 server
2) Deploy a message queue in AS5 (create more then 25 message queues)
3) Wait for some time so JON will discover it
4) Remove some message queues from the file system

Observation:
Removed message queues shows unavailable in JON, other message queues shows available and collecting metrics. There isn't any error/exception in server or agent log.

Comment 10 Corey Welton 2011-05-24 01:15:12 UTC

Bookkeeping - closing bug - fixed in recent release.

Comment 11 Corey Welton 2011-05-24 01:15:20 UTC

Bookkeeping - closing bug - fixed in recent release.

Comment 12 Corey Welton 2011-05-24 01:15:22 UTC

Bookkeeping - closing bug - fixed in recent release.

Comment 13 Corey Welton 2011-05-24 01:15:23 UTC

Bookkeeping - closing bug - fixed in recent release.