Bug 833143 (EMBJOPR-367)

Summary: [as5 plugin] lack of concurrency around the ManagementView instance used for each managed AS instance can cause exceptions when managementView.load() is called by one thread while another thread is trying to call accessors on the management view
Product: [Other] RHQ Project Reporter: Ian Springer <ian.springer>
Component: PluginsAssignee: Stefan Negrea <snegrea>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4CC: ccrouch, hrupp, loleary, spinder
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-04 03:48:34 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 795918    

Description Ian Springer 2012-06-18 13:23:16 EDT

    
Comment 1 Ian Springer 2012-06-18 13:30:24 EDT
The same management view object is actually shared by all the resource components and discovery components associated with a particular JBAS instance. Unfortunately, in a whole bunch of places in the plugin code, we call managementView.load(), which clears the compByCompType Map and a whole bunch of other internal Maps. There is no synchronization around load() and the various accessors in management view, so it's not surprising we're seeing concurrency issues.

This is not going to be a trivial fix. We'll either need to 1) refactor the entire plugin to use separate management view instances, or 2) wrap some sort of concurrency around the management view instance that we use. 1) is probably the better option, since the blocking 2) would introduce could cause performance issues.

This issue was first discovered in the admin-console (see EMBJOPR-367), but I am pretty sure it will affect RHQ as well. To reproduce it, I would try the following:

1) inventory an EAP 5.1.2 instance that has HornetQ deployed and uses the 'all' configuration
2) crank up the metric schedules for all HornetQ JMS Queue Resources (this could be done via the metric template for this type)
3) restart the Agent on the machine where the EAP 5.1.2 instance is running w/ DEBUG logging enabled
4) run "discovery -f" a few times from the prompt of the same Agent
5) look for exceptions similar to the following in the Agent log:

DEBUG [org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler] (http-0.0.0.0-8080-2) Call to [org.jboss.as.integration.hornetq.jopr.JMSQueueComponent.getValues()] with args [[org.rhq.core.domain.measurement.MeasurementReport@560399f8, [MeasurementScheduleRequest[scheduleId=1, name=getDeliveringCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessageCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getConsumerCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessagesAdded, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getScheduledCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null]]]] failed.
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invokeInNewThreadWithLock(ResourceContainer.java:446)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invoke(ResourceContainer.java:434)
at $Proxy193.getValues(Unknown Source)
at org.rhq.core.pc.measurement.MeasurementManager.getRealTimeMeasurementValue(MeasurementManager.java:435)
at org.jboss.on.embedded.bean.MeasurementUtils.loadMeasurementData(MeasurementUtils.java:126)
at org.jboss.on.embedded.ui.MetricAction.view(MetricAction.java:107)
...
Caused by: java.lang.NullPointerException
at org.jboss.as.integration.hornetq.jopr.util.ManagementSupport.getOperation(ManagementSupport.java:40)
at org.jboss.as.integration.hornetq.jopr.JMSResourceComponent.getValues(JMSResourceComponent.java:86)
at sun.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
Comment 2 JBoss JIRA Server 2012-07-03 12:18:44 EDT
Stefan Negrea <snegrea@redhat.com> made a comment on jira EMBJOPR-367

Retested with the patch from JBPAPP-9406 and could not replicate the problem. This issue will be fully fixed with the EAP 5.1.x patch. No changes required in the Admin Console code.
Comment 3 JBoss JIRA Server 2012-07-03 16:17:42 EDT
Stefan Negrea <snegrea@redhat.com> updated the status of jira EMBJOPR-367 to Resolved
Comment 4 JBoss JIRA Server 2012-07-03 16:17:42 EDT
Stefan Negrea <snegrea@redhat.com> made a comment on jira EMBJOPR-367

Fixed by JBPAPP-9406.
Comment 5 JBoss JIRA Server 2012-10-26 14:54:05 EDT
Larry O'Leary <loleary@redhat.com> updated the status of jira EMBJOPR-367 to Closed
Comment 6 JBoss JIRA Server 2012-10-26 14:54:05 EDT
Larry O'Leary <loleary@redhat.com> made a comment on jira EMBJOPR-367

Closing as downstream bugs have confirmed this fix has been verified.
Comment 8 Heiko W. Rupp 2013-09-04 03:48:34 EDT
Bulk closing of some old issues