Bug 833143 - (EMBJOPR-367) [as5 plugin] lack of concurrency around the ManagementView instance used for each managed AS instance can cause exceptions when managementView.load() is called by one thread while another thread is trying to call accessors on the management view
[as5 plugin] lack of concurrency around the ManagementView instance used for ...
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Plugins (Show other bugs)
4.4
Unspecified Unspecified
unspecified Severity high (vote)
: ---
: ---
Assigned To: Stefan Negrea
Mike Foley
:
Depends On:
Blocks: 795918
  Show dependency treegraph
 
Reported: 2012-06-18 13:23 EDT by Ian Springer
Modified: 2013-09-04 03:48 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-04 03:48:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker EMBJOPR-367 Critical Closed subtype=QueueManage temporarily disappears from ManagementView compByCompType resulting in "There are currently no nume... 2012-11-27 00:50:30 EST

  None (edit)
Description Ian Springer 2012-06-18 13:23:16 EDT

    
Comment 1 Ian Springer 2012-06-18 13:30:24 EDT
The same management view object is actually shared by all the resource components and discovery components associated with a particular JBAS instance. Unfortunately, in a whole bunch of places in the plugin code, we call managementView.load(), which clears the compByCompType Map and a whole bunch of other internal Maps. There is no synchronization around load() and the various accessors in management view, so it's not surprising we're seeing concurrency issues.

This is not going to be a trivial fix. We'll either need to 1) refactor the entire plugin to use separate management view instances, or 2) wrap some sort of concurrency around the management view instance that we use. 1) is probably the better option, since the blocking 2) would introduce could cause performance issues.

This issue was first discovered in the admin-console (see EMBJOPR-367), but I am pretty sure it will affect RHQ as well. To reproduce it, I would try the following:

1) inventory an EAP 5.1.2 instance that has HornetQ deployed and uses the 'all' configuration
2) crank up the metric schedules for all HornetQ JMS Queue Resources (this could be done via the metric template for this type)
3) restart the Agent on the machine where the EAP 5.1.2 instance is running w/ DEBUG logging enabled
4) run "discovery -f" a few times from the prompt of the same Agent
5) look for exceptions similar to the following in the Agent log:

DEBUG [org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler] (http-0.0.0.0-8080-2) Call to [org.jboss.as.integration.hornetq.jopr.JMSQueueComponent.getValues()] with args [[org.rhq.core.domain.measurement.MeasurementReport@560399f8, [MeasurementScheduleRequest[scheduleId=1, name=getDeliveringCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessageCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getConsumerCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessagesAdded, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getScheduledCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null]]]] failed.
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invokeInNewThreadWithLock(ResourceContainer.java:446)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invoke(ResourceContainer.java:434)
at $Proxy193.getValues(Unknown Source)
at org.rhq.core.pc.measurement.MeasurementManager.getRealTimeMeasurementValue(MeasurementManager.java:435)
at org.jboss.on.embedded.bean.MeasurementUtils.loadMeasurementData(MeasurementUtils.java:126)
at org.jboss.on.embedded.ui.MetricAction.view(MetricAction.java:107)
...
Caused by: java.lang.NullPointerException
at org.jboss.as.integration.hornetq.jopr.util.ManagementSupport.getOperation(ManagementSupport.java:40)
at org.jboss.as.integration.hornetq.jopr.JMSResourceComponent.getValues(JMSResourceComponent.java:86)
at sun.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
Comment 2 JBoss JIRA Server 2012-07-03 12:18:44 EDT
Stefan Negrea <snegrea@redhat.com> made a comment on jira EMBJOPR-367

Retested with the patch from JBPAPP-9406 and could not replicate the problem. This issue will be fully fixed with the EAP 5.1.x patch. No changes required in the Admin Console code.
Comment 3 JBoss JIRA Server 2012-07-03 16:17:42 EDT
Stefan Negrea <snegrea@redhat.com> updated the status of jira EMBJOPR-367 to Resolved
Comment 4 JBoss JIRA Server 2012-07-03 16:17:42 EDT
Stefan Negrea <snegrea@redhat.com> made a comment on jira EMBJOPR-367

Fixed by JBPAPP-9406.
Comment 5 JBoss JIRA Server 2012-10-26 14:54:05 EDT
Larry O'Leary <loleary@redhat.com> updated the status of jira EMBJOPR-367 to Closed
Comment 6 JBoss JIRA Server 2012-10-26 14:54:05 EDT
Larry O'Leary <loleary@redhat.com> made a comment on jira EMBJOPR-367

Closing as downstream bugs have confirmed this fix has been verified.
Comment 8 Heiko W. Rupp 2013-09-04 03:48:34 EDT
Bulk closing of some old issues

Note You need to log in before you can comment on or make changes to this bug.