Bug 833143 (EMBJOPR-367)

The same management view object is actually shared by all the resource components and discovery components associated with a particular JBAS instance. Unfortunately, in a whole bunch of places in the plugin code, we call managementView.load(), which clears the compByCompType Map and a whole bunch of other internal Maps. There is no synchronization around load() and the various accessors in management view, so it's not surprising we're seeing concurrency issues.

This is not going to be a trivial fix. We'll either need to 1) refactor the entire plugin to use separate management view instances, or 2) wrap some sort of concurrency around the management view instance that we use. 1) is probably the better option, since the blocking 2) would introduce could cause performance issues.

This issue was first discovered in the admin-console (see EMBJOPR-367), but I am pretty sure it will affect RHQ as well. To reproduce it, I would try the following:

1) inventory an EAP 5.1.2 instance that has HornetQ deployed and uses the 'all' configuration
2) crank up the metric schedules for all HornetQ JMS Queue Resources (this could be done via the metric template for this type)
3) restart the Agent on the machine where the EAP 5.1.2 instance is running w/ DEBUG logging enabled
4) run "discovery -f" a few times from the prompt of the same Agent
5) look for exceptions similar to the following in the Agent log:

DEBUG [org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler] (http-0.0.0.0-8080-2) Call to [org.jboss.as.integration.hornetq.jopr.JMSQueueComponent.getValues()] with args [[org.rhq.core.domain.measurement.MeasurementReport@560399f8, [MeasurementScheduleRequest[scheduleId=1, name=getDeliveringCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessageCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getConsumerCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getMessagesAdded, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null], MeasurementScheduleRequest[scheduleId=1, name=getScheduledCount, interval=0, enabled=true, dataType=MEASUREMENT, rawNumericType=null]]]] failed.
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invokeInNewThreadWithLock(ResourceContainer.java:446)
at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invoke(ResourceContainer.java:434)
at $Proxy193.getValues(Unknown Source)
at org.rhq.core.pc.measurement.MeasurementManager.getRealTimeMeasurementValue(MeasurementManager.java:435)
at org.jboss.on.embedded.bean.MeasurementUtils.loadMeasurementData(MeasurementUtils.java:126)
at org.jboss.on.embedded.ui.MetricAction.view(MetricAction.java:107)
...
Caused by: java.lang.NullPointerException
at org.jboss.as.integration.hornetq.jopr.util.ManagementSupport.getOperation(ManagementSupport.java:40)
at org.jboss.as.integration.hornetq.jopr.JMSResourceComponent.getValues(JMSResourceComponent.java:86)
at sun.reflect.GeneratedMethodAccessor622.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more

Stefan Negrea <snegrea> made a comment on jira EMBJOPR-367

Retested with the patch from JBPAPP-9406 and could not replicate the problem. This issue will be fully fixed with the EAP 5.1.x patch. No changes required in the Admin Console code.

Stefan Negrea <snegrea> updated the status of jira EMBJOPR-367 to Resolved

Stefan Negrea <snegrea> made a comment on jira EMBJOPR-367

Fixed by JBPAPP-9406.

Larry O'Leary <loleary> updated the status of jira EMBJOPR-367 to Closed

Larry O'Leary <loleary> made a comment on jira EMBJOPR-367

Closing as downstream bugs have confirmed this fix has been verified.

Bulk closing of some old issues