Bug 1122670

Summary:	ConcurrentModificationException in runtime discovery
Product:	[Other] RHQ Project	Reporter:	Elias Ross <genman>
Component:	Agent	Assignee:	Nobody <nobody>
Status:	NEW ---	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.12	CC:	hrupp
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Elias Ross 2014-07-23 18:13:23 UTC

Description of problem:

2014-07-23 18:07:51,305 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Removing stale resource [Resource[id=0, uuid=7126dc53-dff1-41f1-9a04-eecf352be8db, type={datatorrent}container, key=container_1402467549835_0049_01_000009, name=c_000009, parent=Application]]
2014-07-23 18:07:51,308 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error in runtime discovery
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$KeyIterator.next(HashMap.java:828)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.removeStaleResources(RuntimeDiscoveryExecutor.java:321)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:298)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:304)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.discoverForResource(RuntimeDiscoveryExecutor.java:304)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:146)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:104)
        at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:60)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

...

    private void removeStaleResources(Resource parent, ResourceType childResourceType,
        Map<String, Resource> mergedResources) {
        Set<Resource> existingChildResources = parent.getChildResources();
        for (Resource existingChildResource : existingChildResources) {

This loop needs to get fixed.

It is unclear how to reproduce this, as it is a concurrency issue. It seems that copying the children won't really work as locking is really required.