Bug 536368 (RHQ-725)

Summary: When importing server from AD - ConcurrentModificationException
Product: [Other] RHQ Project Reporter: Jay Shaughnessy <jshaughn>
Component: Plugin ContainerAssignee: Charles Crouch <ccrouch>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.1preCC: ccrouch, cwelton, hbrock
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
URL: http://jira.rhq-project.org/browse/RHQ-725
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Vista, RHQ 1.1 SNAPSHOT r1218
Last Closed: 2013-09-02 03:17:45 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Jay Shaughnessy 2008-08-11 15:01:00 EDT
2008-08-11 14:38:13,059 DEBUG [InventoryManager.discovery-1] (org.rhq.plugins.postgres.PostgresTableDiscoveryComponent)- Discovering postgres tables for mazz
2008-08-11 14:38:13,193 DEBUG [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running Runtime discovery on server: Resource[id=500676, type=Database, key=jon3, name=jon3 Database] for children of type: ResourceType[id=0, category=Service, name=Table, plugin=Postgres]

This may very well be an intermittent corner case but I did get hit by this when importing a server under an already imported platform.  I did the following odd sequence of steps:

Clean db
Import platform and its servers
uninventory (postgres) server
exit agent
restart agent --purgedata
agent plunks postgres server back in AD
import from AD

from the agent log...

2008-08-11 14:38:13,193 DEBUG [InventoryManager.discovery-1] (org.rhq.plugins.postgres.PostgresTableDiscoveryComponent)- Discovering postgres tables for jon3
2008-08-11 14:38:13,366 DEBUG [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running Runtime discovery on server: Resource[id=500667, type=Database, key=postgres, name=postgres Database] for children of type: ResourceType[id=0, category=Service, name=Table, plugin=Postgres]
2008-08-11 14:38:13,366 DEBUG [InventoryManager.discovery-1] (org.rhq.plugins.postgres.PostgresTableDiscoveryComponent)- Discovering postgres tables for postgres
2008-08-11 14:38:13,567 DEBUG [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running Runtime discovery on server: Resource[id=500665, type=Postgres Server, key=jdbc:postgresql://jon03.qa.atl2.redhat.com:5432/postgres, name=Postgres [postgres]] for children of type: ResourceType[id=0, category=Service, name=User, plugin=Postgres]
2008-08-11 14:38:13,567 INFO  [InventoryManager.discovery-1] (org.rhq.plugins.postgres.PostgresTableDiscoveryComponent)- Discovering postgres users
2008-08-11 14:38:14,902 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error running runtime report
java.util.ConcurrentModificationException
	at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:365)
	at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:376)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.runtimeDiscover(RuntimeDiscoveryExecutor.java:133)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:96)
	at org.rhq.core.pc.inventory.RuntimeDiscoveryExecutor.call(RuntimeDiscoveryExecutor.java:53)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
	at java.util.concurrent.FutureTask.run(FutureTask.java:123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:65)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:168)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
	at java.lang.Thread.run(Thread.java:595)
2008-08-11 14:38:16,856 DEBUG [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Running Availability Scan...


The problem is in RuntimeDiscoveryExecutor:

    private void runtimeDiscover(InventoryReport report) throws PluginContainerException {
        // Always start out by refreshing availabilities, since we will only scan servers that are available.
        this.inventoryManager.executeAvailabilityScanImmediately(true);
        if (this.resource == null) {
            // Run a full scan for all resources in the inventory
            Resource platform = this.inventoryManager.getPlatform();

            // Discover platform services here
            discoverForResource(platform, report, false);

            Set<Resource> servers = platform.getChildResources();
            for (Resource server : servers) {
                discoverForResource(server, report, false);
            }
        } else {
            // Run a single scan for just a resource and its descendants
            discoverForResource(resource, report, false);
        }
    }

 It looks like 'platform''s child resources Set is getting modified, perhaps we can just protect the returned Set but maybe it's a deeper concurrency issue.
Comment 1 Red Hat Bugzilla 2009-11-10 16:15:37 EST
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-725
Comment 2 Corey Welton 2010-08-12 15:57:24 EDT
joseph - still an issue?
Comment 3 Joseph Marques 2010-08-12 16:29:21 EDT
commit 1043bd79d2744f13a4b8db9b54fc7df9811fa619
Author: Joseph Marques <joseph@redhat.com>
Date:   Thu Aug 12 16:24:14 2010 -0400

    BZ-536368 - fix for ConcurrentModificationException during resource discovery

-----

due to the rarity of occurrence, this fix requires code inspection by a developer instead of regular QE.  mazz, can you review and sign off on this?
Comment 4 John Mazzitelli 2010-08-24 14:31:07 EDT
looks good. you can close this
Comment 8 Mike Foley 2011-07-29 15:49:14 EDT
closing, per mazz comment#4
Comment 9 Heiko W. Rupp 2013-09-02 03:17:45 EDT
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.