Bug 616149 - inventory import/sync fails due to ConcurrentModificationException
Summary: inventory import/sync fails due to ConcurrentModificationException
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: Plugin Container   
(Show other bugs)
Version: 3.0.0
Hardware: All
OS: All
medium
medium vote
Target Milestone: ---
: ---
Assignee: Charles Crouch
QA Contact: Mike Foley
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: jon-sprint12-bugs
TreeView+ depends on / blocked
 
Reported: 2010-07-19 18:35 UTC by John Mazzitelli
Modified: 2015-02-01 23:26 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-04 07:49:21 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description John Mazzitelli 2010-07-19 18:35:13 UTC
see by ghinkle when he imported lots of perf plugin resources:

14:15:44,417 WARN  [DiscoveryBossBean] Could not perform commit synchronization with agent for server [server-a-6]
java.lang.RuntimeException: java.util.ConcurrentModificationException
	at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:944)
	at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:1729)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
	at $Proxy0.execute(Unknown Source)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
	at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
	at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1590)
	at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1043)
	at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018)
	at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380)
	at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:922)
	... 25 more

Looks like possible solution is if we change resource.getChildResources() to new ArrayList<Resource>(resource.getChildResources()) on line 1590, the problem gets fixed:

    private void deactivateResource(Resource resource) {
        this.inventoryLock.writeLock().lock();
        try {
            ResourceContainer container = getResourceContainer(resource);
            if ((container != null) && (container.getResourceComponentState() == ResourceComponentState.STARTED)) {
                for (Resource child : resource.getChildResources()) { <<<=====
                    deactivateResource(child);
                }

Comment 1 John Mazzitelli 2010-07-19 18:54:08 UTC
this involves removing uninventoried resources:

org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018)
 at
org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380)


I'm not sure how to replicate, but clearly it involves uninventorying a
resource and perhaps then quickly committing new resources in inventory?

Comment 2 Ian Springer 2010-07-21 13:59:30 UTC
We should consider changing the impl of Resource.getChildResources() to:

    public Set<Resource> getChildResources() {
        return new LinkedHashSet(this.childResources);
        // or maybe even: return Collections.unmodifiableSet(new LinkedHashSet(this.childResources));
    }

And document that the returned Set should not (or can not) be modified. That way callers won't have to remember to wrap the Set before iterating it.

Comment 6 Charles Crouch 2012-03-19 23:01:53 UTC
Pushing to ON_QA since this should be addressed by https://bugzilla.redhat.com/show_bug.cgi?id=794790

Comment 7 Heiko W. Rupp 2013-09-04 07:49:21 UTC
Bulk closing of some old issues


Note You need to log in before you can comment on or make changes to this bug.