This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 616149 - inventory import/sync fails due to ConcurrentModificationException
inventory import/sync fails due to ConcurrentModificationException
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Plugin Container (Show other bugs)
3.0.0
All All
medium Severity medium (vote)
: ---
: ---
Assigned To: Charles Crouch
Mike Foley
:
Depends On:
Blocks: jon-sprint12-bugs
  Show dependency treegraph
 
Reported: 2010-07-19 14:35 EDT by John Mazzitelli
Modified: 2015-02-01 18:26 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-04 03:49:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description John Mazzitelli 2010-07-19 14:35:13 EDT
see by ghinkle when he imported lots of perf plugin resources:

14:15:44,417 WARN  [DiscoveryBossBean] Could not perform commit synchronization with agent for server [server-a-6]
java.lang.RuntimeException: java.util.ConcurrentModificationException
	at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:944)
	at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:1729)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
	at $Proxy0.execute(Unknown Source)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
	at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
	at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1590)
	at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1043)
	at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018)
	at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380)
	at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:922)
	... 25 more

Looks like possible solution is if we change resource.getChildResources() to new ArrayList<Resource>(resource.getChildResources()) on line 1590, the problem gets fixed:

    private void deactivateResource(Resource resource) {
        this.inventoryLock.writeLock().lock();
        try {
            ResourceContainer container = getResourceContainer(resource);
            if ((container != null) && (container.getResourceComponentState() == ResourceComponentState.STARTED)) {
                for (Resource child : resource.getChildResources()) { <<<=====
                    deactivateResource(child);
                }
Comment 1 John Mazzitelli 2010-07-19 14:54:08 EDT
this involves removing uninventoried resources:

org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018)
 at
org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380)


I'm not sure how to replicate, but clearly it involves uninventorying a
resource and perhaps then quickly committing new resources in inventory?
Comment 2 Ian Springer 2010-07-21 09:59:30 EDT
We should consider changing the impl of Resource.getChildResources() to:

    public Set<Resource> getChildResources() {
        return new LinkedHashSet(this.childResources);
        // or maybe even: return Collections.unmodifiableSet(new LinkedHashSet(this.childResources));
    }

And document that the returned Set should not (or can not) be modified. That way callers won't have to remember to wrap the Set before iterating it.
Comment 6 Charles Crouch 2012-03-19 19:01:53 EDT
Pushing to ON_QA since this should be addressed by https://bugzilla.redhat.com/show_bug.cgi?id=794790
Comment 7 Heiko W. Rupp 2013-09-04 03:49:21 EDT
Bulk closing of some old issues

Note You need to log in before you can comment on or make changes to this bug.