Bug 786980 - ConcurrentModificationException during resource deactivation on agent
ConcurrentModificationException during resource deactivation on agent
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Plugin Container (Show other bugs)
4.2
Unspecified Unspecified
high Severity high (vote)
: ---
: RHQ 4.3.0
Assigned To: Charles Crouch
Mike Foley
:
Depends On:
Blocks: jon310-sprint11/rhq44-sprint11
  Show dependency treegraph
 
Reported: 2012-02-02 14:54 EST by Lukas Krejci
Modified: 2015-02-01 18:27 EST (History)
4 users (show)

See Also:
Fixed In Version: 4.3
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-04 05:37:51 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lukas Krejci 2012-02-02 14:54:44 EST
Description of problem:

20:43:30,545 WARN  [DiscoveryBossBean] Could not perform commit synchronization with agent for server [EAP localhost.localdomain:1099 default]
java.lang.RuntimeException: java.util.ConcurrentModificationException
        at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1066)
        at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:2004)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
        at $Proxy0.execute(Unknown Source)
        at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)
        at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
        at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.util.ConcurrentModificationException
        at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
        at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
        at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1854)
        at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1167)
        at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1140)
        at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2746)
        at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1037)
        ... 25 more


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Lukas Krejci 2012-02-02 14:57:43 EST
I was too quick to hit "Save", so here's some more info.

This happened to me with a secured EAP resource that didn't have the credentials filled in in the connection settings, i.e. it was permanently down.

Also I might have debugged another thread.

Still this is a race condition that shouldn't occur.
Comment 2 John Mazzitelli 2012-02-02 15:03:11 EST
we've had several of these concurrent mod exceptions in the Inventory Manager over the years. It always had to do with things like looping through the collection of resources while another thread comes in and wants to modify the same collection. To solve, we typically copy the collection of resources prior to looping through them to allow another thread to come in and concurrent want to modify that collection. I'm sure if you search bugzilla for "rhq ConcurrentModificationException" you'll find them.
Comment 3 Ian Springer 2012-02-02 17:48:52 EST
Fixed in master by copying the members of the Set to an array and then iterating the array instead of the Set - 
http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=4ef5591

However, there are a whole bunch more places where Resource.getChildResources() is directly iterated that need to be fixed similarly. We should create a BZ for finding all such occurrences and fixing them.
Comment 4 Charles Crouch 2012-03-19 18:34:18 EDT
Ian, any reason this can't go ON_QA, given we have https://bugzilla.redhat.com/show_bug.cgi?id=794790 for dealing with the other instances?
Comment 6 Heiko W. Rupp 2013-09-04 05:37:51 EDT
Closing as this has been implemented for a long time.

Note You need to log in before you can comment on or make changes to this bug.