Bug 786980

Summary: ConcurrentModificationException during resource deactivation on agent
Product: [Other] RHQ Project Reporter: Lukas Krejci <lkrejci>
Component: Plugin ContainerAssignee: Charles Crouch <ccrouch>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: high    
Version: 4.2CC: ccrouch, hbrock, hrupp, mazz
Target Milestone: ---   
Target Release: RHQ 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.3 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-04 05:37:51 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 782579    

Description Lukas Krejci 2012-02-02 14:54:44 EST
Description of problem:

20:43:30,545 WARN  [DiscoveryBossBean] Could not perform commit synchronization with agent for server [EAP localhost.localdomain:1099 default]
java.lang.RuntimeException: java.util.ConcurrentModificationException
        at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1066)
        at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:2004)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
        at $Proxy0.execute(Unknown Source)
        at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)
        at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
        at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.util.ConcurrentModificationException
        at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
        at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384)
        at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1854)
        at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1167)
        at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1140)
        at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2746)
        at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1037)
        ... 25 more


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Lukas Krejci 2012-02-02 14:57:43 EST
I was too quick to hit "Save", so here's some more info.

This happened to me with a secured EAP resource that didn't have the credentials filled in in the connection settings, i.e. it was permanently down.

Also I might have debugged another thread.

Still this is a race condition that shouldn't occur.
Comment 2 John Mazzitelli 2012-02-02 15:03:11 EST
we've had several of these concurrent mod exceptions in the Inventory Manager over the years. It always had to do with things like looping through the collection of resources while another thread comes in and wants to modify the same collection. To solve, we typically copy the collection of resources prior to looping through them to allow another thread to come in and concurrent want to modify that collection. I'm sure if you search bugzilla for "rhq ConcurrentModificationException" you'll find them.
Comment 3 Ian Springer 2012-02-02 17:48:52 EST
Fixed in master by copying the members of the Set to an array and then iterating the array instead of the Set - 
http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=4ef5591

However, there are a whole bunch more places where Resource.getChildResources() is directly iterated that need to be fixed similarly. We should create a BZ for finding all such occurrences and fixing them.
Comment 4 Charles Crouch 2012-03-19 18:34:18 EDT
Ian, any reason this can't go ON_QA, given we have https://bugzilla.redhat.com/show_bug.cgi?id=794790 for dealing with the other instances?
Comment 6 Heiko W. Rupp 2013-09-04 05:37:51 EDT
Closing as this has been implemented for a long time.