Description of problem: 20:43:30,545 WARN [DiscoveryBossBean] Could not perform commit synchronization with agent for server [EAP localhost.localdomain:1099 default] java.lang.RuntimeException: java.util.ConcurrentModificationException at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1066) at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:2004) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288) at $Proxy0.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:406) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173) Caused by: java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373) at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384) at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1854) at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1167) at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1140) at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2746) at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:1037) ... 25 more Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
I was too quick to hit "Save", so here's some more info. This happened to me with a secured EAP resource that didn't have the credentials filled in in the connection settings, i.e. it was permanently down. Also I might have debugged another thread. Still this is a race condition that shouldn't occur.
we've had several of these concurrent mod exceptions in the Inventory Manager over the years. It always had to do with things like looping through the collection of resources while another thread comes in and wants to modify the same collection. To solve, we typically copy the collection of resources prior to looping through them to allow another thread to come in and concurrent want to modify that collection. I'm sure if you search bugzilla for "rhq ConcurrentModificationException" you'll find them.
Fixed in master by copying the members of the Set to an array and then iterating the array instead of the Set - http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=commitdiff;h=4ef5591 However, there are a whole bunch more places where Resource.getChildResources() is directly iterated that need to be fixed similarly. We should create a BZ for finding all such occurrences and fixing them.
Ian, any reason this can't go ON_QA, given we have https://bugzilla.redhat.com/show_bug.cgi?id=794790 for dealing with the other instances?
Closing as this has been implemented for a long time.