Fedora Account System
Red Hat Associate
Red Hat Customer
see by ghinkle when he imported lots of perf plugin resources: 14:15:44,417 WARN [DiscoveryBossBean] Could not perform commit synchronization with agent for server [server-a-6] java.lang.RuntimeException: java.util.ConcurrentModificationException at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:944) at org.rhq.core.pc.inventory.InventoryManager.synchronizeInventory(InventoryManager.java:1729) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288) at $Proxy0.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173) Caused by: java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373) at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:384) at org.rhq.core.pc.inventory.InventoryManager.deactivateResource(InventoryManager.java:1590) at org.rhq.core.pc.inventory.InventoryManager.removeResourceAndIndicateIfScanIsNeeded(InventoryManager.java:1043) at org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018) at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380) at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:922) ... 25 more Looks like possible solution is if we change resource.getChildResources() to new ArrayList<Resource>(resource.getChildResources()) on line 1590, the problem gets fixed: private void deactivateResource(Resource resource) { this.inventoryLock.writeLock().lock(); try { ResourceContainer container = getResourceContainer(resource); if ((container != null) && (container.getResourceComponentState() == ResourceComponentState.STARTED)) { for (Resource child : resource.getChildResources()) { <<<===== deactivateResource(child); }
this involves removing uninventoried resources: org.rhq.core.pc.inventory.InventoryManager.uninventoryResource(InventoryManager.java:1018) at org.rhq.core.pc.inventory.InventoryManager.purgeObsoleteResources(InventoryManager.java:2380) I'm not sure how to replicate, but clearly it involves uninventorying a resource and perhaps then quickly committing new resources in inventory?
We should consider changing the impl of Resource.getChildResources() to: public Set<Resource> getChildResources() { return new LinkedHashSet(this.childResources); // or maybe even: return Collections.unmodifiableSet(new LinkedHashSet(this.childResources)); } And document that the returned Set should not (or can not) be modified. That way callers won't have to remember to wrap the Set before iterating it.
Pushing to ON_QA since this should be addressed by https://bugzilla.redhat.com/show_bug.cgi?id=794790
Bulk closing of some old issues