See case 304090 for more details: Seen when setting the resource connection properties: Unable to update this resource's connection properties. java.util.NoSuchElementException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:844) at java.util.HashMap$KeyIterator.next(HashMap.java:877) at java.util.AbstractCollection.toArray(AbstractCollection.java:176) at org.rhq.core.pc.inventory.InventoryManager.fireResourceActivated(InventoryManager.java:1624) at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1153) at org.rhq.core.pc.inventory.InventoryManager.updatePluginConfiguration(InventoryManager.java:347) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414) at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy1.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165) Looks similar to http://www.netbeans.org/issues/show_bug.cgi?id=145775. Which leads to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6377302 : AbstractCollection.toArray is not safe for concurrent collections The stack above comes from a call to inventoryEventListeners.toArray in InventoryManager. Since InventoryManager can be accessed by multiple threads it looks feasible this is the issue. This would appear to be an issue in at least fireResourcesAdded, fireResourceActivated and fireResourcesRemoved methods. But we should probably take a look at other places where we do .toArray to see if they are also being done on unsynchronized collections which multiple threads could add/remove to/from.
Push to 1.4
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2131
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs. keyword: new = Tracking + FutureFeature + SubBug
making sure we're not missing any bugs in rhq_triage
I see four instances in InventoryManager that use inventoryEventListeners.toArray We just need to change it to, say, copy via constructor to a new object.
Created attachment 408131 [details] bz535436.patch
attached .patch file that should work around this bug. All adding/removing/copying performed on the listeners' set are protected by synchronized block now.
gwt branch commit ac4460c69494fc6720de5d9ba03af5c6b89ed989 we can either cherry-pick that over to master or wait until the next gwt->master merge
cherry pick into master is done commit 901a44092dd5031653ac056bd9fd925dc97ab43e Date: Wed Apr 21 13:36:48 2010 -0400
Mazz Is there anything we can do to test this is fixed?
No replication tests that I know of. All I did was (based on that sun bug report) removed calls to toArray and ensured the code is synchronized. I never saw this break, but since toArray is no longer used, that problem should go away. The synchronization code I added was pretty basic so that stuff should be good.
Closing bug as per discussion with Corey and Mazz.