Bug 535436 (RHQ-2131)

Summary: hitting JDK1.5 concurrency bug in inventorymanager
Product: [Other] RHQ Project Reporter: Charles Crouch <ccrouch>
Component: No ComponentAssignee: John Mazzitelli <mazz>
Status: CLOSED CURRENTRELEASE QA Contact: Rajan Timaniya <rtimaniy>
Severity: medium Docs Contact:
Priority: high    
Version: 1.2CC: cwelton, hbrock, jshaughn, mazz, rtimaniy, tao
Target Milestone: ---Keywords: SubBug
Target Release: ---   
Hardware: All   
OS: All   
URL: http://jira.rhq-project.org/browse/RHQ-2131
Whiteboard:
Fixed In Version: 1.4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-05-20 12:45:51 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 565628, 591531    
Attachments:
Description Flags
bz535436.patch none

Description Charles Crouch 2009-06-05 23:13:00 UTC 
See case 304090 for more details:

Seen when setting the resource connection properties:


Unable to update this resource's connection properties. java.util.NoSuchElementException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:844) at java.util.HashMap$KeyIterator.next(HashMap.java:877) at java.util.AbstractCollection.toArray(AbstractCollection.java:176) at org.rhq.core.pc.inventory.InventoryManager.fireResourceActivated(InventoryManager.java:1624) at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1153) at org.rhq.core.pc.inventory.InventoryManager.updatePluginConfiguration(InventoryManager.java:347) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414) at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy1.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)

Looks similar to http://www.netbeans.org/issues/show_bug.cgi?id=145775. Which leads to

 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6377302  : AbstractCollection.toArray is not safe for concurrent collections

The stack above comes from a call to inventoryEventListeners.toArray in InventoryManager. Since InventoryManager can be accessed by multiple threads it looks feasible this is the issue. This would appear to be an issue in at least fireResourcesAdded, fireResourceActivated and fireResourcesRemoved methods. But we should probably take a look at other places where we do .toArray to see if they are also being done on unsynchronized collections which multiple threads could add/remove to/from.
Comment 1 Corey Welton 2009-08-05 18:03:16 UTC 
Push to  1.4
Comment 2 Red Hat Bugzilla 2009-11-10 20:58:27 UTC 
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2131
Comment 3 wes hayutin 2010-02-16 16:54:39 UTC 
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug
Comment 4 wes hayutin 2010-02-16 17:00:06 UTC 
making sure we're not missing any bugs in rhq_triage
Comment 5 John Mazzitelli 2010-04-21 16:10:20 UTC 
I see four instances in InventoryManager that use

inventoryEventListeners.toArray

We just need to change it to, say, copy via constructor to a new object.
Comment 6 John Mazzitelli 2010-04-21 17:32:15 UTC 
Created attachment 408131 [details]
bz535436.patch
Comment 7 John Mazzitelli 2010-04-21 17:34:35 UTC 
attached .patch file that should work around this bug. All adding/removing/copying performed on the listeners' set are protected by synchronized block now.
Comment 8 John Mazzitelli 2010-04-21 17:38:27 UTC 
gwt branch commit ac4460c69494fc6720de5d9ba03af5c6b89ed989
we can either cherry-pick that over to master or wait until the next gwt->master merge
Comment 9 John Mazzitelli 2010-04-27 16:59:19 UTC 
cherry pick into master is done

commit 901a44092dd5031653ac056bd9fd925dc97ab43e
Date:   Wed Apr 21 13:36:48 2010 -0400
Comment 10 Charles Crouch 2010-05-13 19:28:31 UTC 
Mazz
Is there anything we can do to test this is fixed?
Comment 11 John Mazzitelli 2010-05-13 19:40:41 UTC 
No replication tests that I know of. All I did was (based on that sun bug report) removed calls to toArray and ensured the code is synchronized. I never saw this break, but since toArray is no longer used, that problem should go away. The synchronization code I added was pretty basic so that stuff should be good.
Comment 12 Rajan Timaniya 2010-05-20 12:45:51 UTC 
Closing bug as per discussion with Corey and Mazz.