Bug 535436 (RHQ-2131) - hitting JDK1.5 concurrency bug in inventorymanager
Summary: hitting JDK1.5 concurrency bug in inventorymanager
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: RHQ-2131
Product: RHQ Project
Classification: Other
Component: No Component
Version: 1.2
Hardware: All
OS: All
high
medium vote
Target Milestone: ---
: ---
Assignee: John Mazzitelli
QA Contact: Rajan Timaniya
URL: http://jira.rhq-project.org/browse/RH...
Whiteboard:
Depends On:
Blocks: rhq_triage jon-sprint10-bugs
TreeView+ depends on / blocked
 
Reported: 2009-06-05 23:13 UTC by Charles Crouch
Modified: 2018-10-27 16:13 UTC (History)
6 users (show)

Fixed In Version: 1.4
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-05-20 12:45:51 UTC


Attachments (Terms of Use)
bz535436.patch (4.18 KB, application/octet-stream)
2010-04-21 17:32 UTC, John Mazzitelli
no flags Details

Description Charles Crouch 2009-06-05 23:13:00 UTC
See case 304090 for more details:

Seen when setting the resource connection properties:


Unable to update this resource's connection properties. java.util.NoSuchElementException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:844) at java.util.HashMap$KeyIterator.next(HashMap.java:877) at java.util.AbstractCollection.toArray(AbstractCollection.java:176) at org.rhq.core.pc.inventory.InventoryManager.fireResourceActivated(InventoryManager.java:1624) at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1153) at org.rhq.core.pc.inventory.InventoryManager.updatePluginConfiguration(InventoryManager.java:347) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414) at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy1.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)

Looks similar to http://www.netbeans.org/issues/show_bug.cgi?id=145775. Which leads to

 http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6377302  : AbstractCollection.toArray is not safe for concurrent collections

The stack above comes from a call to inventoryEventListeners.toArray in InventoryManager. Since InventoryManager can be accessed by multiple threads it looks feasible this is the issue. This would appear to be an issue in at least fireResourcesAdded, fireResourceActivated and fireResourcesRemoved methods. But we should probably take a look at other places where we do .toArray to see if they are also being done on unsynchronized collections which multiple threads could add/remove to/from.


Comment 1 Corey Welton 2009-08-05 18:03:16 UTC
Push to  1.4

Comment 2 Red Hat Bugzilla 2009-11-10 20:58:27 UTC
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2131


Comment 3 wes hayutin 2010-02-16 16:54:39 UTC
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug

Comment 4 wes hayutin 2010-02-16 17:00:06 UTC
making sure we're not missing any bugs in rhq_triage

Comment 5 John Mazzitelli 2010-04-21 16:10:20 UTC
I see four instances in InventoryManager that use

inventoryEventListeners.toArray

We just need to change it to, say, copy via constructor to a new object.

Comment 6 John Mazzitelli 2010-04-21 17:32:15 UTC
Created attachment 408131 [details]
bz535436.patch

Comment 7 John Mazzitelli 2010-04-21 17:34:35 UTC
attached .patch file that should work around this bug. All adding/removing/copying performed on the listeners' set are protected by synchronized block now.

Comment 8 John Mazzitelli 2010-04-21 17:38:27 UTC
gwt branch commit ac4460c69494fc6720de5d9ba03af5c6b89ed989
we can either cherry-pick that over to master or wait until the next gwt->master merge

Comment 9 John Mazzitelli 2010-04-27 16:59:19 UTC
cherry pick into master is done

commit 901a44092dd5031653ac056bd9fd925dc97ab43e
Date:   Wed Apr 21 13:36:48 2010 -0400

Comment 10 Charles Crouch 2010-05-13 19:28:31 UTC
Mazz
Is there anything we can do to test this is fixed?

Comment 11 John Mazzitelli 2010-05-13 19:40:41 UTC
No replication tests that I know of. All I did was (based on that sun bug report) removed calls to toArray and ensured the code is synchronized. I never saw this break, but since toArray is no longer used, that problem should go away. The synchronization code I added was pretty basic so that stuff should be good.

Comment 12 Rajan Timaniya 2010-05-20 12:45:51 UTC
Closing bug as per discussion with Corey and Mazz.


Note You need to log in before you can comment on or make changes to this bug.