Bug 1025185

Summary:	Remoting subsystem: Concurrent modification exception during server shutdown
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Jitka Kozana <jkudrnac>
Component:	Remoting	Assignee:	Brian Stansberry <brian.stansberry>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jitka Kozana <jkudrnac>
Severity:	high	Docs Contact:	Russell Dickenson <rdickens>
Priority:	unspecified
Version:	6.2.0	CC:	brian.stansberry, dandread, jawilson, jdoyle, jkudrnac, lcosti, myarboro, rsvoboda
Target Milestone:	CR1
Target Release:	EAP 6.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	A service in the remoting subsystem would iterate over a collection in a non-thread-safe manner. As a result of this issue, a `ConcurrentModificationException` exception would occasionally be thrown during server shutdown. Other than this exception, the shutdown would complete normally. ---- WARN [org.jboss.msc.service.fail] (MSC service thread 1-55) MSC000004: Failure during stop of service jboss.remoting.endpoint.management.channel.management: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [rt.jar:1.6.0_45] at java.util.HashMap$KeyIterator.next(HashMap.java:828) [rt.jar:1.6.0_45] at java.util.AbstractCollection.addAll(AbstractCollection.java:305) [rt.jar:1.6.0_45] at java.util.HashSet.<init>(HashSet.java:100) [rt.jar:1.6.0_45] at org.jboss.as.remoting.AbstractChannelOpenListenerService.stop(AbstractChannelOpenListenerService.java:123) ... ---- This issue has been fixed in this release of JBoss EAP 6. Iteration over the collection is now done in a thread-safe manner, with the collection object's monitor held by the iterating thread. As a result, `ConcurrentModificationException` exceptions are no longer thrown.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-15 16:20:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jitka Kozana 2013-10-31 08:06:37 UTC

The following exception was sometimes logged during server shutdown.
Environment: Oracle JDK6, Solaris 10, sparc 64
EAP 6.2.0.ER7.

[0m[33m01:46:29,696 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-55) MSC000004: Failure during stop of service jboss.remoting.endpoint.management.channel.management: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [rt.jar:1.6.0_45]
	at java.util.HashMap$KeyIterator.next(HashMap.java:828) [rt.jar:1.6.0_45]
	at java.util.AbstractCollection.addAll(AbstractCollection.java:305) [rt.jar:1.6.0_45]
	at java.util.HashSet.<init>(HashSet.java:100) [rt.jar:1.6.0_45]
	at org.jboss.as.remoting.AbstractChannelOpenListenerService.stop(AbstractChannelOpenListenerService.java:123)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

See the server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-acceptance-startup-solaris/jdk=java16_default,label=sol10_sparc64/24/artifact/target/manu-reports/units/manu.testsuite.eap.eap6.acceptance.startup.CleanStartupAcceptanceEap6/stdout-jboss-eap-6.2_start-serverType=standalone_profile=standalone-full.0.log

Comment 1 Rostislav Svoboda 2013-10-31 12:08:49 UTC

https://github.com/jbossas/jboss-eap/blob/6.x/remoting/src/main/java/org/jboss/as/remoting/AbstractChannelOpenListenerService.java#L123

final Set<ManagementChannelInitialization.ManagementChannelShutdownHandle> handles = new HashSet<ManagementChannelInitialization.ManagementChannelShutdownHandle>(this.handles);

In registrationTerminated method there is lock on 'handles', it should be probably added to line 123 too.
 
https://github.com/jbossas/jboss-eap/blob/6.x/remoting/src/main/java/org/jboss/as/remoting/AbstractChannelOpenListenerService.java#L185

synchronized (handles) {
    // Copy off the set to avoid ConcurrentModificationException
    copy = new HashSet<ManagementChannelInitialization.ManagementChannelShutdownHandle>(handles);
}

Comment 2 John Doyle 2013-10-31 12:16:33 UTC

We should try to fix this but I would not block the release for an error on shutdown.

Comment 4 David M. Lloyd 2013-10-31 12:28:51 UTC

Updated component.  The exception appears in the Remoting subsystem, not MSC.

Comment 10 Jitka Kozana 2013-11-11 13:00:42 UTC

This issue was verified using the 6.2.0.CR1 preview bits.