Bug 1025185 - Remoting subsystem: Concurrent modification exception during server shutdown
Remoting subsystem: Concurrent modification exception during server shutdown
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Remoting (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high
: CR1
: EAP 6.2.0
Assigned To: Brian Stansberry
Jitka Kozana
Russell Dickenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-31 04:06 EDT by Jitka Kozana
Modified: 2013-12-15 11:20 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
A service in the remoting subsystem would iterate over a collection in a non-thread-safe manner. As a result of this issue, a `ConcurrentModificationException` exception would occasionally be thrown during server shutdown. Other than this exception, the shutdown would complete normally. ---- WARN [org.jboss.msc.service.fail] (MSC service thread 1-55) MSC000004: Failure during stop of service jboss.remoting.endpoint.management.channel.management: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [rt.jar:1.6.0_45] at java.util.HashMap$KeyIterator.next(HashMap.java:828) [rt.jar:1.6.0_45] at java.util.AbstractCollection.addAll(AbstractCollection.java:305) [rt.jar:1.6.0_45] at java.util.HashSet.<init>(HashSet.java:100) [rt.jar:1.6.0_45] at org.jboss.as.remoting.AbstractChannelOpenListenerService.stop(AbstractChannelOpenListenerService.java:123) ... ---- This issue has been fixed in this release of JBoss EAP 6. Iteration over the collection is now done in a thread-safe manner, with the collection object's monitor held by the iterating thread. As a result, `ConcurrentModificationException` exceptions are no longer thrown.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-15 11:20:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jitka Kozana 2013-10-31 04:06:37 EDT
The following exception was sometimes logged during server shutdown.
Environment: Oracle JDK6, Solaris 10, sparc 64
EAP 6.2.0.ER7.

[0m[33m01:46:29,696 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-55) MSC000004: Failure during stop of service jboss.remoting.endpoint.management.channel.management: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [rt.jar:1.6.0_45]
	at java.util.HashMap$KeyIterator.next(HashMap.java:828) [rt.jar:1.6.0_45]
	at java.util.AbstractCollection.addAll(AbstractCollection.java:305) [rt.jar:1.6.0_45]
	at java.util.HashSet.<init>(HashSet.java:100) [rt.jar:1.6.0_45]
	at org.jboss.as.remoting.AbstractChannelOpenListenerService.stop(AbstractChannelOpenListenerService.java:123)
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

See the server log here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-manu-acceptance-startup-solaris/jdk=java16_default,label=sol10_sparc64/24/artifact/target/manu-reports/units/manu.testsuite.eap.eap6.acceptance.startup.CleanStartupAcceptanceEap6/stdout-jboss-eap-6.2_start-serverType=standalone_profile=standalone-full.0.log
Comment 1 Rostislav Svoboda 2013-10-31 08:08:49 EDT
https://github.com/jbossas/jboss-eap/blob/6.x/remoting/src/main/java/org/jboss/as/remoting/AbstractChannelOpenListenerService.java#L123

final Set<ManagementChannelInitialization.ManagementChannelShutdownHandle> handles = new HashSet<ManagementChannelInitialization.ManagementChannelShutdownHandle>(this.handles);

In registrationTerminated method there is lock on 'handles', it should be probably added to line 123 too.
 
https://github.com/jbossas/jboss-eap/blob/6.x/remoting/src/main/java/org/jboss/as/remoting/AbstractChannelOpenListenerService.java#L185

synchronized (handles) {
    // Copy off the set to avoid ConcurrentModificationException
    copy = new HashSet<ManagementChannelInitialization.ManagementChannelShutdownHandle>(handles);
}
Comment 2 John Doyle 2013-10-31 08:16:33 EDT
We should try to fix this but I would not block the release for an error on shutdown.
Comment 4 David M. Lloyd 2013-10-31 08:28:51 EDT
Updated component.  The exception appears in the Remoting subsystem, not MSC.
Comment 10 Jitka Kozana 2013-11-11 08:00:42 EST
This issue was verified using the 6.2.0.CR1 preview bits.

Note You need to log in before you can comment on or make changes to this bug.