Bug 1401506

Summary:	[GSS](6.4.z) ConcurrentModificationException being thrown during Operation ("add") when Jboss is starting
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Shaun Appleton <sappleto>
Component:	MSC	Assignee:	Radovan Netuka <rnetuka>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jiří Bílek <jbilek>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	6.4.9	CC:	bmaxwell, david.lloyd, fnasser, jason.greene, jbilek, msochure, onagano, rnetuka, ropalka, sappleto
Target Milestone:	CR1
Target Release:	EAP 6.4.14
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-03-23 08:26:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1401452, 1412035

Description Shaun Appleton 2016-12-05 13:03:11 UTC

Description of problem:

ConcurrentModificationException being thrown during Operation ("add") when Jboss is starting


Version-Release number of selected component (if applicable):
1.1

How reproducible:
occasionally


Actual results:

2016-11-22 05:09:00,453 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 48) JBAS014612: Operation ("add") failed - address: ([("subsystem" => "security")]): java.util.ConcurrentModificationException
	at java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex(IdentityHashMap.java:732) [rt.jar:1.7.0_95]
	at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:838) [rt.jar:1.7.0_95]
	at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:832) [rt.jar:1.7.0_95]
	at java.util.IdentityHashMap.putAll(IdentityHashMap.java:503) [rt.jar:1.7.0_95]
	at org.jboss.msc.service.ServiceBuilderImpl.addListenerNoCheck(ServiceBuilderImpl.java:320) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceTargetImpl.apply(ServiceTargetImpl.java:228) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceContainerImpl.apply(ServiceContainerImpl.java:680) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:693) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:243) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2433) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:345) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1]
	at org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:1688) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.security.SecuritySubsystemRootResourceDefinition$NewSecuritySubsystemAdd.performBoottime(SecuritySubsystemRootResourceDefinition.java:139)
	at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:69) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:75) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:355) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]


Expected results:
for this not to occur


Additional info:
Fix via synchronization upstream see PR https://github.com/jboss-msc/jboss-msc/commit/aa1c87294cb9eaa3d993ac9748fc3d0ab2624b1c

Comment 2 Osamu Nagano 2016-12-15 05:53:51 UTC

Sent a PR to 1.1 branch of the upstream.
https://github.com/jboss-msc/jboss-msc/pull/39

EAP 7 is using branch 1.2 which already has the fix.

After merging the PR, backport to the product MSC and an EAP component upgrade are required.

Comment 3 Richard Opalka 2016-12-15 23:15:27 UTC

PR merged and JBoss MSC 1.1.7.Final was released so EAP can consume it.

Comment 7 Shaun Appleton 2017-02-06 11:31:31 UTC

I don't think there is a guaranteed way to reproduce this.
 
The customer only saw the issue once.

The commit that fixes the issue at https://github.com/jboss-msc/jboss-msc/commit/aa1c87294cb9eaa3d993ac9748fc3d0ab2624b1c
does not appear to have a jira associated with it so I can't find a test case used by eng.

Comment 8 Jiří Bílek 2017-02-23 10:05:20 UTC

I am not able to reproduce the issue.
According to code examination fix is present.
Verified with EAP 6.4.14.CP.CR1

Comment 9 Petr Penicka 2017-03-23 08:26:26 UTC

Released with EAP 6.4.14 on March 14 (ZIPs) and March 22 (RPMs).