Description of problem: ConcurrentModificationException being thrown during Operation ("add") when Jboss is starting Version-Release number of selected component (if applicable): 1.1 How reproducible: occasionally Actual results: 2016-11-22 05:09:00,453 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 48) JBAS014612: Operation ("add") failed - address: ([("subsystem" => "security")]): java.util.ConcurrentModificationException at java.util.IdentityHashMap$IdentityHashMapIterator.nextIndex(IdentityHashMap.java:732) [rt.jar:1.7.0_95] at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:838) [rt.jar:1.7.0_95] at java.util.IdentityHashMap$EntryIterator.next(IdentityHashMap.java:832) [rt.jar:1.7.0_95] at java.util.IdentityHashMap.putAll(IdentityHashMap.java:503) [rt.jar:1.7.0_95] at org.jboss.msc.service.ServiceBuilderImpl.addListenerNoCheck(ServiceBuilderImpl.java:320) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceTargetImpl.apply(ServiceTargetImpl.java:228) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceContainerImpl.apply(ServiceContainerImpl.java:680) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:693) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:243) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2433) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:345) [jboss-msc-1.1.6.Final-redhat-1.jar:1.1.6.Final-redhat-1] at org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:1688) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.security.SecuritySubsystemRootResourceDefinition$NewSecuritySubsystemAdd.performBoottime(SecuritySubsystemRootResourceDefinition.java:139) at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:69) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:75) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:355) [jboss-as-controller-7.5.9.Final-redhat-2.jar:7.5.9.Final-redhat-2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1] Expected results: for this not to occur Additional info: Fix via synchronization upstream see PR https://github.com/jboss-msc/jboss-msc/commit/aa1c87294cb9eaa3d993ac9748fc3d0ab2624b1c
Sent a PR to 1.1 branch of the upstream. https://github.com/jboss-msc/jboss-msc/pull/39 EAP 7 is using branch 1.2 which already has the fix. After merging the PR, backport to the product MSC and an EAP component upgrade are required.
PR merged and JBoss MSC 1.1.7.Final was released so EAP can consume it.
I don't think there is a guaranteed way to reproduce this. The customer only saw the issue once. The commit that fixes the issue at https://github.com/jboss-msc/jboss-msc/commit/aa1c87294cb9eaa3d993ac9748fc3d0ab2624b1c does not appear to have a jira associated with it so I can't find a test case used by eng.
I am not able to reproduce the issue. According to code examination fix is present. Verified with EAP 6.4.14.CP.CR1
Released with EAP 6.4.14 on March 14 (ZIPs) and March 22 (RPMs).