We have some development Wildfly 10.0.0 servers running as slaves in a domain that frequently have WAR files redeployed. We have noticed that these slaves will often go offline after a redeployment of WAR files with the following stack trace: {code} 2016-05-06 05:05:51,306 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 1012) WFLYCTL0190: Step handler org.jboss.as.domain.controller.operations.deployment.DeploymentFullReplaceHandler@3f68226b for operation {"operation" => "full-replace-deployment","name" => "whatever.war","enabled" => true,"content" => [{"hash" => bytes { 0x5d, 0x12, 0x18, 0x2b, 0x1c, 0x86, 0x71, 0x27, 0x08, 0x3d, 0xf1, 0x75, 0x08, 0x29, 0xa6, 0x49, 0x1f, 0x16, 0xe8, 0x22 }}],"operation-headers" => {"access-mechanism" => "NATIVE","domain-uuid" => "802ab616-dd2c-4081-a79c-c4d54e14c384","push-to-servers" => undefined},"address" => [],"runtime-name" => undefined} at address [] failed handling operation rollback -- java.lang.NullPointerException: java.lang.NullPointerException at org.jboss.as.repository.LocalDeploymentFileRepository.deleteDeployment(LocalDeploymentFileRepository.java:59) at org.jboss.as.host.controller.RemoteDomainConnectionService$RemoteFileRepository.deleteDeployment(RemoteDomainConnectionService.java:756) at org.jboss.as.domain.controller.operations.deployment.DeploymentFullReplaceHandler$1.handleResult(DeploymentFullReplaceHandler.java:181) at org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384) at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311) at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185) at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767) at org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:753) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:680) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:217) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:247) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:185) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:138) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:134) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:81) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:157) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:153) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:153) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:363) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:472) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) {code} This error will usually only happen for 2 out of the 4 identically configured slaves, and seems to happen randomly, although frequently enough.
This fix have to be delivered with https://issues.jboss.org/browse/JBEAP-8138 (there is not 6.4.x clone) Without that it is not possible to verify the BZ. Server tries to rollback after redeploy but fails with NoSuchElementException and slaves takes ofline. Actual stack trace is: 11:46:45,539 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 73) JBAS014781: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$4@501c8696 for operation {"address" => [],"operation" => "full-replace-deployment","name" => "app.war","content" => [{"hash" => bytes { 0x91, 0x26, 0x35, 0x94, 0x93, 0x5a, 0x88, 0x95, 0xb4, 0xa5, 0x2f, 0x29, 0x31, 0xd1, 0xa4, 0x84, 0x67, 0xa4, 0xa7, 0xf5 }}],"operation-headers" => {"access-mechanism" => "NATIVE","domain-uuid" => "ad739e7b-4722-4594-957a-6abde22def0f","push-to-servers" => undefined},"persistent" => true,"owner" => undefined} at address [] failed handling operation rollback -- java.util.NoSuchElementException: No child 'name' exists: java.util.NoSuchElementException: No child 'name' exists at org.jboss.dmr.ModelValue.requireChild(ModelValue.java:369) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.dmr.ObjectModelValue.requireChild(ObjectModelValue.java:299) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.dmr.ModelNode.require(ModelNode.java:870) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.as.server.deployment.DeploymentHandlerUtil$4$1.handleResult(DeploymentHandlerUtil.java:283) [jboss-as-server-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1167) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1124) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1099) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1044) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:565) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1152) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:335) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:248) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:186) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:139) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:135) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_65] at javax.security.auth.Subject.doAs(Subject.java:360) [rt.jar:1.8.0_65] at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:83) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:158) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:154) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_65] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:154) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:364) [jboss-as-protocol-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:473) [jboss-as-protocol-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1] 11:46:45,541 INFO [org.jboss.as.controller] (ServerService Thread Pool -- 73) JBAS014774: Service status report JBAS014775: New missing/unsatisfied dependencies: service jboss.module.service."deployment.app.war".main (missing) dependents: [service jboss.deployment.unit."app.war".FIRST_MODULE_USE] service jboss.module.spec.service."deployment.app.war".main (missing) dependents: [service jboss.module.service."deployment.app.war".main] service jboss.module.spec.service."deployment.nonexistent".main (missing) dependents: [service jboss.module.resolve.phase."deployment.app.war".main.1, service jboss.module.service."deployment.app.war".main] service module.resolved.service."deployment.app.war".main (missing) dependents: [service jboss.module.service."deployment.app.war".main]
Verified with EAP 6.4.15.CP.CR2
Released on May 18 as part of EAP 6.4.15.