Bug 1406562
| Summary: | [GSS](6.4.z) NPE thrown during application redeployment, slaves taken offline | ||
|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Brad Maxwell <bmaxwell> |
| Component: | Domain Management | Assignee: | Radovan Netuka <rnetuka> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Martin Simka <msimka> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.4.12 | CC: | brian.stansberry, cdewolf, dandread, jbilek, msochure, pjurak, rnetuka |
| Target Milestone: | CR1 | ||
| Target Release: | EAP 6.4.15 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-05-19 08:07:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1419353 | ||
This fix have to be delivered with https://issues.jboss.org/browse/JBEAP-8138 (there is not 6.4.x clone) Without that it is not possible to verify the BZ. Server tries to rollback after redeploy but fails with NoSuchElementException and slaves takes ofline. Actual stack trace is: 11:46:45,539 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 73) JBAS014781: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$4@501c8696 for operation {"address" => [],"operation" => "full-replace-deployment","name" => "app.war","content" => [{"hash" => bytes { 0x91, 0x26, 0x35, 0x94, 0x93, 0x5a, 0x88, 0x95, 0xb4, 0xa5, 0x2f, 0x29, 0x31, 0xd1, 0xa4, 0x84, 0x67, 0xa4, 0xa7, 0xf5 }}],"operation-headers" => {"access-mechanism" => "NATIVE","domain-uuid" => "ad739e7b-4722-4594-957a-6abde22def0f","push-to-servers" => undefined},"persistent" => true,"owner" => undefined} at address [] failed handling operation rollback -- java.util.NoSuchElementException: No child 'name' exists: java.util.NoSuchElementException: No child 'name' exists at org.jboss.dmr.ModelValue.requireChild(ModelValue.java:369) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.dmr.ObjectModelValue.requireChild(ObjectModelValue.java:299) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.dmr.ModelNode.require(ModelNode.java:870) [jboss-dmr-1.2.2.Final-redhat-1.jar:1.2.2.Final-redhat-1] at org.jboss.as.server.deployment.DeploymentHandlerUtil$4$1.handleResult(DeploymentHandlerUtil.java:283) [jboss-as-server-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1167) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1124) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1099) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1044) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:565) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1152) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:335) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:204) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:248) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:186) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:139) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:135) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_65] at javax.security.auth.Subject.doAs(Subject.java:360) [rt.jar:1.8.0_65] at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:83) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:158) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:154) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_65] at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:154) [jboss-as-controller-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:364) [jboss-as-protocol-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:473) [jboss-as-protocol-7.5.14.Final-redhat-1.jar:7.5.14.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1] 11:46:45,541 INFO [org.jboss.as.controller] (ServerService Thread Pool -- 73) JBAS014774: Service status report JBAS014775: New missing/unsatisfied dependencies: service jboss.module.service."deployment.app.war".main (missing) dependents: [service jboss.deployment.unit."app.war".FIRST_MODULE_USE] service jboss.module.spec.service."deployment.app.war".main (missing) dependents: [service jboss.module.service."deployment.app.war".main] service jboss.module.spec.service."deployment.nonexistent".main (missing) dependents: [service jboss.module.resolve.phase."deployment.app.war".main.1, service jboss.module.service."deployment.app.war".main] service module.resolved.service."deployment.app.war".main (missing) dependents: [service jboss.module.service."deployment.app.war".main] Verified with EAP 6.4.15.CP.CR2 Released on May 18 as part of EAP 6.4.15. |
We have some development Wildfly 10.0.0 servers running as slaves in a domain that frequently have WAR files redeployed. We have noticed that these slaves will often go offline after a redeployment of WAR files with the following stack trace: {code} 2016-05-06 05:05:51,306 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 1012) WFLYCTL0190: Step handler org.jboss.as.domain.controller.operations.deployment.DeploymentFullReplaceHandler@3f68226b for operation {"operation" => "full-replace-deployment","name" => "whatever.war","enabled" => true,"content" => [{"hash" => bytes { 0x5d, 0x12, 0x18, 0x2b, 0x1c, 0x86, 0x71, 0x27, 0x08, 0x3d, 0xf1, 0x75, 0x08, 0x29, 0xa6, 0x49, 0x1f, 0x16, 0xe8, 0x22 }}],"operation-headers" => {"access-mechanism" => "NATIVE","domain-uuid" => "802ab616-dd2c-4081-a79c-c4d54e14c384","push-to-servers" => undefined},"address" => [],"runtime-name" => undefined} at address [] failed handling operation rollback -- java.lang.NullPointerException: java.lang.NullPointerException at org.jboss.as.repository.LocalDeploymentFileRepository.deleteDeployment(LocalDeploymentFileRepository.java:59) at org.jboss.as.host.controller.RemoteDomainConnectionService$RemoteFileRepository.deleteDeployment(RemoteDomainConnectionService.java:756) at org.jboss.as.domain.controller.operations.deployment.DeploymentFullReplaceHandler$1.handleResult(DeploymentFullReplaceHandler.java:181) at org.jboss.as.controller.AbstractOperationContext$Step.invokeResultHandler(AbstractOperationContext.java:1384) at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1366) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328) at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1311) at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185) at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:767) at org.jboss.as.controller.AbstractOperationContext.executeDoneStage(AbstractOperationContext.java:753) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:680) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1344) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:392) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:217) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler.internalExecute(TransactionalProtocolOperationHandler.java:247) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:185) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:138) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.run(TransactionalProtocolOperationHandler.java:134) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:81) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:157) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2$1.run(TransactionalProtocolOperationHandler.java:153) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$2.execute(TransactionalProtocolOperationHandler.java:153) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$ManagementRequestContextImpl$1.doExecute(AbstractMessageHandler.java:363) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:472) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) {code} This error will usually only happen for 2 out of the 4 identically configured slaves, and seems to happen randomly, although frequently enough.