Bug 1299102

Summary: [GSS](6.4.z) NPE when calling write-attribute on the keystore-password in SecurityRealmAddHandler.java:643
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Brad Maxwell <bmaxwell>
Component: SecurityAssignee: Radovan Netuka <rnetuka>
Status: CLOSED CURRENTRELEASE QA Contact: Josef Cacek <jcacek>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.5CC: anmiller, bbaranow, bdawidow, darran.lofthouse, ihradek, jtruhlar, ppalaga, pskopek, rnetuka
Target Milestone: CR1   
Target Release: EAP 6.4.12   
Hardware: Unspecified   
OS: Unspecified   
URL: https://github.com/jbossas/jboss-eap/pull/2746
Whiteboard: eap6412-proposed
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-17 13:10:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1375585    

Description Brad Maxwell 2016-01-15 23:38:56 UTC
Description of problem:

Setting up a security-realm using ssl with keystore as per:

https://access.redhat.com/documentation/en/red-hat-jboss-enterprise-application-platform/version-6.4/how-to-configure-server-security/#one_create_a_keystore_to_secure_the_management_console

Then if we try to change the keystore-password with this CLI command:

/core-service=management/security-realm=ManagementRealmHTTPS/server-identity=ssl/:write-attribute(name=keystore-password,value=newPassword)

It will fail with a NullPointerException :

[standalone@localhost:9999 /] /core-service=management/security-realm=ManagementRealmHTTPS/server-identity=ssl/:write-attribute(name=keystore-password,value=password1)
{
    "outcome" => "failed",
    "failure-description" => "JBAS014749: Operation handler failed: java.lang.NullPointerException",
    "rolled-back" => true
}

Invoking the command again will succeed and the standalone.xml will be updated correctly.

Workaround it invoke the command again.

[standalone@localhost:9999 /] /core-service=management/security-realm=ManagementRealmHTTPS/server-identity=ssl/:write-attribute(name=keystore-password,value=password1)
{"outcome" => "success"}


The server side log is below:

17:36:18,361 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 2) JBAS014612: Operation ("write-attribute") failed - address: ([
    ("core-service" => "management"),
    ("security-realm" => "ManagementRealmHTTPS"),
    ("server-identity" => "ssl")
]): java.lang.NullPointerException
	at org.jboss.as.domain.management.security.SecurityRealmAddHandler.addSSLServices(SecurityRealmAddHandler.java:643) [jboss-as-domain-management-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.domain.management.security.SecurityRealmAddHandler.installServices(SecurityRealmAddHandler.java:201) [jboss-as-domain-management-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.domain.management.security.SecurityRealmChildWriteAttributeHandler.recreateParentService(SecurityRealmChildWriteAttributeHandler.java:77) [jboss-as-domain-management-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.RestartParentWriteAttributeHandler.applyUpdateToRuntime(RestartParentWriteAttributeHandler.java:94) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.AbstractWriteAttributeHandler$1.execute(AbstractWriteAttributeHandler.java:135) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:208) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:135) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:157) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at javax.security.auth.Subject.doAs(Subject.java:396) [rt.jar:1.6.0_45]
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:153) [jboss-as-controller-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298) [jboss-as-protocol-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537) [jboss-as-protocol-7.5.5.Final-redhat-3.jar:7.5.5.Final-redhat-3]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]

Comment 9 Radovan Netuka 2016-04-06 12:14:44 UTC
Looking at the code, there is definitely something wrong. 

Just see SecurityRealmChildWriteAttributeHandler#recreateParentService. This method calls SecurityRealmAddHandler.INSTANCE.installServices with newControllers hardcoded as null.

Let's see (simplified) call stack

1. SecurityRealmChildWriteAttributeHandler#recreateParentService
2. SecurityRealmAddHandler#installServices
3. SecurityRealmAddHandler#addSSLServices

In [1], the value is passed as hardcoded null. It is not overwritten anywhere inside [2], so the variable remains set to null and is passed to [3]. Here, the code tries to call a method on this variable, thus a NullPointerException.

Comment 10 Radovan Netuka 2016-04-08 08:57:44 UTC
I've tried both fixes (as baranowb) suggested above. Unfortunatelly, the application hangs for 5 minutes after which the result is "failed" anyway.

By running setup.sh

When setting up incorrect password and performing reload (last commands of setup.sh), if you then try to change the keystore password again, it triggers restartServices=true and then removeServices inside
RestartParentWriteAttributeHandler#applyUpdateToRuntime

This subsequently calls awaitContainerStability is called with timeout 300000 (= 5 mins) in OperationContextImpl#ensureWriteLockForRuntime. But at this point, the container has incorrect keystore password set in standalone.xml, thus is not stable and the operation fails after the timeout.

Nor the remove nor writing the new password is performed.

Comment 11 Radovan Netuka 2016-04-12 13:41:04 UTC
The change above fixes the NullPointerException.

The third setting of keystore password still fails (now after 5 min timeout), but this is caused by the fact that the container has invalid configuration (wrong keystore password) and every CLI operation first waits for container validity. Since it's not valid, the operation fails with restart-required message. After container restart, the keystore password can be changed without problems.

Comment 12 Ivo Hradek 2016-11-23 15:02:09 UTC
Verified with EAP 6.4.12.CP.CR1;

Note: Regarding the timeout, changing the password is still working as @Radovan explained in comment 11.

Comment 13 Petr Penicka 2017-01-17 13:10:39 UTC
Retroactively bulk-closing issues from released EAP 6.4 cummulative patches.