Bug 901275 (JBPAPP6-1735)

Summary: [GSS] (6.3.0) Usage of vault for system-properties throws java.lang.SecurityException
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Hisanobu Okuda <hokuda>
Component: Domain ManagementAssignee: Brian Stansberry <brian.stansberry>
Status: CLOSED CURRENTRELEASE QA Contact: Josef Cacek <jcacek>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.0CC: atangrin, bkundal, brian.stansberry, cdolphy, dehort, emuckenh, fbogyai, hokuda, jawilson, jcacek, jdoyle, jsightle, kkhan, myarboro, nsurtani, pjindal, pslavice, sgilda, smendenh, smumford, tfonteyn
Target Milestone: DR2   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/JBPAPP6-1735
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previous versions of JBoss EAP 6 could fail to boot if a system property with a vault expression in its value attribute was used. This was because the system property resources in the XML configuration were processed before the security vault resources. This meant that the security vault was not available as a source for resolving expressions. Any resources using a vault attribute would result in the following error: ---- java.lang.SecurityException: JBAS013322: Vault is not initialized ---- In this release of the product, if a system property value expression fails to resolve, a second attempt is made later in the boot process, at a point after the vault (if one is configured) is installed. This resolves the issue and systems using vault expressions now boot as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-28 15:40:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1076507, 1174871, 1179497    

Description Hisanobu Okuda 2012-12-18 11:43:37 UTC
Help Desk Ticket Reference: https://na7.salesforce.com/500A000000BfC5g
Steps to Reproduce: 1. add the lines in standalone.xml:-

{code}
    <system-properties>
      <property name="javax.net.ssl.trustStore" value="/dir/truststore.jks"/>
      <property name="javax.net.ssl.trustStorePassword" value="${VAULT::blk::attr::MTBlYWUzNDEtMjI2MC00NWYzLWFiZmUtM2EyYjZkNDgyZWM4TElORV9CUkVBS215YWxpYXM=}" />
    </system-properties>
    <vault>
        <vault-option name="KEYSTORE_URL" value="/opt/jboss-eap-6.0/standalone/configuration/vault.jks"/>
        <vault-option name="KEYSTORE_PASSWORD" value="MASK-2gTQPnrWJaqrh0pURMKAOw"/>
        <vault-option name="KEYSTORE_ALIAS" value="myalias"/>
        <vault-option name="SALT" value="12345678"/>
        <vault-option name="ITERATION_COUNT" value="44"/>
        <vault-option name="ENC_FILE_DIR" value="/opt/jboss-eap-6.0/standalone/configuration/"/>
    </vault>
{code}

2. start EAP6 in standalone mode

project_key: JBPAPP6

Usage of vault for system-properties throws java.lang.SecurityException.

boot.log:-
{code}
20:35:30,267 ERROR [org.jboss.as.controller.management-operation] JBAS014612: Operation ("add") failed - address: ([("system-property" => "javax.net.ssl.trustStorePassword")]): java.lang.SecurityException: JBAS013322: Vault is not initialized
	at org.jboss.as.security.vault.RuntimeVaultReader.retrieveFromVault(RuntimeVaultReader.java:98) [jboss-as-security-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:45) [jboss-as-server-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:58) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:40) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:455) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:689) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.operations.common.SystemPropertyAddHandler.execute(SystemPropertyAddHandler.java:112) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:175) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:191) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.server.ServerService.boot(ServerService.java:295) [jboss-as-server-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.server.ServerService.boot(ServerService.java:270) [jboss-as-server-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:156) [jboss-as-controller-7.1.2.Final-redhat-1.jar:7.1.2.Final-redhat-1]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_37]

{code}

Comment 2 Brian Stansberry 2013-03-06 21:16:59 UTC
Changing how this works may have far reaching affects. I think it's too late in the 6.1 cycle to consider this.

Comment 6 Brian Stansberry 2013-06-04 21:50:34 UTC
There's a chicken and egg problem here. If we defer system property processing until after the vault is initialized, system properties cannot be used in the vault configuration itself. (It would also impact path configuration.)

Solving this would require something clever (aka hacky) like having the handler for the system property add using an OperationContext attachment to signal the handler for the vault add to store the system property. But only if the handler for the system property add can't do it due to the missing vault. Add a step to fail if the signal isn't received (i.e. there's no vault add in the set of ops.)

Comment 8 Tom Fonteyne 2013-07-09 12:25:18 UTC
Could this not be solved like in domain mode where a system property has an attribute "boot-time" ?

I have vaulted system properties set in <server-group> with "boot-time=false" and they work fine.

Comment 9 Brian Stansberry 2013-07-09 12:55:55 UTC
Tom, please send me an example config where this works. I don't see why it would, so perhaps your config will give me a nice clue!

In standalone, all system properties in standalone.xml have "boot-time=false" semantics.

Comment 10 Tom Fonteyne 2013-07-09 13:45:47 UTC
@Brian:

In domain mode, in host.xml I have the vault as per normal

Then in domain.xml, I have:

<server-groups>
  <server-group name="idpgroup" profile="default">
    <socket-binding-group ref="standard-sockets"/>
    <deployments>
       <deployment name="ShowSystemProperties.war"
                   runtime-name="ShowSystemProperties.war"/>
       </deployments>
    <system-properties>
      <property name="testvault" value="${VAULT::system::property::NjQ0Njg1MmUtYzdjMy00NjkwLWE3OWItMmM3OTc5M2I3YTM4TElORV9CUkVBS3ZhdWx0}"
              boot-time="false"/>
    </system-properties>
  </server-group>
</server-groups>

My deployment just loops over all system props and print them... and it shows the decoded "testvault" property.

The main difference that can be observed, is that "ps -auxww" will show this instance with the -Dtestvault=${...} when using boot-time=true

while with boot-time=false, the -Dtestvault=${...} does not show on ps -auxww

Comment 11 Brian Stansberry 2013-07-09 20:11:53 UTC
@Tom:

With 6.1, when server-group config includes a system property with an expression, I see it getting resolved either way, e.g.

<system-properties>
      <property name="testprop" value="${foo:hello}"
              boot-time="false"/>
</system-properties>

will result in testprop having a value of "hello" regardless of the value of boot-time.

This matches what I expect from reading the code.

This works because the HostController process actually resolves the expression before feeding it to the server process. That's arguably a bug in the boot-time=false case, since the server process should really be allowed to resolve it itself. In practice it's fine since the HC's resolution process is going to match what the server process would do anyway.

In EAP 6.0, the expression wasn't resolved no matter what the value of boot-time was. The resolution we see in 6.1 was the result of fixing a different bug.

In any case, this doesn't help with standalone, where there is no separate process to do the resolution.

Comment 12 Hisanobu Okuda 2013-09-30 04:33:44 UTC
Looks the request for inclusion to EAP6.2.0 is missed the deadline. I want to request inclusion to EAP 6.3.0.

Comment 13 Samuel Mendenhall 2014-02-24 03:33:24 UTC
Customer requesting this fix in https://c.na7.visual.force.com/apex/Case_View?id=500A000000HoP8k&sfdc.override=1 as well.

Comment 15 FIlip Bogyai 2014-03-10 14:36:38 UTC
Verified in EAP 6.3.0.DR2

Comment 21 sgilda 2014-05-12 19:34:00 UTC
Replace <programlisting> tag with tick (`) to fix Bug 1096865