Bug 1070262 - (JON3-40, PRODMGT-411) Adding keystore/trusstore password encryption/obfuscation in rhq-server.properties, agent-configuration.xml files and java prefs
Adding keystore/trusstore password encryption/obfuscation in rhq-server.prope...
Status: CLOSED CURRENTRELEASE
Product: JBoss Operations Network
Classification: JBoss
Component: Agent, Core Server, Installer (Show other bugs)
JON 3.3.0
Unspecified Unspecified
unspecified Severity urgent
: CR02
: JON 3.3.0
Assigned To: Stefan Negrea
Garik Khachikyan
:
: 577239 1070279 (view as bug list)
Depends On: 577239 1128929
Blocks: 1022289
  Show dependency treegraph
 
Reported: 2014-02-26 09:01 EST by Heiko W. Rupp
Modified: 2015-08-11 15:46 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-12-11 08:59:40 EST
Type: Enhancement
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
prefs.xml (with that property unmasked) (6.57 KB, application/xml)
2014-11-04 09:40 EST, Garik Khachikyan
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker JON3-40 Major Verified Adding keystore/trusstore password encryption/obfuscation in rhq-server.properties and agent-configuration.xml files [PR... 2015-09-16 13:28:28 EDT
JBoss Issue Tracker PRODMGT-411 Major Closed Adding keystore/trusstore password encryption/obfuscation in rhq-server.properties and agent-configuration.xml files 2015-09-16 13:28:28 EDT

  None (edit)
Description Heiko W. Rupp 2014-02-26 09:01:52 EST
Currently, keystore and truststore passwords are stored in the rhq-server.properties, agent-configuration.xml files and java preferences in plaintext. We should implement a new feature that will use encrypted values for keystore and truststore passwords.

See also Bug 823965 that deals with updates - we need to encrypt/obfuscate on the fly if requested be the user (or if the user does not explicitly opt-out)
Comment 1 Larry O'Leary 2014-03-06 00:50:45 EST
*** Bug 1070279 has been marked as a duplicate of this bug. ***
Comment 2 John Mazzitelli 2014-03-24 16:49:02 EDT
it looks like this is going to require the user to configure the vault - because the secure web connector settings requires the values in standalone.xml. It can't be obfuscated in the way we do it in rhq-server.properties - if you don't want to put the key/truststore pw in clear text, you have to use the vault.

Perhaps we can automate the creation of a vault for the sole use of our secure web connector.
Comment 4 John Mazzitelli 2014-03-28 15:58:32 EDT
(In reply to John Mazzitelli from comment #2)
> it looks like this is going to require the user to configure the vault -
> because the secure web connector settings requires the values in
> standalone.xml. It can't be obfuscated in the way we do it in
> rhq-server.properties - if you don't want to put the key/truststore pw in
> clear text, you have to use the vault.
> 
> Perhaps we can automate the creation of a vault for the sole use of our
> secure web connector.

I just realized that this is also asking for obfuscation of agent-configuration.xml - which is agent-side only and thus doesn't even involve EAP so we can't use the vault for that.

In addition, the vault needs to be provided a keystore itself - and thus needs a keystore password that is then obfuscated to configure the vault. This would require manual steps since its the user providing the keystore password and running vault.sh to mask that password. So we would be just kicking the can down the road - to automate something like this we still need to know the keystore password in some fashion (even if its masked).

Still investigating a way to obfuscate these values on both server AND agent and how not to require manual user intervention but allow for automatic setup.

The only other alternative is just to be able to support the vault that the user manually sets up (in other words, we would just provide the steps on how to do this in the documentation, and support vault masked-passwords in rhq-server.properties). This still doesn't solve the agent problem since it doesn't have access to the EAP vault anyway.
Comment 5 John Mazzitelli 2014-05-28 08:00:34 EDT
*** Bug 577239 has been marked as a duplicate of this bug. ***
Comment 6 Stefan Negrea 2014-07-23 12:55:28 EDT
The obfuscation feature for agent and server for sensitive properties has been merged in master. The work was done via PR in github. The feature is backwards compatible with existing installations that do not have protection for sensitive fields.

PR link:
https://github.com/rhq-project/rhq/pull/75


A community wiki with the documenatin will be published shortly.
Comment 7 Stefan Negrea 2014-07-23 17:24:21 EDT
Last feature commit in master: https://github.com/rhq-project/rhq/commit/b7b7ed9b20fe86fb859a69b3b9e4239ea5345d80
Comment 8 JBoss JIRA Server 2014-07-24 02:13:08 EDT
Heiko Rupp <hrupp@redhat.com> updated the status of jira JON3-40 to Resolved
Comment 9 Simeon Pinder 2014-07-31 11:51:51 EDT
Moving to ON_QA as available to test with brew build of DR01: https://brewweb.devel.redhat.com//buildinfo?buildID=373993
Comment 11 Heiko W. Rupp 2014-08-12 04:13:53 EDT
This is the community wiki: https://docs.jboss.org/author/display/RHQ/Protect+Sensitive+Server+And+Agent+Configuration
Comment 12 Heiko W. Rupp 2014-08-12 09:14:24 EDT
As written in the BZ 1128929, I think the use case described in it is not valid and should not fail this BZ.
Comment 13 Viet Nguyen 2014-08-13 10:38:12 EDT
Dev, please revise the community wiki, Section 1.b.ii. Otherwise the use case in BZ 1128929 is valid. Thanks.

test run: 
https://tcms.engineering.redhat.com/run/167010/?from_plan=14896
Comment 14 JBoss JIRA Server 2014-08-15 19:00:44 EDT
mfoley user <mfoley@redhat.com> updated the status of jira JON3-40 to Reopened
Comment 15 Stefan Negrea 2014-08-26 17:57:25 EDT
Testing for this feature can resume since blocking bug has been resolved.
Comment 16 Simeon Pinder 2014-09-03 16:31:38 EDT
Moving to ON_QA as available for test with the following brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=381194
Comment 18 Garik Khachikyan 2014-09-22 06:34:38 EDT
# COMMENT

Scenario 1:
Check following commands on installed JON 3.3 ER03
===
grep -E "truststore|keystore" ~/.java/.userPrefs/rhq-agent/default/prefs.xml
grep -E "truststore|keystore" ~/current-jon/bin/rhq-server.properties # server
grep -E "truststore|keystore" ~/.java/.userPrefs/rhq-server/default/prefs.xml # server
grep -E "truststore|keystore" ~/rhq-agent/conf/agent-configuration.xml
===

no plain text found on outputs.
Comment 19 Garik Khachikyan 2014-09-22 06:37:32 EDT
Scenario 2:
Check command outputs above on JON 3.2.0 GA agent upgraded to JON 3.3 ER03
Comment 20 Garik Khachikyan 2014-09-22 06:41:11 EDT
Scenario 3:
Check command outputs above on JON 3.2.0 GA server upgraded to JON 3.3 ER03
Comment 21 Garik Khachikyan 2014-09-22 09:31:01 EDT
# REOPEN

doing upgrade of a server from 3.2.0.GA to 3.3.ER03 fails to mask those sensitive info:

scenario to reproduce:
1. install JON 3.2.0.GA
2. stop all services
3. take the 3.3 ER03 and unzip
4. run `./rhqctl upgrade --from-server-dir=/home/hudson/jon-server-3.2.0.GA`
5. refer to: grep -E "truststore|keystore" ~/jon-server-3.3.0.ER03/bin/rhq-server.properties

they are unmasked plain-text keystore info, etc.
Comment 22 Stefan Negrea 2014-09-30 16:49:29 EDT
release/jon3.3.x branch commits:

commit 2065651c5e7dfd7d0b390e65cf4e82928664b6e1
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Mon Sep 29 09:15:24 2014 -0500

    [BZ 1070262] Few more changes to the agent config update to allow the agent to continue with the startup proce
    (cherry picked from commit 99d8ae11fc1cea26e4b3a116a127757b6be68846)
    
    Signed-off-by: John Mazzitelli <mazz@redhat.com>

commit c233a8d7c6134cd9a76bbe1eef608dce84dbda74
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Fri Sep 26 19:40:47 2014 -0500

    [BZ 1070262] Obfuscate properties in agent-configuration.xml when the file originates from older agents.
    (cherry picked from commit 67f780ae17aa76164006a1af405cb12382952822)
    
    Signed-off-by: John Mazzitelli <mazz@redhat.com>

commit 27ba7a32aab8b2e99c8bd53057ccbc6ebe427837
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Fri Sep 26 15:13:26 2014 -0500

    [BZ 1070262] Snip superfluous "public" access modifier.
    (cherry picked from commit 3ff036a33887fdc3cfa4463f2e1179d183e0c5e2)
    
    Signed-off-by: John Mazzitelli <mazz@redhat.com>

commit e3f67686ed6e4f2d0c1e9263bed51e32c5ff31d9
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Fri Sep 26 11:19:59 2014 -0500

    [BZ 1070262] Updates to the configuration upgrade code for the server and the agent.
    
    This commit resolves the upgrade obfuscation for rhq-server.properties file and some minor tweaks for the agen
    (cherry picked from commit 4d27730aee50a12910e754157ce6e6d13556693a)
    
    Signed-off-by: John Mazzitelli <mazz@redhat.com>
Comment 23 Stefan Negrea 2014-09-30 16:51:00 EDT
Garik, please re-test both the server and agent upgrade process. On the agent upgrade you need to enable a few obfuscuted properties. By default they are commented out (not active) in the agent-configuration.xml.
Comment 24 Simeon Pinder 2014-10-01 17:33:07 EDT
Moving to ON_QA as available for test with build:
https://brewweb.devel.redhat.com/buildinfo?buildID=388959
Comment 25 Garik Khachikyan 2014-10-15 04:42:31 EDT
# REOPEN

it is not working correct in case of agent-side "~/.java/.userPrefs/rhq-agent/default/prefs.xml"

I do have fresh/latest JON 3.3 ER04 and with ./rhq-agent.sh -l --advanced I did specified the keystore/trustsore passwords (both client and server side) - all those passwords were written as: "RESTRICTED::mysecretpassword" (plain word of the password i entered and not the hash-ed one).

scenario:
having agent installed from JON 3.3 ER04 jar, perform:
`./rhq-agent.sh -l --advanced`
provide password for the keystore files (and even if connection to the server is not well-configured) look at the default prefs.xml.
Comment 27 Stefan Negrea 2014-10-20 11:47:30 EDT
The problem from comment #25 has been resolved by commits from bug 1070262. Please retest with ER05.
Comment 28 Simeon Pinder 2014-10-20 13:26:39 EDT
Moving to CR01 as missed ER05 initial and extended cutoffs.
Comment 29 Stefan Negrea 2014-10-20 15:58:15 EDT
Just to clarify one more time, this bug has been fixed after ER4. New code was committed before ER5 release. Please retest.
Comment 30 Simeon Pinder 2014-10-21 16:24:28 EDT
Moving to ON_QA as available to test with the latest brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=394734
Comment 34 Garik Khachikyan 2014-10-27 08:26:44 EDT
# REOPEN

the scenario of update 3.2.0.GA -> 3.3.0.ER05 not encrypting the passwords. It shows up: RESTRICTED::rhqpwd
Comment 35 Garik Khachikyan 2014-10-27 08:46:23 EDT
the file is: .java/.userPrefs/rhq-agent/default/prefs.xml (agent configs)
Comment 36 Michael Burman 2014-10-27 09:10:34 EDT
This is referencing to the default keystore passwords:

rhq@rhqstorage:~/agent-bug/rhq-agent/bin$ grep password ~/.java/.userPrefs/rhq-agent/default/prefs.xml 
  <entry key="rhq.agent.client.security.keystore.key-password" value="RESTRICTED::rhqpwd"/>
  <entry key="rhq.agent.client.security.keystore.password" value="RESTRICTED::rhqpwd"/>
  <entry key="rhq.agent.client.security.truststore.password" value="RESTRICTED::null"/>
  <entry key="rhq.communications.connector.security.keystore.key-password" value="RESTRICTED::rhqpwd"/>
  <entry key="rhq.communications.connector.security.keystore.password" value="RESTRICTED::rhqpwd"/>
  <entry key="rhq.communications.connector.security.truststore.password" value=""/>
rhq@rhqstorage:~/agent-bug/rhq-agent/bin$
Comment 37 Garik Khachikyan 2014-10-27 12:05:28 EDT
please consider agent update through RPM as well.

Scenario:
===
an 3.2.0.GA agent is being installed and configured to get 2-side certificates enabled.
yum update agent.rpm should take care to mask those password fields.
===
Comment 39 Stefan Negrea 2014-10-28 11:27:59 EDT
master branch commit that fixes the agent packaging issues:

commit 26fc58f042b8c4c4cfbdc44245b466ce63c33d50
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Tue Oct 28 10:15:05 2014 -0500

    [BZ 1070262] Fix agent packaging to set the picketbox version for ant-run.xml script via ma
Comment 47 Simeon Pinder 2014-11-03 14:03:38 EST
Moving to ON_QA as available to test with latest brew build:
https://brewweb.devel.redhat.com//buildinfo?buildID=396547
Comment 48 Garik Khachikyan 2014-11-04 06:24:54 EST
@Stefan:

now it looks pretty promising, but:

"<entry key="rhq.communications.connector.security.truststore.password" value="plaintext"/>" #(still plain text)

could you please also specify what is the current version of picketbox (and the one that would be updated to have this fixed) - and what is that file name, where is it located etc. (so I could next time investigate, include the version of that package in bug comment(s) as well). thanks

for now let me reopen this bug.
Comment 50 Garik Khachikyan 2014-11-04 06:31:15 EST
... (In reply to Garik Khachikyan from comment #48)
> @Stefan:
> 
> now it looks pretty promising, but:
> 
> "<entry key="rhq.communications.connector.security.truststore.password"
> value="plaintext"/>" #(still plain text)
> 
> could you please also specify what is the current version of picketbox (and
> the one that would be updated to have this fixed) - and what is that file
> name, where is it located etc. (so I could next time investigate, include
> the version of that package in bug comment(s) as well). thanks
> 
> for now let me reopen this bug.

and the file is: ~/.java/.userPrefs/rhq-agent/default/prefs.xml
Comment 51 Garik Khachikyan 2014-11-04 07:08:05 EST
note to myself: rpm-based agent has same property not masked too. (latest brew agent rpm).
Comment 52 Garik Khachikyan 2014-11-04 09:40:52 EST
Created attachment 953643 [details]
prefs.xml (with that property unmasked)
Comment 53 Stefan Negrea 2014-11-04 10:04:11 EST
The property that was not encoded was missed from the list of properties to be encoded for the agent configuration. The fix was simple, I just added the property to the list.


master branch commit: 

commit 5807d59ed2186bc94927c3ee3929e96007457a60
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Tue Nov 4 08:59:57 2014 -0600

    [BZ 1070262] Adding one missed property to the list of @RESTRICTED properties for the agent configuration file.
Comment 54 Jay Shaughnessy 2014-11-06 12:19:04 EST
release/jon3.3.x commit 4a4f4e6fc1e0c92f55f679e616977f3d2f692669
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Tue Nov 4 08:59:57 2014 -0600

    (cherry picked from commit 5807d59ed2186bc94927c3ee3929e96007457a60)
    Signed-off-by: Jay Shaughnessy <jshaughn@redhat.com>
Comment 55 Simeon Pinder 2014-11-13 23:48:20 EST
Moving to ON_QA as available for test with build:
https://brewweb.devel.redhat.com//buildinfo?buildID=398756
Comment 56 Garik Khachikyan 2014-11-14 08:01:18 EST
# REOPEN

another "hidden" property left not masked during 3.2.0 -> 3.3 upgrade:
===

scenario:
1. install 3.2.0.GA; configure settings to enable 2-side certification
2. unzip the 3.3 zip
3. configure rhq-server.properties of 3.3 (with plain typed passwords) exactly with values from the current 3.2.0 setup
4. run the upgrade
5. check following:
---
grep -E "truststore|keystore" ~/.java/.userPrefs/rhq-server/default/prefs.xml # this is OK, ALL properties got masked (which is good)
grep -E "truststore|keystore" ~/jon-server-3.3.0.GA/bin/rhq-server.properties

here i do see 2 properties not masked still (rest of them are however):
---
rhq.server.tomcat.security.keystore.password=secret
rhq.server.tomcat.security.truststore.password=secret

I do keeping the reproducer server setup, ping me for the access pls.
Comment 59 Stefan Negrea 2014-11-14 10:08:46 EST
Added all the tomcat password related properties to the server configuration constants file and the upgrade code. Both of these will be picked up by the upgrade and usage code automatically.



master branch commit:

commit b9bcae050c1613cfe9a300d8b5e465e072448a67
Author: Stefan Negrea <snegrea@redhat.com>
Date:   Fri Nov 14 09:07:11 2014 -0600

    [BZ 1070262] Added all the tomcat password related properties to the ServerConfigurationConstants file a
Comment 62 Garik Khachikyan 2014-11-14 10:27:26 EST
https://bugzilla.redhat.com/show_bug.cgi?id=1164299 prepared to track the #56

otherwise: verified.
Comment 67 JBoss JIRA Server 2015-08-11 15:46:23 EDT
mfoley user <mfoley@redhat.com> updated the status of jira JON3-40 to Resolved

Note You need to log in before you can comment on or make changes to this bug.