Hawkular has its own component. Re-assigning.

Hi

I provided a patch for this issue on FIPS environments: https://github.com/openshift/openshift-ansible/pull/11391


It is not merge yet, and need some review process. But you can test in your environment, that should solve the problem.

Created a PR in Hawkular metrics for sha256crypt option for htpasswd.  

https://github.com/hawkular/hawkular-metrics/pull/1019

Used in combination with the inventory variable provided by Ruben's PR set like this:

openshift_metrics_hawkular_htpasswd_encryption_algorithm=sha256_crypt

Per ansible doc for htpasswd:
https://docs.ansible.com/ansible/2.7/modules/htpasswd_module.html?highlight=htpasswd

Still needs further integration testing.  Ruben, let me know what you think.  This BasicAuthenticator.java was the only file I found doing anything directly with the passwords but it's possible I missed something else. I did find that the IP address was hashed with MD5 to create a unique ID in UUIDGen.java.

Note - for me to successfully to install with this patch, I also had to change the following:

/usr/lib/python2.7/site-packages/passlib/handlers/digests.py
#=============================================================================
# predefined handlers
#=============================================================================
#hex_md4     = create_hex_hash(md4,              "md4")
#hex_md5     = create_hex_hash(hashlib.md5,      "md5")
#hex_md5.django_name = "unsalted_md5"


The lines above were commented out to avoid a stop due to SELinux rules enforcing FIPS compliance.

@Brian

The change on the PR looks good so far, but I think is risky to change the hash on UUIDGen.java. So, I'm not sure if without that change hawkular metrics is 100% FIPS compliant and if that matters or we only want a "functional fix" approach.

(In reply to Ruben Vargas Palma from comment #15)
> @Brian
> 
> The change on the PR looks good so far, but I think is risky to change the
> hash on UUIDGen.java. So, I'm not sure if without that change hawkular
> metrics is 100% FIPS compliant and if that matters or we only want a
> "functional fix" approach.

FIPS compliance is a legal requirement [0] for US Government deployments and is also required for several regulated industries like Healthcare. Important to use FIPS-validated crypto libraries, ciphers, and algorithms.

Having incremental progress would certainly be great, e.g. "Deploy on a FIPS-enabled RHEL host now for functional product deployments. Version n+1 will swap to FIPS validated crypto libraries." But the end goal is still FIPS compliance.

[0] https://csrc.nist.gov/Projects/cryptographic-module-validation-program

@Ruben Thanks, take a look at the latest.  I went ahead and added PBKDF2 algorithms since there was some "question" over just cyrpt with sha256 alone.  Again, still would need further integration testing.

https://github.com/hawkular/hawkular-metrics/pull/1020

@Shawn, I ran into several conflicting statements over what is truly "required" from FIPS standpoint on password hashing.  At any rate, I figured it wasn't too hard to add PBKDF2 which appears pretty standard in a lot of places today. 

However, adding PBKDF2 to the list means that the openshift-ansible project would be required to run a custom password create script (included in PR), since this password hashing algorithm is not standardized across languages/libraries (passlib in python appears to implement it differently than java standard crypto libraries).

(In reply to Brian Ward from comment #17)
> @Ruben Thanks, take a look at the latest.  I went ahead and added PBKDF2
> algorithms since there was some "question" over just cyrpt with sha256
> alone.  Again, still would need further integration testing.
> 
> https://github.com/hawkular/hawkular-metrics/pull/1020
> 
> @Shawn, I ran into several conflicting statements over what is truly
> "required" from FIPS standpoint on password hashing.  At any rate, I figured
> it wasn't too hard to add PBKDF2 which appears pretty standard in a lot of
> places today. 

Definitely believe it.  For PBKDF there is FIPS SP 800-132, "Recommendation for Password-Based Key Derivation" [0]. But that's more implementation-level guidance.

For FIPS, the best language seems to be directly from the NIST website [1]:

   "If the [customer] specifies that the information or data be cryptographically
   protected, then FIPS 140-2 is applicable. In essence, if cryptography is
   required, then it must be validated." 

So, for this conversation, whether passlib's sha256_crypt is used, or some library provides PBKDF2, the implementation must follow the applicable NIST guidelines and then be FIPS validated.

For other products, like Ansible Tower, we've moved towards using python-cryptography. Behind the scenes python cryptography uses the OpenSSL libraries of the underlying host, which for RHEL, have undergone FIPS validation. This means Ansible Tower will inherit FIPS validations when running on RHEL. Or for OpenShift, OpenShift will call OpenSSL to generate PKI certificates, thus ensuring FIPS inheritance.

If PBKDF2 is used, can we show how it's using a FIPS validated cryptography library, such as OpenSSL? A full listing of the RHEL-provided FIPS crypto is at [2].


> 
> However, adding PBKDF2 to the list means that the openshift-ansible project
> would be required to run a custom password create script (included in PR),
> since this password hashing algorithm is not standardized across
> languages/libraries (passlib in python appears to implement it differently
> than java standard crypto libraries).



[0] https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-132.pdf
[1] https://csrc.nist.gov/Projects/cryptographic-module-validation-program
[2] https://www.redhat.com/en/about/press-releases/red-hat-completes-fips-140-2-re-certification-red-hat-enterprise-linux-7

Hi Aditya

I've already merged the change that accept new hash functions, (FIPS compliant hash functions), from that side, I see no problems. 

But on ansible side I need to test how it plays with the ansible-installer, I'm still waiting feedback if it is possible to use passlib , which is the library used by openshift-ansibl  and the library used by htpasswd ansible module, or we need to move to another solution to be FIPS compliant.


@Shawn @Brian Any comments on that? I'm not FIPS expert.


Thanks!

Hello Team,

Can we get the recent status of this bugzilla? Have to keep posted the end customer accordingly.

-Niket

Hi

I've already productized an image with the htpasswd utility, and sent a PR[1] for integrate it with ansible-installer, unfortunately I did some assumptions about the availability of docker on the node at installation time, I'm working on a another approach. I'll keep this BZ updated.



1) https://github.com/openshift/openshift-ansible/pull/11809

(In reply to Ruben Vargas Palma from comment #27)
> Hi
> 
> I've already productized an image with the htpasswd utility, and sent a
> PR[1] for integrate it with ansible-installer, unfortunately I did some
> assumptions about the availability of docker on the node at installation
> time, I'm working on a another approach. I'll keep this BZ updated.
> 
> 
> 
> 1) https://github.com/openshift/openshift-ansible/pull/11809

Can all of these private comments with no customer/confidential information please be changed to public comments?

# openshift version
openshift v3.11.218

FIPS is enabled on each node
# cat /proc/sys/crypto/fips_enabled
1

Tested with
# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.11.218-1.git.0.6f55149.el7.noarch
openshift-ansible-playbooks-3.11.218-1.git.0.6f55149.el7.noarch
openshift-ansible-3.11.218-1.git.0.6f55149.el7.noarch
openshift-ansible-docs-3.11.218-1.git.0.6f55149.el7.noarch

if we run the inventory file from any node of the cluster, no matter it is Master or Non-Master, we would get installation error, if we use a vm not from the cluster, if would be successful
******************
TASK [openshift_metrics : generate htpasswd file for hawkular metrics] ********************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_metrics/tasks/generate_hawkular_certificates.yaml:25
Friday 22 May 2020  04:06:41 -0400 (0:00:00.289)       0:00:49.332 ************ 
Using module file /usr/lib/python2.7/site-packages/ansible/modules/web_infrastructure/htpasswd.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '/usr/bin/python2 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_UPIY6c/ansible_module_htpasswd.py", line 102, in <module>
    from passlib.apache import HtpasswdFile, htpasswd_context
  File "/usr/lib/python2.7/site-packages/passlib/apache.py", line 17, in <module>
    from passlib.hash import htdigest
  File "/usr/lib/python2.7/site-packages/passlib/registry.py", line 37, in __getattr__
    handler = get_crypt_handler(attr, None)
  File "/usr/lib/python2.7/site-packages/passlib/registry.py", line 340, in get_crypt_handler
    mod = __import__(modname, fromlist=[modattr], level=0)
  File "/usr/lib/python2.7/site-packages/passlib/handlers/digests.py", line 73, in <module>
    hex_md5     = create_hex_hash(hashlib.md5,      "md5")
  File "/usr/lib/python2.7/site-packages/passlib/handlers/digests.py", line 56, in create_hex_hash
    h = hash()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
fatal: [ci-vm-10-0-150-230.hosted.upshift.rdu2.redhat.com -> localhost]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_UPIY6c/ansible_module_htpasswd.py\", line 102, in <module>\n    from passlib.apache import HtpasswdFile, htpasswd_context\n  File \"/usr/lib/python2.7/site-packages/passlib/apache.py\", line 17, in <module>\n    from passlib.hash import htdigest\n  File \"/usr/lib/python2.7/site-packages/passlib/registry.py\", line 37, in __getattr__\n    handler = get_crypt_handler(attr, None)\n  File \"/usr/lib/python2.7/site-packages/passlib/registry.py\", line 340, in get_crypt_handler\n    mod = __import__(modname, fromlist=[modattr], level=0)\n  File \"/usr/lib/python2.7/site-packages/passlib/handlers/digests.py\", line 73, in <module>\n    hex_md5     = create_hex_hash(hashlib.md5,      \"md5\")\n  File \"/usr/lib/python2.7/site-packages/passlib/handlers/digests.py\", line 56, in create_hex_hash\n    h = hash()\nValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
******************

Created attachment 1690986 [details]
Installation logs

run the inventory file from any node of the cluster, it would be failed

Hi Junqi

Could you attach your inventory file?

Thanks

Just a comment

You need to set openshift_metrics_hawkular_fips=true in your inventory.

(In reply to Ruben Vargas Palma from comment #45)
> Just a comment
> 
> You need to set openshift_metrics_hawkular_fips=true in your inventory.

thanks, after add it, the installation is successful.
attach the inventory file

Created attachment 1692081 [details]
inventory file on FIPS cluster

Tested with
# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.11.219-1.git.0.8845382.el7.noarch
openshift-ansible-playbooks-3.11.219-1.git.0.8845382.el7.noarch
openshift-ansible-docs-3.11.219-1.git.0.8845382.el7.noarch
openshift-ansible-3.11.219-1.git.0.8845382.el7.noarch

Junqi

Is the hawkular metrics stack working at expected after the installation?

I did the test on my laptop, just want to make sure.


Thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2215