Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1957310

Summary: FIPS mode is not being enabled in containers
Product: OpenShift Container Platform Reporter: Juan Antonio Osorio <josorior>
Component: NodeAssignee: Urvashi Mohnani <umohnani>
Node sub component: CRI-O QA Contact: Sunil Choudhary <schoudha>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: unspecified CC: aos-bugs, danken, dcaspin, fdeutsch, jhrozek, josorior, mfiedler, miabbott, nagrawal, ssorce, xiyuan, xtian
Version: 4.8   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-10 05:52:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juan Antonio Osorio 2021-05-05 15:02:34 UTC
Description of problem:

On a FIPS-enabled cluster (from installation time) the expectation is that UBI-based containers
get the appropriate bind-mounts by default to enable FIPS.

On investigation, this doesn't seem to be the case.

Here's a sample container:

$ oc exec -n openshift-compliance compliance-operator-54f69dddf6-sxbpr -- update-crypto-policies --show                                                                                               
DEFAULT

....

Here's another sample, but with a core OpenShift component:

$ oc exec -n openshift-oauth-apiserver apiserver-7c59c9bbf9-468vg -- update-crypto-policies --show
Defaulted container "oauth-apiserver" out of: oauth-apiserver, fix-audit-permissions (init)
DEFAULT

...

On the other hand, the FIPS sysctl seems to be set:

$ oc exec -n openshift-oauth-apiserver apiserver-7c59c9bbf9-468vg -- cat /proc/sys/crypto/fips_enabled
Defaulted container "oauth-apiserver" out of: oauth-apiserver, fix-audit-permissions (init)
1

...

Finally, the bind-mounts setting the appropriate crypto policy are not there. They point to DEFAULT, instead of FIPS:

$ oc exec -n openshift-oauth-apiserver apiserver-7c59c9bbf9-468vg -- ls -la /etc/crypto-policies/back-ends/
Defaulted container "oauth-apiserver" out of: oauth-apiserver, fix-audit-permissions (init)
total 0
drwxr-xr-x. 2 root root 259 Mar  9 07:12 .
drwxr-xr-x. 6 root root  81 Mar  9 07:11 ..
lrwxrwxrwx. 1 root root  36 Mar  9 07:11 .config -> /usr/share/crypto-policies/DEFAULT/.
lrwxrwxrwx. 1 root root  43 Mar  9 07:12 bind.config -> /usr/share/crypto-policies/DEFAULT/bind.txt
lrwxrwxrwx. 1 root root  45 Mar  9 07:12 gnutls.config -> /usr/share/crypto-policies/DEFAULT/gnutls.txt
lrwxrwxrwx. 1 root root  43 Mar  9 07:12 java.config -> /usr/share/crypto-policies/DEFAULT/java.txt
lrwxrwxrwx. 1 root root  43 Mar  9 07:12 krb5.config -> /usr/share/crypto-policies/DEFAULT/krb5.txt
lrwxrwxrwx. 1 root root  48 Mar  9 07:12 libreswan.config -> /usr/share/crypto-policies/DEFAULT/libreswan.txt
lrwxrwxrwx. 1 root root  45 Mar  9 07:12 libssh.config -> /usr/share/crypto-policies/DEFAULT/libssh.txt
lrwxrwxrwx. 1 root root  42 Mar  9 07:12 nss.config -> /usr/share/crypto-policies/DEFAULT/nss.txt
lrwxrwxrwx. 1 root root  46 Mar  9 07:12 openssh.config -> /usr/share/crypto-policies/DEFAULT/openssh.txt
lrwxrwxrwx. 1 root root  52 Mar  9 07:12 opensshserver.config -> /usr/share/crypto-policies/DEFAULT/opensshserver.txt
lrwxrwxrwx. 1 root root  46 Mar  9 07:12 openssl.config -> /usr/share/crypto-policies/DEFAULT/openssl.txt
lrwxrwxrwx. 1 root root  49 Mar  9 07:12 opensslcnf.config -> /usr/share/crypto-policies/DEFAULT/opensslcnf.txt

Version-Release number of selected component (if applicable):

$ oc version                                                                                                                                                                                          
Client Version: 4.8.0-0.nightly-2021-04-30-102231
Server Version: 4.8.0-0.nightly-2021-04-26-151924
Kubernetes Version: v1.21.0-rc.0+6143dea

CRI-O version:
cri-o-1.21.0-78.rhaos4.8.gitdd0b833.el8.x86_64


How reproducible:


Steps to Reproduce:
1. Deploy a cluster with `fips: true` in the install-config.yaml
2. run `oc exec -n <namespace> <container name> -- ls -la /etc/crypto-policies/back-ends/` on a UBI-container

Actual results:

the container mounts show the usage of the DEFAULT policies

Expected results:

The container mounts show the usage of the FIPS policies


Additional info:

Note that I did double-check if the cluster was FIPS enabled. And it was done as follows:

$ oc debug -q node/ip-10-0-128-114.ec2.internal -- chroot /host /bin/bash -c "update-crypto-policies --show"
FIPS
$ oc debug -q node/ip-10-0-128-114.ec2.internal -- chroot /host /bin/bash -c "cat /etc/system-fips"
# FIPS module installation complete

Comment 2 Peter Hunt 2021-05-05 18:27:55 UTC
Urvashi, can you PTAL

Comment 4 Simo Sorce 2021-05-07 14:05:54 UTC
Please DO NOT use update-crypto-policies --show to check if your system is in FIPS mode.
That command only shows whatthe *crypto policies* are, but tells nothing on whether the system is correctly in fips mode.

Please check if /proc/sys/crypto/fips_enabled is 1

That is the only real indicator that the system is in fips mode, and the only indicator crypto libraries use to switch to FIPS mode.
The FIPs crypto policisy is just an additional helper and users are totally allowed to even create their own completely custom policy and name it anything and still be fips compliant.

Please change your tests to reflect above (and documentation as well if necessary).

Comment 5 Urvashi Mohnani 2021-05-07 14:21:45 UTC
Thanks @Simo!

From the description we see that /proc/sys/crypto/fips_enabled is set to 1, so the container is in FIPS mode as it should be. I saw the same thing on a 4.7 and 4.8 cluster in FIPS mode when testing this out.
Another tool I used was `fips-mode-setup --check` and it showed that FIPS mode was enabled in the container. The update-crypto-policies tool needs to be updated to be able to correctly determine the policy being used in the container based on the information available inside the container.

@Juan, will close this as not a bug unless there is anything else?

Comment 6 Juan Antonio Osorio 2021-05-10 05:52:24 UTC
Nope. We're good! Thanks for the clarification!

Comment 7 Dan Kenigsberg 2021-05-10 07:48:38 UTC
(In reply to Simo Sorce from comment #4)
> Please DO NOT use update-crypto-policies --show to check if your system is
> in FIPS mode.
> That command only shows whatthe *crypto policies* are, but tells nothing on
> whether the system is correctly in fips mode.
> 
> Please check if /proc/sys/crypto/fips_enabled is 1
> 
> That is the only real indicator that the system is in fips mode, and the
> only indicator crypto libraries use to switch to FIPS mode.
> The FIPs crypto policisy is just an additional helper and users are totally
> allowed to even create their own completely custom policy and name it
> anything and still be fips compliant.
> 
> Please change your tests to reflect above (and documentation as well if
> necessary).


I am confused by the RHEL docs at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#enabling-fips-mode-in-a-container_using-the-system-wide-cryptographic-policies .
It seems to suggest that the container runner has to mount /etc/system-fips and run `update-crypto-policies --set FIPS` inside the container before FIPS takes effect on the container processes.
Should the docs be modified to something like "as long as your container uses compliant crypto libraries such as those supplied by UBI8 and the host has fips_enabled, the container code would comply with FIPS"?

Comment 8 Juan Antonio Osorio 2021-05-10 07:52:33 UTC
I closed this one, but I do suggest opening another bug against the RHEL docs. That's also what I was reading and where this confusion came from to begin with.

Comment 10 Dan Kenigsberg 2021-05-11 06:13:51 UTC
Juan, Simo, would you help me outline the correct up-to-date RHEL docs?
Is the following a reasonable begining?

"as long as your container uses compliant crypto libraries such as those supplied by UBI8 and the host has fips_enabled, the container code would comply with FIPS"