Bug 1804193 - Podman support for FIPS Mode requires a bind mount inside the container [container-tools-rhel8-rhel-8.3.0/podman]
Summary: Podman support for FIPS Mode requires a bind mount inside the container [cont...
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.3
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: Tom Sweeney
QA Contact: atomic-bugs@redhat.com
Gabriela Nečasová
Depends On:
Blocks: 1784950 1804543
TreeView+ depends on / blocked
Reported: 2020-02-18 12:32 UTC by Jindrich Novy
Modified: 2020-11-04 21:37 UTC (History)
18 users (show)

Fixed In Version: podman-1.9.3 and newer
Doc Type: Bug Fix
Doc Text:
.Notes on FIPS support with Podman The Federal Information Processing Standard (FIPS) requires certified modules to be used. Previously, Podman correctly installed certified modules in containers by enabling the proper flags at startup. However, in this release, Podman does not properly set up the additional application helpers normally provided by the system in the form of the FIPS system-wide crypto-policy. Although setting the system-wide crypto-policy is not required by the certified modules it does improve the ability of applications to use crypto modules in compliant ways. To work around this problem, change your container to run the `update-crypto-policies --set FIPS` command before any other application code was executed. The `update-crypto-policies --set FIPS` command is no longer required with this fix.
Clone Of:
Last Closed: 2020-11-04 03:05:10 UTC
Type: ---
Target Upstream Version:

Attachments (Terms of Use)
strace log of running the container with fips mode enabled. (202.44 KB, text/plain)
2020-04-01 05:30 UTC, Edward Shen
no flags Details
strace log of running the container with fips mode enabled. (3.57 MB, text/plain)
2020-04-01 11:50 UTC, Edward Shen
no flags Details

Description Jindrich Novy 2020-02-18 12:32:48 UTC
This is a tracking bug assuring the fix for [bug 1784950] gets applied in stream-container-tools-2.0-rhel-8.2.0 branch of podman.

Comment 11 Daniel Walsh 2020-03-25 15:16:41 UTC
Yes I agree, this is not fixing the issue fully in RHEL8. We will not be getting this fixed quickly, but the question is how much is broken.  Will this prevent a container platform from getting fully certified as FIPS 140.

IE Do we have the same level of support as we do for a RHEL7 container and is this enough?

Comment 12 Daniel Walsh 2020-03-25 20:23:58 UTC
Ok we did some experimenting on fips mode. I believe adding 

echo /etc/crypto-policies/back-ends:/etc/crypto-policies/back-ends >> /usr/share/containers/mounts.conf

Should fix the issue.

The fips-mode-check tool will NOT work to check if the container is in fips mode, since it requires
dracut and an initrd to be available inside of the container, so we don't have a decent tool to check
if everything is setup correctly.

Don't know of other tools.

Comment 13 Tom Sweeney 2020-03-25 20:42:18 UTC
Per Dan's comment #12, we've added:

# cat /usr/share/containers/mounts.conf

And then ran:

# podman run -it --rm  --privileged  ubi8
[root@a5fd343a83ea /]# 

[root@a5fd343a83ea /]# /usr/bin/fips-mode-setup --check
/usr/bin/fips-mode-setup: line 66: lsinitrd: command not found
Installation of FIPS modules is not completed.
FIPS mode is enabled.
Inconsistent state detected.

[root@a5fd343a83ea /]# ls /usr/bin/lsinitrd
ls: cannot access '/usr/bin/lsinitrd': No such file or directory

[root@a5fd343a83ea /]# yum install dracut
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.

  dracut-049-27.git20190906.el8_1.1.x86_64     kbd-2.0.4-8.el8.x86_64                      hardlink-1:1.3-6.el8.x86_64       pigz-2.4-2.el8.x86_64                 
  kpartx-0.8.0-5.el8.x86_64                    kbd-legacy-2.0.4-8.el8.noarch               procps-ng-3.3.15-1.el8.x86_64     systemd-udev-239-18.el8_1.4.x86_64    
  xz-5.2.4-3.el8.x86_64                        libkcapi-hmaccalc-1.1.1-16_1.el8.x86_64     kmod-25-13.el8.x86_64             cpio-2.12-8.el8.x86_64                
  kbd-misc-2.0.4-8.el8.noarch                  libkcapi-1.1.1-16_1.el8.x86_64             


[root@a5fd343a83ea /]# /usr/bin/fips-mode-setup --check
No <initramfs file> specified and the default image '/boot/initramfs-4.18.0-147.el8.x86_64.img' cannot be accessed!

Usage: lsinitrd [options] [<initramfs file> [<filename> [<filename> [...] ]]]
Usage: lsinitrd [options] -k <kernel version>

-h, --help                  print a help message and exit.
-s, --size                  sort the contents of the initramfs by size.
-m, --mod                   list modules.
-f, --file <filename>       print the contents of <filename>.
--unpack                    unpack the initramfs, instead of displaying the contents.
                            If optional filenames are given, will only unpack specified files,
                            else the whole image will be unpacked. Won't unpack anything from early cpio part.
--unpackearly               unpack the early microcode part of the initramfs.
                            Same as --unpack, but only unpack files from early cpio part.
-v, --verbose               unpack verbosely.
-k, --kver <kernel version> inspect the initramfs of <kernel version>.

Installation of FIPS modules is not completed.
FIPS mode is enabled.
Inconsistent state detected.

If there are adjustments or additions I could make to the above, please let me know.

Comment 14 Simo Sorce 2020-03-25 21:06:35 UTC
If this is mounting the host's directory in the container what will happen is that the container will blow up if the host and the container used different version of RHEL for their packages.
The working mechanism is described here:
The description field contains the necessary steps.

To test if fips mode is properly activated you can write a small program that tries to use EVP_md5() from openssl to make a digest, it will be blocked in FIPS mode.

Comment 15 Simo Sorce 2020-03-25 21:07:33 UTC
Actually a smal lgo program that tries to generate a md5 sum should also fail, and that will also test that the golang tooling properly works in fips mode.

Comment 16 Daniel Walsh 2020-03-25 21:20:48 UTC

Should fix the issue.

The problem is the documentation  for this is WRONG.


The following
# mount --bind /usr/share/crypto-policies/back-ends/FIPS /etc/crypto-policies/back-ends

Should be:
# mount --bind /usr/share/crypto-policies/FIPS /etc/crypto-policies/back-ends

We coded up the original mount command and no one ever verified it was working until now.

Comment 17 Tomas Mraz 2020-03-26 09:02:07 UTC

This is simply wrong! The documentation is right and the
mount --bind /usr/share/crypto-policies/back-ends/FIPS /etc/crypto-policies/back-ends

is correct!

Dan, you're probably looking at some old version of crypto-policies package. Possibly from RHEL-8.1!

Comment 18 Daniel Walsh 2020-03-26 10:18:53 UTC
Ok so this means that we need to test against a UBI Image build on RHEL8.2 packages.

So this needs to go back to QE,

Comment 19 Daniel Walsh 2020-03-26 10:21:26 UTC
Edward could you retest with an updated UBI Image.

Comment 32 Edward Shen 2020-04-01 05:30:51 UTC
Created attachment 1675296 [details]
strace log of running the container with fips mode enabled.

Comment 35 Edward Shen 2020-04-01 11:50:10 UTC
Created attachment 1675391 [details]
strace log of running the container with fips mode enabled.

Comment 37 Tomas Mraz 2020-04-01 12:29:41 UTC
Thank you. So from the strace this is the problem:

11157 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay-containers/954ba027da6523bb9908a23db0885961063eb4ede35ea38df10c12a72b1e9275/userdata/usr/share/crypto-policies/back-ends/FIPS", 0xc000418ed8, 0) = -1 ENOENT (No such file or directory)

The ubi8:8.2-203 image should contain the /usr/share/crypto-policies/back-ends/FIPS (as verified by the buildah working with this image) so there must be something else in play here.

Comment 40 Daniel Walsh 2020-05-13 20:43:36 UTC
This should be fixed,  Tomas please confirm.

Comment 41 Tomas Mraz 2020-05-19 15:12:06 UTC
Which version of podman are we talking here about?

When I run:

podman run -it --rm -v /etc/system-fips:/etc/system-fips ubi8

It will still not bind-mount the /usr/share/crypto-policies/back-ends/FIPS and I still see just the symlinks to DEFAULT.

[root@7f4578701937 /]# ls -l /etc/crypto-policies/back-ends/
total 0
lrwxrwxrwx. 1 root root 43 Apr 23 05:14 bind.config -> /usr/share/crypto-policies/DEFAULT/bind.txt
lrwxrwxrwx. 1 root root 45 Apr 23 05:14 gnutls.config -> /usr/share/crypto-policies/DEFAULT/gnutls.txt
lrwxrwxrwx. 1 root root 43 Apr 23 05:14 java.config -> /usr/share/crypto-policies/DEFAULT/java.txt
lrwxrwxrwx. 1 root root 43 Apr 23 05:14 krb5.config -> /usr/share/crypto-policies/DEFAULT/krb5.txt
lrwxrwxrwx. 1 root root 48 Apr 23 05:14 libreswan.config -> /usr/share/crypto-policies/DEFAULT/libreswan.txt
lrwxrwxrwx. 1 root root 45 Apr 23 05:14 libssh.config -> /usr/share/crypto-policies/DEFAULT/libssh.txt
lrwxrwxrwx. 1 root root 42 Apr 23 05:14 nss.config -> /usr/share/crypto-policies/DEFAULT/nss.txt
lrwxrwxrwx. 1 root root 46 Apr 23 05:14 openssh.config -> /usr/share/crypto-policies/DEFAULT/openssh.txt
lrwxrwxrwx. 1 root root 52 Apr 23 05:14 opensshserver.config -> /usr/share/crypto-policies/DEFAULT/opensshserver.txt
lrwxrwxrwx. 1 root root 46 Apr 23 05:14 openssl.config -> /usr/share/crypto-policies/DEFAULT/openssl.txt
lrwxrwxrwx. 1 root root 49 Apr 23 05:14 opensslcnf.config -> /usr/share/crypto-policies/DEFAULT/opensslcnf.txt

This is with podman-1.9.2-1.module+el8.2.1+6626+598993b4.x86_64

And this is in the strace:
6289  newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay-containers/9739d7e09ee94eefc73b94a8b524318c8e76260e5fb5c6e06f9a02da257ffd9a/userdata/usr/share/crypto-policies/back-ends/FIPS", 0xc000444378, 0) = -1 ENOENT (No such file or directory)

So that does not seem to be fixed at least with the version above.

Comment 42 Daniel Walsh 2020-05-19 15:49:40 UTC
You should not need to do the `-v /etc/system-fips:/etc/system-fips`, this should happen automatically.  And /etc/system-fips -> /run/securet/system-fips.

Podman should complete the link if the system is in Fips mode.

If podman is looking for the /usr/share/crypto-policies/back-ends/FIPS in userdata then this would be a bug.  It should be looking for that content in the image.

Comment 43 Tom Sweeney 2020-05-19 22:02:14 UTC
Jindrich, heads up.

New PR to further address this issue:  https://github.com/containers/libpod/pull/6281

Comment 50 Simo Sorce 2020-05-21 19:02:19 UTC
you should add the caveat of crypto policies not being enforced correctly in 8.2 based products as a release note or similar, but the libraries themselves do not rely on crypto-policicies for FIPS validation so even w/o the correct policy the claim Openshift makes of using FIPS validated cryptography is fulfilled.


Comment 51 Daniel Walsh 2020-05-21 19:10:21 UTC
I agree that we can claim FIPS compliance, and the crypto policies will be fixed in podman-1.9.3 which should be in RHEL8.2.1 in about 10 weeks.

Comment 52 Jindrich Novy 2020-05-22 11:41:30 UTC
Moving this to 8.3.0 as it's already too late for 8.2.1 unless podman-1.9.3 is released before 29th May.

Comment 53 Tom Sweeney 2020-05-22 15:55:27 UTC
Simo if you could create a release note on your caveat and add it to the Doc Text of this BZ, I'd very much appreciate that.

Comment 54 Simo Sorce 2020-06-01 14:04:02 UTC
Gabriela asked for help on reviewing the release notes text.

For tracing purposes I am setting the text I suggested her on this bug so that others can also see an comment if they like.

Comment 58 Daniel Walsh 2020-06-03 14:04:57 UTC
Since we have this in podman 2.0 I am moving to Post.

Comment 76 errata-xmlrpc 2020-11-04 03:05:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.