Bug 1868661

Description of problem:
kernel-fips-mode test 

https://github.com/CKI-project/tests-beaker/blob/master/misc/kernel-fips-mode/runtest.sh

is failing with 

crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE

during second reboot


Version-Release number of selected component (if applicable):

dracut-049-95.git20200804.el8 with openssl-libs-1:1.1.1g-11.el8

How reproducible:
Easily with the kernel-fips-mode test


Steps to Reproduce:
1. fips-mode-setup --enable
2. reboot
3. fips-mode-setup --disable
4. rm -f /etc/dracut.conf.d/40-fips.conf /etc/system-fips && dracut -v -f
5. reboot - fails, dropped to dracut debug shell

Actual results:
reboot fails

Expected results:
machine is rebooted as expected

Additional info:

From original mail about this problem

<quote>
Hi,

kernel-fips-mode test is failing with openssl "FATAL FIPS SELFTEST FAILURE".

<console>
Starting Reboot...
[  413.647149] printk: systemd-shutdow: 58 output lines suppressed due to ratelimiting
[  413.820088] systemd-shutdown[1]: Syncing filesystems and block devices.
[  414.033123] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  414.050174] systemd-journald[1153]: Received SIGTERM from PID 1 (systemd-shutdow).
[  414.107939] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  414.113159] systemd-shutdown[1]: Unmounting file systems.
[  414.116445] [10437]: Remounting '/' read-only in with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota'.
[  414.619803] systemd-shutdown[1]: All filesystems unmounted.
[  414.619813] systemd-shutdown[1]: Deactivating swaps.
[  414.619903] systemd-shutdown[1]: All swaps deactivated.
[  414.619905] systemd-shutdown[1]: Detaching loop devices.
[  414.620075] systemd-shutdown[1]: All loop devices detached.
[  414.620078] systemd-shutdown[1]: Detaching DM devices.
[  414.713194] printk: /shutdown: 8 output lines suppressed due to ratelimiting
[  414.792478] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[  414.935212] XFS (dm-0): Unmounting Filesystem
[  414.953890] dracut Warning: Unmounted /oldroot.
[  414.996832] dracut: Disassembling device-mapper devices crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
[  415.082685] Core dump to |/bin/false pipe failed
/shutdown: line 113: 10483 Aborted                 $ACTION -f -d -n
[  415.083379] dracut Warning: reboot failed!
dracut Warning: reboot failed!
</console>

Here are links to the failed jobs.

https://beaker.engineering.redhat.com/recipes/8668216
https://beaker.engineering.redhat.com/recipes/8668212
https://beaker.engineering.redhat.com/recipes/8668208

Here is what I think is going on, but I don't understand the code, so I
for sure may be missing something.

The test is doing the following:
1) enable fips
2) reboot
3) disable fips
4) regenerate initramfs
5) reboot

https://github.com/CKI-project/tests-beaker/blob/master/misc/kernel-fips-mode/runtest.sh#L108

The second reboot(step 5) fails because the newly generated
initramfs no longer contains all necessary files for fips mode.
Namely checksums(.hmac) files for libcrypto and libssl. This
is the reason why openssl verify_checksums fails(abort) with
"FATAL FIPS SELFTEST" and we get dropped to dracut debug shell.

This is IMHO related to the "update to the 1.1.1g release" openssl commit
http://pkgs.devel.redhat.com/cgit/rpms/openssl/commit/?h=rhel-8.3.0&id=21d4a8eded185d957a50e33c724e5b7aa9138924

Particularly to the change how the fips mode is detected in openssl.
Before this commit it checked only existence of /etc/system-fips file,
but with the above update it also checks /proc/sys/crypto/fips_enabled.

Now in step 5, the newly generated initramfs is unpacked to /run/initramfs
by dracut-initramfs-restore(dracut-shutdown.service) and we end up
running dracut's shutdown from the initramfs.

https://github.com/dracutdevs/dracut/blob/master/modules.d/99shutdown/shutdown.sh#L121

Since reboot has dependency on libssl it will fail during fips
selftests, because we don't have .hmac files in the initramfs.

I guess the test may be fixed by removing the initramfs regeneration,
but I of course don't know how what's the expected openssl/dracut
behaviour in such situtaion. Again please I may be completely wrong, so
could you please take a look at this.

@@ -10067,7 +9882,7 @@ diff -up openssl-1.1.1b/crypto/o_init.c.fips openssl-1.1.1b/crypto/o_init.c
 +# include <stdlib.h>
 +# include <openssl/rand.h>
 +# include <openssl/fips.h>
-+# include "internal/fips_int.h"
++# include "crypto/fips.h"
 +
 +# define FIPS_MODE_SWITCH_FILE "/proc/sys/crypto/fips_enabled"
 +
@@ -10076,16 +9891,20 @@ diff -up openssl-1.1.1b/crypto/o_init.c.fips openssl-1.1.1b/crypto/o_init.c
 +    char buf[2] = "0";
 +    int fd;
 +
-+    /* Ensure the selftests always run */
-+    /* XXX: TO SOLVE - premature initialization due to selftests */
-+    FIPS_mode_set(1);
-+
 +    if (secure_getenv("OPENSSL_FORCE_FIPS_MODE") != NULL) {
 +        buf[0] = '1';
 +    } else if ((fd = open(FIPS_MODE_SWITCH_FILE, O_RDONLY)) >= 0) {
 +        while (read(fd, buf, sizeof(buf)) < 0 && errno == EINTR) ;
 +        close(fd);
 +    }
++
++    if (buf[0] != '1' && !FIPS_module_installed())
++        return;
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is IMHO the change causing this.

++
++    /* Ensure the selftests always run */
++    /* XXX: TO SOLVE - premature initialization due to selftests */
++    FIPS_mode_set(1);
++
 +    /* Failure reading the fips mode switch file means just not
 +     * switching into FIPS mode. We would break too many things
 +     * otherwise..
@@ -10110,21 +9929,18 @@ diff -up openssl-1.1.1b/crypto/o_init.c.fips openssl-1.1.1b/crypto/o_init.c
 +    if (done)
 +        return;
 +    done = 1;
-+    if (!FIPS_module_installed()) {
-+        return;
-+    }
 +    init_fips_mode();
 +}
 +#endif

Many thanks
</quote>

Per Tomas Mraz dracut needs to always include .hmac files for libcrypto and libssl in initramfs.