Description of problem: Once set into FIPS mode, RHV-H fails to reboot. On the serial console we see: [ 13.854806] dracut: FATAL: FIPS integrity test failed [ 13.859879] dracut: Refusing to continue [ 13.823293] dracut-pre-pivot[1171]: Warning: /boot/.vmlinuz-3.10.0-1062.el7.x86_64.hmac does not exist [ 15.502617] System halted. but .vmlinuz-3.10.0-1062.el7.x86_64.hmac is there on FS: [root@dell-r210ii-10 ~]# ls -l /boot/.vmlinuz-3.10.0-1062.el7.x86_64.hmac -rw-r--r--. 1 root root 167 1 ago 15.35 /boot/.vmlinuz-3.10.0-1062.el7.x86_64.hmac [root@dell-r210ii-10 ~]# rpm -qf /boot/.vmlinuz-3.10.0-1062.el7.x86_64.hmac kernel-3.10.0-1062.el7.x86_64 although: [root@dell-r210ii-10 ~]# FIPSCHECK_DEBUG=error fipscheck /boot/vmlinuz-3.10.0-1062.el7.x86_64 fipscheck: Hmac mismatch on file '/boot/vmlinuz-3.10.0-1062.el7.x86_64' : No such file or directory Version-Release number of selected component (if applicable): [root@dell-r210ii-10 boot]# nodectl info layers: rhvh-4.3.5.2-0.20190722.0: rhvh-4.3.5.2-0.20190722.0+1 bootloader: default: rhvh-4.3.5.2-0.20190722.0 (3.10.0-1062.el7.x86_64) entries: rhvh-4.3.5.2-0.20190722.0 (3.10.0-1062.el7.x86_64): index: 0 title: rhvh-4.3.5.2-0.20190722.0 (3.10.0-1062.el7.x86_64) kernel: /boot/rhvh-4.3.5.2-0.20190722.0+1/vmlinuz-3.10.0-1062.el7.x86_64 args: "ro nofb quiet default_hugepagesz=1GB hugepagesz=1GB hugepages=4 hugepagesz=2M hugepages=1024console=tty0 crashkernel=auto rd.lvm.lv=rhvh_dell-r210ii-10/swap rd.lvm.lv=rhvh_dell-r210ii-10/rhvh-4.3.5.2-0.20190722.0+1 console=ttyS1,115200 LANG=en_US.UTF-8 img.bootid=rhvh-4.3.5.2-0.20190722.0+1" initrd: /boot/rhvh-4.3.5.2-0.20190722.0+1/initramfs-3.10.0-1062.el7.x86_64.img root: /dev/rhvh_dell-r210ii-10/rhvh-4.3.5.2-0.20190722.0+1 current_layer: rhvh-4.3.5.2-0.20190722.0+1 How reproducible: 2 hosts over 2 Steps to Reproduce: 1. deploy RHV-H 2. add the host to the engine choosing: Kernel parameters -> FIPS 3. reboot the host Actual results: The host doesn't boot with: [ 13.854806] dracut: FATAL: FIPS integrity test failed [ 13.859879] dracut: Refusing to continue [ 13.823293] dracut-pre-pivot[1171]: Warning: /boot/.vmlinuz-3.10.0-1062.el7.x86_64.hmac does not exist [ 15.502617] System halted. Expected results: the host successfully reboots Additional info:
Additional info: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-federal_standards_and_regulations explicitly says: "If your /boot or /boot/EFI/ partitions reside on separate partitions, add the boot=<partition> (where <partition> stands for /boot) parameter to the kernel command line as well." and this is definitively our case on RHV-H. I'm just wondering if host-deploy simply skips that step.
The issue is probably here: https://github.com/oVirt/ovirt-engine/blob/master/frontend/webadmin/modules/uicommonweb/src/main/java/org/ovirt/engine/ui/uicommonweb/models/hosts/KernelCmdlineUtil.java#L110 The engine computes the whole parameter line for the kernel and, for the FIPS case, it will add only fips=1 without setting also boot=UUID=... as documented https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-federal_standards_and_regulations so the host will not reboot in FIPS mode if /boot is not on the root partition as in RHV-H case. Unfortunately I fear that the engine doesn't really know the UUID of the boot partition so it's probably not a straightforward fix.
Workaround: detect the missing UUID parameter with something like: findmnt --output=UUID --noheadings --target=/boot and instead of just clicking on FIPS checkbox, edit the Kernel command line field entering fips=1 boot=UUID=<boot_p_uuid> as in the attached screenshot
Created attachment 1600017 [details] Workaround
Fixing it on engine side is quite complex because the engine doesn't directly know the UUID of the boot partition for each host. What we can do is intercept the kernel parameters string on host-deploy here: https://github.com/oVirt/ovirt-host-deploy/blob/master/src/plugins/ovirt-host-deploy/kernel/kernel.py#L93 detect if 'fips=1' is there with no 'boot=' and in that case detect the missing value and inject the missing parameter. Once the host will reboot, host-monitoring should detect the new kernel cmd line and so the engine will detect it for the future. Please notice the python host-deploy is going to be deprecated/replaced in favour of a pure ansible implementation in 4.4 so this fix has to be re-applied there.
reducing severity to high and postponing to 4.3.6 since a simple workaround exists ( comment #3 )
Moving to the engine since ovirt-host-deploy is not going to be shipped in 4.4 and this should work with the ansible deployment as well.
Verified with: - RHV 4.4.0-0.32.master.el8ev - Red Hat Virtualization Host 4.4.0 (el8.2) Verification steps: 1. Remove RHV-H Host from an existing environment 2. Re-add the host with fips=1 as a kernel parameter (without specifying boot parameter) 3. Reboot host (in order to apply fips mode) Result: - The host was added successfully with FIPS enabled.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV Engine and Host Common Packages 4.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:3309