Bug 1612340

Summary: Kernel used by upgrade tool will not boot if FIPS is enabled
Product: Red Hat Enterprise Linux 6 Reporter: Welterlen Benoit <bwelterl>
Component: preupgrade-assistant-el6toel7Assignee: Petr Stodulka <pstodulk>
Status: CLOSED WONTFIX QA Contact: Alois Mahdal <amahdal>
Severity: high Docs Contact: Lenka Špačková <lkuprova>
Priority: high    
Version: 6.10CC: amahdal, asosedki, bwelterl, fedoraproject, fkrska, jzeleny, lkuprova, mgazdik, mkolaja, mreznik, phracek, pravisha, pstodulk, tbowling
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
In-place upgrade from a RHEL 6 system to RHEL 7.6 is impossible with FIPS mode enabled When upgrading a RHEL 6 system to RHEL 7.6 using the Red Hat Upgrade Tool with FIPS mode enabled, missing Hash-based Message Authentication Code (HMAC) prevents kernel data from being correctly verified. As a consequence, the Red Hat Upgrade Tool cannot boot into the target system kernel and the process fails. The recommended approach is to perform a clean installation instead. In case the administrator disables FIPS mode for the duration of the upgrade, all cryptographic keys must be regenerated and the FIPS compliance of the converted system must be reevaluated. For more information, see link:https://access.redhat.com/solutions/137833.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-15 13:42:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Welterlen Benoit 2018-08-03 17:15:24 UTC
Description of problem:
The kernel of the upgrader will no boot if FIPS is enabled.

Version-Release number of selected component (if applicable):
RHEL 6.x to RHEL7.x with FIPS enabled

How reproducible:
Easy, always

Steps to Reproduce:
Following the documentation here to upgrade RHEL6.10 to RHEL 7.5:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/migration_planning_guide/chap-red_hat_enterprise_linux-migration_planning_guide-upgrading

1. install or update to Red Hat 6.10, enable FIPS
2. Triggering redhat-upgrade-tool: redhat preupgrade scritps 
3. reboot to the upgrader
The kernel will not boot complaining about the .hmac missing


Actual results:
- not able to boot the upgrader kernel, and so the upgrade is not possible without disabling FIPS

Expected results:
- upgrade without issue

Additional info:
Workaround:
- The workaround is, before the reboot to proceed to the update, to extract the .hmac file from the RPM (from the repo or the ISO) and the kernel also because it is identified as vmlinuz-3.10.0-862.el7.x86_64.

For example if the ISO is mounted on /mnt:
# rpm2cpio /mnt/Packages/kernel-3.10.0-862.el7.x86_64.rpm | cpio -iv --to-stdout ./boot/.vmlinuz-3.10.0-862.el7.x86_64.hmac > /boot/.vmlinuz-3.10.0-862.el7.x86_64.hmac
# rpm2cpio /mnt/Packages/kernel-3.10.0-862.el7.x86_64.rpm | cpio -iv --to-stdout ./boot/vmlinuz-3.10.0-862.el7.x86_64 > /boot/vmlinuz-3.10.0-862.el7.x86_64	

Then the reboot to the kernel of the updater tool will work.

Comment 2 Petr Stodulka 2018-09-24 09:57:36 UTC
I see the issue. Unfortunately our team is in in time pressure so the solution will be postponed probably till the RHEL 7.7 GA time and the Known Issue will be published, meanwhile.

Comment 3 Petr Stodulka 2018-10-17 08:48:42 UTC
Hi Welterlen,
we tried your workround manytimes (even with various additional changes) but we have not been able to proceed the upgrade. Every time we've got warning that:
  /boot/.vmlinuz-3.10.0-957.el7.x86_64.hmac
does not exists. Even when we see that we are able to find it on the mentioned path with the same permissions as the others. The vmlinuz has been put there as well. Only workround that work for us is set fips=0 in kernel commandline.

As we are not able to fix it now and even use your workround, we will document the "fips=0" solution as a possible workround in known issue. Any help here is welcomed.

Comment 4 Welterlen Benoit 2018-10-17 09:01:59 UTC
Hello,

When debugging the issue, I was able to use the workaround.
Did you extract the same kernel that the one used during the boot ? I can see that your version is not the same than mine. 
Are the names and paths correct ?

Thank you

Benoit

Comment 5 Petr Stodulka 2018-10-17 09:19:25 UTC
Yes,
we extracted the files from the same kernel rpm that is used during the upgrade process (in case of upgrade using the network you can find the rpm in /system-upgrade directory). We checked in two people that paths has been correct. I can try to proceed the upgrade again later with clean virtual machine to let you know.

My version-release of kernel is different as we are working on upgrades RHEL 6.10 -> RHEL 7.6 nowadays.

Comment 15 Alois Mahdal 2019-10-14 18:52:56 UTC
Notes for QE
============

According to comment 0 this should be relatively straightforward to test;
just enable FIPS (I think it's basically just a kernel option; we could
have a test case / hook for that in morf) and run upgrade.


Granting qa_ack+.

Comment 16 Alois Mahdal 2019-10-14 18:54:45 UTC
(I accidentally dropped needinfo for Marcel; putting back...)