Bug 1734171 - RHV-M 4.3.4 does not boot with fips=1 when OpenSCAP profile used resulting in non FIPS compliant certificates
Summary: RHV-M 4.3.4 does not boot with fips=1 when OpenSCAP profile used resulting in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 4.3.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-4.3.5-2
: ---
Assignee: Simone Tiraboschi
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On:
Blocks: 1547768
TreeView+ depends on / blocked
 
Reported: 2019-07-29 20:52 UTC by amashah
Modified: 2020-05-28 09:28 UTC (History)
12 users (show)

Fixed In Version: ovirt-ansible-hosted-engine-setup-1.0.26-1
Doc Type: Bug Fix
Doc Text:
After performing OpenSCAP remediation, a reboot of the engine VM is required to enable FIPS mode. Previously, The ovirt-ansible-hosted-engine-setup wasn't triggering that reboot. As a result, engine-setup did not run FIPS mode, even though the Manager virtual machine would run in FIPS mode from the next reboot. By default, OpenSSL created PKCS#12 bundles with 40 bits RC2 encryption. However, this cipher is not available on FIPS configured machines. As a result, the first upgrade OpenSSL triggered by engine-setup was failing while parsing the PKCS#12 bundles. The current release fixes these issues for new deployments. If you run into these issues with previous versions, we recommend performing a recovery procedure.
Clone Of:
Environment:
Last Closed: 2019-08-26 09:58:36 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:
lrotenbe: testing_plan_complete+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github oVirt ovirt-ansible-hosted-engine-setup pull 228 0 'None' closed Enable fips mode on the bootstrap VM 2021-02-01 23:22:34 UTC
Red Hat Knowledge Base (Solution) 5047531 0 None None None 2020-05-10 21:59:15 UTC
Red Hat Product Errata RHBA-2019:2559 0 None None None 2019-08-26 09:59:03 UTC

Description amashah 2019-07-29 20:52:14 UTC 
Description of problem:

RHV-M is to support FIPS starting in 4.3 (in new deployments).

Brand new RHV environment was deployed (4.3 June 05 ova appliance). During HE deployment, OpenSCAP profile was applied, HE deployed.

Following the deployment, minor upgrade to latest RHV-M was attempted, but failed with:

~~~
          --== PKI CONFIGURATION ==--

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/jboss.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          140302963394448:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          140302963394448:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          140302963394448:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/websocket-proxy.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          140530020640656:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          140530020640656:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          140530020640656:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          140154234734480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          140154234734480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          140154234734480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/reports.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          140585187067792:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          140585187067792:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          140585187067792:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/imageio-proxy.p12'
          Perhaps it was changed since last Setup.
          Error was:
          MAC verified OK
          Error outputting keys and certificates
          139788111759248:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
          139788111759248:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
          139788111759248:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
~~~



1. Deploy new 4.3 HE environment
2. Try to update to latest minor version of RHV-M
3. Run engine-setup, it will fail like seen above.

Actual results:
Cannot update RHV-M in FIPS mode

Expected results:
Should be able to update to the latest RHV-M with FIPS enabled.

Additional info:

~~~
2019-07-29 15:11:06,840-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 --== PKI CONFIGURATION ==--
2019-07-29 15:11:06,841-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                
2019-07-29 15:11:06,842-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization
2019-07-29 15:11:06,843-0400 DEBUG otopi.context context._executeMethod:136 otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization condition False
2019-07-29 15:11:06,845-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization_upgrade
2019-07-29 15:11:06,847-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:813 execute: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), executable='None', cwd='None', env=None
2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:863 execute-result: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), rc=1
2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:921 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stdout:


2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:926 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stderr:
MAC verified OK
Error outputting keys and certificates
140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:

2019-07-29 15:11:06,874-0400 WARNING otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca ca._extractPKCS12CertificateString:130 Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12'
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 Perhaps it was changed since last Setup.
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 Error was:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 MAC verified OK
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 Error outputting keys and certificates
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND                 140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
2019-07-29 15:11:06,875-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND  
~~~
Comment 2 amashah 2019-07-30 19:49:10 UTC 
To add to this, I was able to reproduce this internally and it appears that the HE does not boot with FIPS=1 when the OpenSCAP profile is used, though the dracut-fips package is installed it boots normally, and this is likely the cause of the issue as the certificates RHV-M creates are likely not FIPS compliant.
Comment 4 Doron Fediuck 2019-07-31 07:57:38 UTC 
The upgrade process accidentally missed the FIPS mode which lead to this state.
We have a solution which will ensure the relevant Ansible role will now handle FIPS mode correctly.
Comment 18 Liran Rotenberg 2019-08-06 15:03:30 UTC 
Verified on:
ovirt-ansible-hosted-engine-setup-1.0.26-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.11-1.el7ev.noarch

Steps:
1. Set a host with FIPS mode enabled:
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot 
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem  for example:  
# blkid /dev/sda1  
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14 
Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....." 
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg 
# reboot

2. Verify host is with FIPS:
# sysctl crypto.fips_enabled 
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled 
1

3. Deploy hosted engine
# hosted-engine --deploy
In the setup when asked, set "yes" to default OpenSCAP profile.

4. Verify the HE VM is set with FIPS as for step 2.

5. Set the cluster in global maintenance mode.

6. Run engine-setup on the HE VM.

7. Verify FIPS as for step 3, also reboot and check again.

Results:
The engine-setup was completed successfully, FIPS was enabled on the HE-VM from the initial deployment, after engine-setup and after the reboot.
Comment 22 errata-xmlrpc 2019-08-26 09:58:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2559
Note You need to log in before you can comment on or make changes to this bug.