After performing OpenSCAP remediation, a reboot of the engine VM is required to enable FIPS mode. Previously, The ovirt-ansible-hosted-engine-setup wasn't triggering that reboot. As a result, engine-setup did not run FIPS mode, even though the Manager virtual machine would run in FIPS mode from the next reboot. By default, OpenSSL created PKCS#12 bundles with 40 bits RC2 encryption. However, this cipher is not available on FIPS configured machines. As a result, the first upgrade OpenSSL triggered by engine-setup was failing while parsing the PKCS#12 bundles.
The current release fixes these issues for new deployments. If you run into these issues with previous versions, we recommend performing a recovery procedure.
Description of problem:
RHV-M is to support FIPS starting in 4.3 (in new deployments).
Brand new RHV environment was deployed (4.3 June 05 ova appliance). During HE deployment, OpenSCAP profile was applied, HE deployed.
Following the deployment, minor upgrade to latest RHV-M was attempted, but failed with:
~~~
--== PKI CONFIGURATION ==--
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/jboss.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
140302963394448:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140302963394448:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140302963394448:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/websocket-proxy.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
140530020640656:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140530020640656:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140530020640656:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
140154234734480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140154234734480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140154234734480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/reports.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
140585187067792:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140585187067792:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140585187067792:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
[WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/imageio-proxy.p12'
Perhaps it was changed since last Setup.
Error was:
MAC verified OK
Error outputting keys and certificates
139788111759248:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
139788111759248:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
139788111759248:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
~~~
1. Deploy new 4.3 HE environment
2. Try to update to latest minor version of RHV-M
3. Run engine-setup, it will fail like seen above.
Actual results:
Cannot update RHV-M in FIPS mode
Expected results:
Should be able to update to the latest RHV-M with FIPS enabled.
Additional info:
~~~
2019-07-29 15:11:06,840-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND --== PKI CONFIGURATION ==--
2019-07-29 15:11:06,841-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND
2019-07-29 15:11:06,842-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization
2019-07-29 15:11:06,843-0400 DEBUG otopi.context context._executeMethod:136 otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization condition False
2019-07-29 15:11:06,845-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization_upgrade
2019-07-29 15:11:06,847-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:813 execute: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), executable='None', cwd='None', env=None
2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:863 execute-result: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), rc=1
2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:921 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stdout:
2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:926 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stderr:
MAC verified OK
Error outputting keys and certificates
140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
2019-07-29 15:11:06,874-0400 WARNING otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca ca._extractPKCS12CertificateString:130 Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12'
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Perhaps it was changed since last Setup.
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Error was:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND MAC verified OK
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Error outputting keys and certificates
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87:
2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139:
2019-07-29 15:11:06,875-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND
~~~
To add to this, I was able to reproduce this internally and it appears that the HE does not boot with FIPS=1 when the OpenSCAP profile is used, though the dracut-fips package is installed it boots normally, and this is likely the cause of the issue as the certificates RHV-M creates are likely not FIPS compliant.
The upgrade process accidentally missed the FIPS mode which lead to this state.
We have a solution which will ensure the relevant Ansible role will now handle FIPS mode correctly.
Verified on:
ovirt-ansible-hosted-engine-setup-1.0.26-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.11-1.el7ev.noarch
Steps:
1. Set a host with FIPS mode enabled:
# yum -y install prelink dracut-fips
# prelink -u -a
# dracut -f
# df /boot
Take the Filesystem value (for example /dev/vda1 or /dev/sda1)
# blkid $filesystem for example:
# blkid /dev/sda1
Take the UUID for example: 21f4da90-4055-47e4-8971-763691191f14
Edit /etc/default/grub fips=1 and boot=$uuid: GRUB_CMDLINE_LINUX="fips=1 boot=UUID=21f4da90-4055-47e4-8971-763691191f14 ....."
Regenerate grub, BIOS host:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot
2. Verify host is with FIPS:
# sysctl crypto.fips_enabled
crypto.fips_enabled = 1
# cat /proc/sys/crypto/fips_enabled
1
3. Deploy hosted engine
# hosted-engine --deploy
In the setup when asked, set "yes" to default OpenSCAP profile.
4. Verify the HE VM is set with FIPS as for step 2.
5. Set the cluster in global maintenance mode.
6. Run engine-setup on the HE VM.
7. Verify FIPS as for step 3, also reboot and check again.
Results:
The engine-setup was completed successfully, FIPS was enabled on the HE-VM from the initial deployment, after engine-setup and after the reboot.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2019:2559
Description of problem: RHV-M is to support FIPS starting in 4.3 (in new deployments). Brand new RHV environment was deployed (4.3 June 05 ova appliance). During HE deployment, OpenSCAP profile was applied, HE deployed. Following the deployment, minor upgrade to latest RHV-M was attempted, but failed with: ~~~ --== PKI CONFIGURATION ==-- [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/jboss.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 140302963394448:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140302963394448:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140302963394448:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/websocket-proxy.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 140530020640656:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140530020640656:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140530020640656:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/apache.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 140154234734480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140154234734480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140154234734480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/reports.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 140585187067792:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140585187067792:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140585187067792:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: [WARNING] Failed to read or parse '/etc/pki/ovirt-engine/keys/imageio-proxy.p12' Perhaps it was changed since last Setup. Error was: MAC verified OK Error outputting keys and certificates 139788111759248:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 139788111759248:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 139788111759248:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: ~~~ 1. Deploy new 4.3 HE environment 2. Try to update to latest minor version of RHV-M 3. Run engine-setup, it will fail like seen above. Actual results: Cannot update RHV-M in FIPS mode Expected results: Should be able to update to the latest RHV-M with FIPS enabled. Additional info: ~~~ 2019-07-29 15:11:06,840-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND --== PKI CONFIGURATION ==-- 2019-07-29 15:11:06,841-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 2019-07-29 15:11:06,842-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization 2019-07-29 15:11:06,843-0400 DEBUG otopi.context context._executeMethod:136 otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization condition False 2019-07-29 15:11:06,845-0400 DEBUG otopi.context context._executeMethod:127 Stage customization METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca.Plugin._customization_upgrade 2019-07-29 15:11:06,847-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:813 execute: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), executable='None', cwd='None', env=None 2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.executeRaw:863 execute-result: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys'), rc=1 2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:921 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stdout: 2019-07-29 15:11:06,873-0400 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:926 execute-output: ('/bin/openssl', 'pkcs12', '-in', '/etc/pki/ovirt-engine/keys/engine.p12', '-passin', 'pass:**FILTERED**', '-nokeys') stderr: MAC verified OK Error outputting keys and certificates 140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: 2019-07-29 15:11:06,874-0400 WARNING otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca ca._extractPKCS12CertificateString:130 Failed to read or parse '/etc/pki/ovirt-engine/keys/engine.p12' 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Perhaps it was changed since last Setup. 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Error was: 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND MAC verified OK 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND Error outputting keys and certificates 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:060740A0:digital envelope routines:EVP_PBE_CipherInit:unknown cipher:evp_pbe.c:181: 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error:p12_decr.c:87: 2019-07-29 15:11:06,874-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND 140048410478480:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:p12_decr.c:139: 2019-07-29 15:11:06,875-0400 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:204 DIALOG:SEND ~~~