Created attachment 1713591 [details]
ovirt-engine-setup-20200903130221-xpqnyo

Reproduced. Copying relevant error from setup log for reference:

2020-09-23 16:20:34,960+0300 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca plugin.execute:926 execute-output: ('/usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh', '--name=engine', '--password=**FILTERED**', '--subject=/C=US/O=lab.eng.tlv2.redhat.com/CN=didi-centos8-engine.lab.eng.tlv2.redhat.com', '--san=DNS:didi-centos8-engine.lab.eng.tlv2.redhat.com') stderr:
Generating RSA private key, 2048 bit long modulus (2 primes)
**********************************************************************************************************************************************************************************************************************************************************************************************************************+++++
*******************************************************************************************************************************************************+++++
e is 65537 (0x010001)
140321139988288:error:060800C8:digital envelope routines:EVP_DigestInit_ex:disabled for FIPS:crypto/evp/digest.c:135:
Cannot create certificate request

# fips-mode-setup --check
FIPS mode is enabled.

# yum list ovirt-engine
Last metadata expiration check: 0:32:51 ago on Mon 02 Nov 2020 12:08:02 PM CET.
Installed Packages
ovirt-engine.noarch                                                                                         4.4.3.9-0.1.el8ev

Created attachment 1725761 [details]
latet rhv on 4.3 candidate

as I failed to install latest engine of FIPS enabled RHEL8 machine I am re-opening it

(In reply to Pavol Brilla from comment #8)
> as I failed to install latest engine of FIPS enabled RHEL8 machine I am
> re-opening it

Are you sure this only happens on FIPS mode?

Attached setup log has:

keytool error: java.security.ProviderException: Could not initialize NSS

I logged in there and ran manually, and got:

[root@brq-ipa2 ovirt-engine]# pwd
/etc/pki/ovirt-engine
[root@brq-ipa2 ovirt-engine]# /usr/lib/jvm/jre-11-openjdk-11.0.9.11-2.el8_3.x86_64/bin/keytool -v -import -noprompt -trustcacerts -alias cacert -keypass mypass -file ca.pem -keystore .truststore -storepass mypass
keytool error: java.security.ProviderException: Could not initialize NSS
java.security.ProviderException: Could not initialize NSS
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:217)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:112)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$1.run(SunPKCS11.java:109)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.configure(SunPKCS11.java:109)
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:251)
        at java.base/sun.security.jca.ProviderConfig$3.run(ProviderConfig.java:242)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.jca.ProviderConfig.doLoadProvider(ProviderConfig.java:242)
        at java.base/sun.security.jca.ProviderConfig.getProvider(ProviderConfig.java:222)
        at java.base/sun.security.jca.ProviderList.getProvider(ProviderList.java:266)
        at java.base/sun.security.jca.ProviderList.getService(ProviderList.java:379)
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:157)
        at java.base/java.security.Security.getImpl(Security.java:717)
        at java.base/java.security.KeyStore.getInstance(KeyStore.java:875)
        at java.base/sun.security.tools.keytool.Main.doCommands(Main.java:925)
        at java.base/sun.security.tools.keytool.Main.run(Main.java:405)
        at java.base/sun.security.tools.keytool.Main.main(Main.java:398)
Caused by: java.io.FileNotFoundException: /usr/lib64/libnss3.so
        at jdk.crypto.cryptoki/sun.security.pkcs11.Secmod.initialize(Secmod.java:193)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:212)
        ... 17 more

Indeed, /usr/lib64/libnss3.so does not exist.

This file is normally part of the package 'nss'. I do not see this package anywhere in the logs on this machine, meaning it was never installed there.

I guess this is a result of some removed (indirect?) dependency somewhere.

Can you compare with some other machine (fips or not) where engine-setup does work?

AFAICT we do not require it anywhere, we just include it in OST runs
in ost-images: 
$ git grep 389-ds
template.ks.in:module --name 389-ds
template.ks.in:# for aaa tests from 389-ds module
template.ks.in:389-ds-base
template.ks.in:389-ds-base-legacy-tools
ost-images/template.ks.in:389-ds-base

on installed OST system:
# rpm -q --whatrequires nss
389-ds-base-1.4.2.4-10.module_el8.2.0+489+38ed056a.x86_64
# rpm -q --whatrequires 389-ds-base
no package requires 389-ds-base

Martine?

I've created BZ1894083 to track that issue on OpenJDK side, if keytool requires nss package library, then OpenJDK should depend on nss

I disabled fips rebooted and run engine-setup on same server
# fips-mode-setup --check
FIPS mode is disabled.
# engine-setup

.....


[ INFO  ] Execution of setup completed successfully

All is fine without FIPS...

(In reply to Martin Perina from comment #11)
> I've created BZ1894083 to track that issue on OpenJDK side, if keytool
> requires nss package library, then OpenJDK should depend on nss

Tried keytool with nss package removed. In non-fips mode, this works. So nss is not an absolute requirement.

Searched and found bug 1759335. It seems to imply that nss is indeed mandatory in fips mode.

Perhaps, instead of, or until, bug 1894083 is fixed, it's ok that the engine will require it.

Martin, WDYT?

(In reply to Yedidyah Bar David from comment #14)
> (In reply to Martin Perina from comment #11)
> > I've created BZ1894083 to track that issue on OpenJDK side, if keytool
> > requires nss package library, then OpenJDK should depend on nss
> 
> Tried keytool with nss package removed. In non-fips mode, this works. So nss
> is not an absolute requirement.

Yes, according to BZ1759335 keytool requires nss only in FIPS mode, because in FIPS it's allow to use security providers only from nss package

> 
> Searched and found bug 1759335. It seems to imply that nss is indeed
> mandatory in fips mode.
> 
> Perhaps, instead of, or until, bug 1894083 is fixed, it's ok that the engine
> will require it.
> 
> Martin, WDYT?

Posted a patch to add this requirement

Still failing:

[ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
[ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine/bin/pki-create-ca.sh' failed to execute
[ INFO  ] DNF Performing DNF transaction rollback

Created attachment 1744736 [details]
ovirt-engine  4.4.4.6-0.1.el8ev  log

(In reply to Pavol Brilla from comment #17)
> Still failing:
> 
> [ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
> [ ERROR ] Failed to execute stage 'Misc configuration': Command
> '/usr/share/ovirt-engine/bin/pki-create-ca.sh' failed to execute
> [ INFO  ] DNF Performing DNF transaction rollback

Can you please clarify the flow? The attached log seems like an upgrade, not new setup - or perhaps as a second attempt after a failed setup. If it's a legit flow, please attach all of /var/log/ovirt-engine/setup/* . Thanks.

I've tried to install latest oVirt 4.4 master on FIPS enabled host to find out gaps in our overall FIPS support and I can confirm the issue. engine-setup is failing when we are trying to add ca.pem into Java keystore:

https://github.com/oVirt/ovirt-engine/blob/master/packaging/bin/pki-create-ca.sh#L133

This code works perfecly fine on non-FIPS host, but fail with very confusing error:

keytool error: java.lang.Exception: Certificate not imported, alias <cacert> already exists
java.lang.Exception: Certificate not imported, alias <cacert> already exists
	at sun.security.tools.keytool.Main.addTrustedCert(Main.java:2869)
	at sun.security.tools.keytool.Main.doCommands(Main.java:1056)
	at sun.security.tools.keytool.Main.run(Main.java:370)
	at sun.security.tools.keytool.Main.main(Main.java:363)

The most confusing part is "alias <cacert> already exists", because I've debugged the issue and keystore file is empty. I was trying to find some information about keytool in FIPS mode and found out below:

https://bugzilla.redhat.com/show_bug.cgi?id=1759335#c2

According to above bug it seems that OpenJDK in FIPS mode doesn't support file based keystores, so we probably need to switch our certificate management to NSS completely to be able to work in both normal and FIPS mode:

https://docs.genesys.com/Documentation/PSDK/latest/Developer/TLSSecurityProviders#PKCS11_Provider:_Mozilla_NSS

If that is confirmed, then we need to perform quite large code changes ...

Additional information are provided also in https://bugzilla.redhat.com/show_bug.cgi?id=1780339

I already replaced environment, so I am not able to provide logs from that env, I did non FIPS setup, engine-cleanup and tried to do FIPS setup, I see that Martin already did more investigation meanwhile.

If needed I can respin new env and provide logs from clean env

(In reply to Martin Perina from comment #20)
> I've tried to install latest oVirt 4.4 master on FIPS enabled host to find
> out gaps in our overall FIPS support and I can confirm the issue.
> engine-setup is failing when we are trying to add ca.pem into Java keystore:
> 
> https://github.com/oVirt/ovirt-engine/blob/master/packaging/bin/pki-create-
> ca.sh#L133
> 
> This code works perfecly fine on non-FIPS host, but fail with very confusing
> error:
> 
> keytool error: java.lang.Exception: Certificate not imported, alias <cacert>
> already exists
> java.lang.Exception: Certificate not imported, alias <cacert> already exists
> 	at sun.security.tools.keytool.Main.addTrustedCert(Main.java:2869)
> 	at sun.security.tools.keytool.Main.doCommands(Main.java:1056)
> 	at sun.security.tools.keytool.Main.run(Main.java:370)
> 	at sun.security.tools.keytool.Main.main(Main.java:363)
> 
> The most confusing part is "alias <cacert> already exists", because I've
> debugged the issue and keystore file is empty. I was trying to find some
> information about keytool in FIPS mode and found out below:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1759335#c2
> 
> According to above bug it seems that OpenJDK in FIPS mode doesn't support
> file based keystores, so we probably need to switch our certificate
> management to NSS completely to be able to work in both normal and FIPS mode:
> 
> https://docs.genesys.com/Documentation/PSDK/latest/Developer/
> TLSSecurityProviders#PKCS11_Provider:_Mozilla_NSS
> 
> If that is confirmed, then we need to perform quite large code changes ...

Martin, can you please clarify the current status? Are we working on above? bug #?
Is this still expected to be included in 4.4.5?

Moving to 4.4.6/NEW for visibility. Not sure it will be done in 4.4.6.

Martin, I did not check recently the situation nor your team's work
relevant to this bug - I am Cced on encryption changed, but not sure
about the status, or whether that's relevant at all (I guess it's not,
not sure). If it's not going to be completed quickly, perhaps add more
bugs as needed and clean dependencies.

[ INFO  ] Execution of setup completed successfully
[root@10-37-138-180 ~]# fips-mode-setup --check
FIPS mode is enabled.
[root@10-37-138-180 ~]# yum list ovirt-engine
Last metadata expiration check: 1:20:47 ago on Wed 12 May 2021 12:51:02 PM CEST.
Installed Packages
ovirt-engine.noarch        4.4.6.7-0.1.el8ev

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager security update (ovirt-engine) [ovirt-4.4.6]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2179