Description of problem: Installation of katello from upstream repos on rhel7 in FIPS mode, foreman-installer --scenario katello fails on executing certutil. This issue occurs twice during the course of installation, once called by Candlepin, once by Qpid: [ WARN 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect. [ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0] [ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0] [ WARN 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect. [ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0] [ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0] When running the command manually on the fips machine, I'm prompted for a password: ~]# certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt' Enter Password or Pin for "NSS FIPS 140-2 Certificate DB": This problem does not occur when installing nightly on non-FIPS rhel Version-Release number of selected component (if applicable): foreman-1.18.0-0.develop.201802231059git8bd79a1.el7.noarch katello-3.7.0-1.nightly.el7.noarch How reproducible: always
It seems like the FIPS mode makes the certutils to enforce the password protection and the fix should be to add the passphrase protection when calling the certutils commands
See https://bugzilla.redhat.com/show_bug.cgi?id=1401606 for more details
The BZ suggests using --empty-password https://bugzilla.redhat.com/show_bug.cgi?id=1401606#c3 when creating the nssdb https://github.com/theforeman/puppet-certs/blob/f4152faff300dbd1ff972e5575e3934e4a8f9a98/manifests/ssltools/nssdb.pp#L34
Has anyone figured this out? I'm seeing the same failure with qpidd on a STIGed RHEL 7 and Sat 6.3 installation. qpidd failing to start I thought at first this was due to FIPS mode being enabled [1]. However, after disabling FIPS [2] and reboot, still get same error. The box has been STIGed via OpenSCAP. From /var/log/foreman-installer/satellite.log [DEBUG 2018-08-16 00:04:20 main] Exit with status code: 6 (signal was 6) [ERROR 2018-08-16 00:04:20 main] Errors encountered during run: [ERROR 2018-08-16 00:04:20 main] Systemd start for qpidd failed! [ERROR 2018-08-16 00:04:20 main] journalctl log for qpidd: [ERROR 2018-08-16 00:04:20 main] -- Logs begin at Wed 2018-08-15 23:49:47 PDT, end at Thu 2018-08-16 00:03:14 PDT. -- [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon.. [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon.... [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com qpidd[15896]: 2018-08-16 00:00:53 [Broker] critical Broker (pid=15896) start-up failed: Couldn't find any network address to listen to [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state. [ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed. [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon.. [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon.... [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE [ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state. [root@dmcisdnirh01v ~]# systemctl status qpidd ● qpidd.service - An AMQP message broker daemon. Loaded: loaded (/usr/lib/systemd/system/qpidd.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2018-08-16 00:03:14 PDT; 5h 38min ago Docs: man:qpidd(1) http://qpid.apache.org/ Main PID: 18720 (code=exited, status=1/FAILURE) Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon.. Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon.... Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state. Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed. [1] https://access.redhat.com/solutions/3524491 [2] https://access.redhat.com/solutions/2422061
Kevin Jones, We ran into the same thing; however we also had some SELinux [1] issues. I noticed that if I ran into the error described above, THEN disabled FIPS and tried again, I would continually get this error. However, on a clean install, disabling FIPS prior to `satellite-installer` everything worked. Ultimately, this is the solution that worked for us: - Install RHEL 7.5 with DISA STIG security profile - Set SELinux to `permissive` mode - Reboot - Go through Satellite 6.3.2 install process UNTIL `./install_packages` - Normal `./install_packages` - Disable FIPS mode [2] - Reboot - `satellite-installer --scenario satellite` - Re-Enable FIPS mode (essentially [2] in reverse, and things like `--remove-args=fips=1` -> `--args=fips=1`) - Reboot - Set SELinux to `enforcing` - Reboot - Continue with satellite install as normal... There's probably some unnecessary reboots in there, but oh well. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1489762 [2] https://access.redhat.com/solutions/2422061
Verified on Satellite 6.5 snap 1 on fips-enabled rhel 7.5, installation proceeds as expected, no certutil issues experienced.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:1222