Bug 1552159 - Installation on FIPS enabled rhel7 failing with with certutil issues
Summary: Installation on FIPS enabled rhel7 failing with with certutil issues
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer
Version: Nightly
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: Released
Assignee: satellite6-bugs
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks: 1170174
TreeView+ depends on / blocked
 
Reported: 2018-03-06 15:33 UTC by Peter Ondrejka
Modified: 2019-10-07 17:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:37:00 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:37:15 UTC

Description Peter Ondrejka 2018-03-06 15:33:15 UTC
Description of problem:

Installation of katello from upstream repos on rhel7 in FIPS mode, foreman-installer --scenario katello fails on executing certutil. This issue occurs twice during the course of installation, once called by Candlepin, once by Qpid:


[ WARN 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
[ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0]
[ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0]


[ WARN 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
[ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]
[ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]

When running the command manually on the fips machine, I'm prompted for a password:

~]# certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'
Enter Password or Pin for "NSS FIPS 140-2 Certificate DB":

This problem does not occur when installing nightly on non-FIPS rhel

Version-Release number of selected component (if applicable):

foreman-1.18.0-0.develop.201802231059git8bd79a1.el7.noarch
katello-3.7.0-1.nightly.el7.noarch

How reproducible:
always

Comment 2 Ivan Necas 2018-03-08 10:03:14 UTC
It seems like the FIPS mode makes the certutils to enforce the password protection and the fix should be to add the passphrase protection when calling the certutils commands

Comment 3 Ivan Necas 2018-03-08 10:04:21 UTC
See https://bugzilla.redhat.com/show_bug.cgi?id=1401606 for more details

Comment 9 Kevin Jones 2018-08-16 12:49:30 UTC
Has anyone figured this out? I'm seeing the same failure with qpidd on a STIGed RHEL 7 and Sat 6.3 installation.

qpidd failing to start

I thought at first this was due to FIPS mode being enabled [1]. However, after disabling FIPS [2] and reboot, still get same error. The box has been STIGed via OpenSCAP.

From /var/log/foreman-installer/satellite.log
[DEBUG 2018-08-16 00:04:20 main] Exit with status code: 6 (signal was 6)
[ERROR 2018-08-16 00:04:20 main] Errors encountered during run:
[ERROR 2018-08-16 00:04:20 main]  Systemd start for qpidd failed!
[ERROR 2018-08-16 00:04:20 main] journalctl log for qpidd:
[ERROR 2018-08-16 00:04:20 main] -- Logs begin at Wed 2018-08-15 23:49:47 PDT, end at Thu 2018-08-16 00:03:14 PDT. --
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com qpidd[15896]: 2018-08-16 00:00:53 [Broker] critical Broker (pid=15896) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed.
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.

[root@dmcisdnirh01v ~]# systemctl status qpidd
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-08-16 00:03:14 PDT; 5h 38min ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
 Main PID: 18720 (code=exited, status=1/FAILURE)

Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed.

[1] https://access.redhat.com/solutions/3524491
[2] https://access.redhat.com/solutions/2422061

Comment 10 Kevin K. 2018-09-04 18:50:17 UTC
Kevin Jones,
We ran into the same thing; however we also had some SELinux [1] issues. I noticed that if I ran into the error described above, THEN disabled FIPS and tried again, I would continually get this error. However, on a clean install, disabling FIPS prior to `satellite-installer` everything worked.

Ultimately, this is the solution that worked for us:

- Install RHEL 7.5 with DISA STIG security profile
- Set SELinux to `permissive` mode
- Reboot
- Go through Satellite 6.3.2 install process UNTIL `./install_packages`
- Normal `./install_packages`
- Disable FIPS mode [2]
- Reboot
- `satellite-installer --scenario satellite`
- Re-Enable FIPS mode (essentially [2] in reverse, and things like `--remove-args=fips=1` -> `--args=fips=1`)
- Reboot
- Set SELinux to `enforcing`
- Reboot
- Continue with satellite install as normal...

There's probably some unnecessary reboots in there, but oh well.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1489762
[2] https://access.redhat.com/solutions/2422061

Comment 11 Peter Ondrejka 2018-10-30 13:13:30 UTC
Verified on Satellite 6.5 snap 1 on fips-enabled rhel 7.5, installation proceeds as expected, no certutil issues experienced.

Comment 15 errata-xmlrpc 2019-05-14 12:37:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.