Bug 1552159

Summary:	Installation on FIPS enabled rhel7 failing with with certutil issues
Product:	Red Hat Satellite	Reporter:	Peter Ondrejka <pondrejk>
Component:	Installation	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED ERRATA	QA Contact:	Peter Ondrejka <pondrejk>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	Nightly	CC:	bkearney, ehelms, inecas, kejones, knapp, mcressma, pcreech, smercurio
Target Milestone:	6.5.0	Keywords:	PrioBumpQA, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-14 12:37:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1170174

Description Peter Ondrejka 2018-03-06 15:33:15 UTC

Description of problem:

Installation of katello from upstream repos on rhel7 in FIPS mode, foreman-installer --scenario katello fails on executing certutil. This issue occurs twice during the course of installation, once called by Candlepin, once by Qpid:


[ WARN 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
[ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0]
[ERROR 2018-03-05T03:36:19 verbose] /Stage[main]/Certs::Candlepin/Certs::Ssltools::Certutil[amqp-client]/Exec[amqp-client]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'' returned 255 instead of one of [0]


[ WARN 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]/returns: certutil: could not authenticate to token NSS FIPS 140-2 Certificate DB.: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
[ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: Failed to call refresh: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]
[ERROR 2018-03-05T03:36:20 verbose] /Stage[main]/Certs::Qpid/Certs::Ssltools::Certutil[ca]/Exec[ca]: 'certutil -A -d '/etc/pki/katello/nssdb' -n 'ca' -t 'TCu,Cu,Tuw' -a -i '/etc/pki/katello/certs/katello-default-ca.crt'' returned 255 instead of one of [0]

When running the command manually on the fips machine, I'm prompted for a password:

~]# certutil -A -d '/etc/pki/katello/nssdb' -n 'amqp-client' -t ',,' -a -i '/etc/pki/katello/certs/java-client.crt'
Enter Password or Pin for "NSS FIPS 140-2 Certificate DB":

This problem does not occur when installing nightly on non-FIPS rhel

Version-Release number of selected component (if applicable):

foreman-1.18.0-0.develop.201802231059git8bd79a1.el7.noarch
katello-3.7.0-1.nightly.el7.noarch

How reproducible:
always

Comment 2 Ivan Necas 2018-03-08 10:03:14 UTC

It seems like the FIPS mode makes the certutils to enforce the password protection and the fix should be to add the passphrase protection when calling the certutils commands

Comment 3 Ivan Necas 2018-03-08 10:04:21 UTC

See https://bugzilla.redhat.com/show_bug.cgi?id=1401606 for more details

Comment 4 Ivan Necas 2018-03-08 10:12:44 UTC

The BZ suggests using --empty-password https://bugzilla.redhat.com/show_bug.cgi?id=1401606#c3 when creating the nssdb https://github.com/theforeman/puppet-certs/blob/f4152faff300dbd1ff972e5575e3934e4a8f9a98/manifests/ssltools/nssdb.pp#L34

Comment 9 Kevin Jones 2018-08-16 12:49:30 UTC

Has anyone figured this out? I'm seeing the same failure with qpidd on a STIGed RHEL 7 and Sat 6.3 installation.

qpidd failing to start

I thought at first this was due to FIPS mode being enabled [1]. However, after disabling FIPS [2] and reboot, still get same error. The box has been STIGed via OpenSCAP.

From /var/log/foreman-installer/satellite.log
[DEBUG 2018-08-16 00:04:20 main] Exit with status code: 6 (signal was 6)
[ERROR 2018-08-16 00:04:20 main] Errors encountered during run:
[ERROR 2018-08-16 00:04:20 main]  Systemd start for qpidd failed!
[ERROR 2018-08-16 00:04:20 main] journalctl log for qpidd:
[ERROR 2018-08-16 00:04:20 main] -- Logs begin at Wed 2018-08-15 23:49:47 PDT, end at Thu 2018-08-16 00:03:14 PDT. --
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:52 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com qpidd[15896]: 2018-08-16 00:00:53 [Broker] critical Broker (pid=15896) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:00:53 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed.
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Broker (pid=17050) start-up failed: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com qpidd[17050]: 2018-08-16 00:02:17 [Broker] critical Unexpected error: Couldn't find any network address to listen to
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
[ERROR 2018-08-16 00:04:20 main] Aug 16 00:02:17 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.

[root@dmcisdnirh01v ~]# systemctl status qpidd
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2018-08-16 00:03:14 PDT; 5h 38min ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
 Main PID: 18720 (code=exited, status=1/FAILURE)

Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Started An AMQP message broker daemon..
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Starting An AMQP message broker daemon....
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Broker (pid=18720) start-up failed: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com qpidd[18720]: 2018-08-16 00:03:14 [Broker] critical Unexpected error: Couldn't find any network address to listen to
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service: main process exited, code=exited, status=1/FAILURE
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: Unit qpidd.service entered failed state.
Aug 16 00:03:14 dmcisdnirh01v.dmci-isf.com systemd[1]: qpidd.service failed.

[1] https://access.redhat.com/solutions/3524491
[2] https://access.redhat.com/solutions/2422061

Comment 10 Kevin K. 2018-09-04 18:50:17 UTC

Kevin Jones,
We ran into the same thing; however we also had some SELinux [1] issues. I noticed that if I ran into the error described above, THEN disabled FIPS and tried again, I would continually get this error. However, on a clean install, disabling FIPS prior to `satellite-installer` everything worked.

Ultimately, this is the solution that worked for us:

- Install RHEL 7.5 with DISA STIG security profile
- Set SELinux to `permissive` mode
- Reboot
- Go through Satellite 6.3.2 install process UNTIL `./install_packages`
- Normal `./install_packages`
- Disable FIPS mode [2]
- Reboot
- `satellite-installer --scenario satellite`
- Re-Enable FIPS mode (essentially [2] in reverse, and things like `--remove-args=fips=1` -> `--args=fips=1`)
- Reboot
- Set SELinux to `enforcing`
- Reboot
- Continue with satellite install as normal...

There's probably some unnecessary reboots in there, but oh well.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1489762
[2] https://access.redhat.com/solutions/2422061

Comment 11 Peter Ondrejka 2018-10-30 13:13:30 UTC

Verified on Satellite 6.5 snap 1 on fips-enabled rhel 7.5, installation proceeds as expected, no certutil issues experienced.

Comment 15 errata-xmlrpc 2019-05-14 12:37:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222