Checked in upstream:

commit fe5fb947c91f27f240a2a5ea50be5e6da40f8294
Author: Alexander Scheel <ascheel>
Date:   Thu Jan 30 09:52:58 2020 -0600

    Fix FIPS detection
    
    The original FIPS detection code fails on python3:
    
        $ python3
        Python 3.7.6 (default, Dec 19 2019, 22:52:49)
        >>> '0' == b'0'
        False
    
    This is because bytes and strings are not directly comparable in all
    scenarios, so the comparison now returns false. Python3's subprocess
    also returns bytes in most scenarios:
    
    > By default, this function will return the data as encoded bytes. The
    > actual encoding of the output data may depend on the command being
    > invoked, so the decoding to text will often need to be handled at the
    > application level.
    
    This results in PKI incorrectly believing that it is in FIPS mode,
    when it really isn't.
    
    Signed-off-by: Alexander Scheel <ascheel>

Tested on:
[root@pki1 test_dir]# rpm -qi pki-ca
Name        : pki-ca
Version     : 10.8.1
Release     : 1.module+el8.2.0+5696+9d4c4fa5
Architecture: noarch
Install Date: Wed 12 Feb 2020 02:38:35 AM EST
Group       : Unspecified
Size        : 2640622
License     : GPLv2 and LGPLv2
Signature   : RSA/SHA256, Tue 11 Feb 2020 05:07:44 AM EST, Key ID 199e2f91fd431d51
Source RPM  : pki-core-10.8.1-1.module+el8.2.0+5696+9d4c4fa5.src.rpm
Build Date  : Mon 10 Feb 2020 02:54:46 PM EST
Build Host  : arm64-025.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.dogtagpki.org/
Summary     : PKI CA Package

Test Procedure:
https://bugzilla.redhat.com/show_bug.cgi?id=1656786#c0

Proof of concept:

# Validated on fips enabled machine:

1. Check fips:
[root@pki1 test_dir]# cat /proc/sys/crypto/fips_enabled 
1

2. Check pkispawn log:

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             kraadmin
      Administrator's PKCS #12 file:
            /opt/topology-01-KRA/kra_admin_cert.p12
INFO: FIPS mode is enabled

      This KRA subsystem of the 'pki-tomcat' instance
      has FIPS mode enabled on this operating system.

      REMINDER:  Don't forget to update the appropriate FIPS
                 algorithms in server.xml in the 'pki-tomcat' instance.

      To check the status of the subsystem:
            systemctl status pki-tomcatd

      To restart the subsystem:
            systemctl restart pki-tomcatd

      The URL for the subsystem is:
            https://pki1.example.com:8443/kra

      PKI instances will be enabled upon system boot


# Validated on fips disabled machine:

1. Check fips:
[root@pki1 ~]# cat /proc/sys/crypto/fips_enabled
0

2. Check pkispawn:

    ==========================================================================
                                INSTALLATION SUMMARY
    ==========================================================================

      Administrator's username:             caadmin
      Administrator's PKCS #12 file:
            /opt/topology-01-CA/ca_admin_cert.p12
INFO: FIPS mode is not enabled

      To check the status of the subsystem:
            systemctl status pki-tomcatd

      To restart the subsystem:
            systemctl restart pki-tomcatd

      The URL for the subsystem is:
            https://pki1.example.com:8443/ca

      PKI instances will be enabled upon system boot

    ==========================================================================

As per the POC, it can be seen that the check is now working as expected.
Hence, Marking this Bugzilla verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1644