Bug 1933349

Description of problem:

may be similar or related to:
bz 1779984 - The ipa-cert-fix command failed. [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/27-renewed.crt'
  https://bugzilla.redhat.com/1779984 [RHEL-8.?][ASSIGNED]
  https://pagure.io/freeipa/issue/8721
  https://github.com/freeipa/freeipa/pull/5579/commits/82bb4280ac5b576c54d4211b399eb7e6a9bbc331
( this seem different:
bz 1930586 - 'pki-server cert-fix' fails when CS.cfg parameter selftests.container.order.startup not present
  https://bugzilla.redhat.com/1930586 [RHEL-8.?][NEW]
  https://github.com/dogtagpki/pki/pull/3466
)

in this case:

RHEL-7.9 FIPS enabled

valid: caSigningCert cert-pki-ca
expired: ocspSigningCert cert-pki-ca
expired: subsystemCert cert-pki-ca
expired: auditSigningCert cert-pki-ca
valid renewed 20210205161709 : Server-Cert cert-pki-ca ( unclear if this was an action from certmonger or ipa-cert-fix or manual getcert resubmit, no traces in the collected CA debug or transations log files)
expired: /var/lib/ipa/ra-agent.pem
expired: LDAP SSL server cert
expired: HTTPD SSL server cert
expired: /var/kerberos/krb5kdc/kdc.crt

this article may have been followed at some point:
How do I manually renew Identity Management (IPA) certificates on RHEL7 after they have expired? (Master IPA Server)
https://access.redhat.com/solutions/3357261

certmonger is broken:
Feb 25 14:22:15 redacted certmonger: 2021-02-25 14:22:15 [116070] Server at https://redacted/ipa/xml failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction, explaining:  Peer's Certificate has expired.).

ipa-cert-fix if failing to renew the IPA expired internal certificates ( see the "Additional info" section for the whole verbose detail information):
ipa-cert-fix -v
...
ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: IOError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'


checked /etc/pki/pki-tomcat/ca/CS.cfg has:
selftests.container.order.startup=SystemCertsVerification:critical, CAPresence:critical

checked the IPA LDAP server config till has
nsslapd-port: 389
nsslapd-security: on
but that does not really matter since the LDAP server SSL cert is expired, and ipa-cert-fix uses LDAPI, which is available.


I may have missed something, could not locate what is populating the /etc/pki/pki-tomcat/certs/{}.crt the logic behind, nor find out why the .crt are missing
I seem to only see in ./freeipa-4.6.8/ipaserver/install/ipa_cert_fix.py , use like:
cert = x509.load_certificate_from_file(cert_path)
cert_path = "/etc/pki/pki-tomcat/certs/{}-renewed.crt" \

and copy and install of the cert file, not many traces of the .crt files.

but seem well related to
  https://pagure.io/freeipa/issue/8721
  https://github.com/freeipa/freeipa/pull/5579/commits/82bb4280ac5b576c54d4211b399eb7e6a9bbc331


Version-Release number of selected component (if applicable):

389-ds-base-1.3.10.2-9.el7_9.x86_64
certmonger-0.78.4-14.el7.x86_64
ipa-server-4.6.8-5.el7.x86_64
pki-ca-10.5.18-7.el7.noarch
redhat-release-server-7.9-5.el7_9.x86_64
FIPS ENABLED


How reproducible:
N/A

Steps to Reproduce:
1. N/A
2.
3.


Actual results:
ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: IOError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'


Expected results:
yes


Additional info:

The following certificates will be renewed:

Dogtag subsystem certificate:
  Subject: CN=CA Subsystem,O=REDACTED
  Serial:  4
  Expires: 2020-09-30 20:37:35

Dogtag ca_ocsp_signing certificate:
  Subject: CN=OCSP Subsystem,O=REDACTED
  Serial:  2
  Expires: 2020-09-30 20:37:35

Dogtag ca_audit_signing certificate:
  Subject: CN=CA Audit,O=REDACTED
  Serial:  5
  Expires: 2020-09-30 20:37:35

IPA IPA RA certificate:
  Subject: CN=IPA RA,O=REDACTED
  Serial:  7
  Expires: 2020-09-30 20:38:06

IPA Apache HTTPS certificate:
  Subject: CN=redacted,O=REDACTED
  Serial:  9
  Expires: 2020-10-11 20:39:45

IPA LDAP certificate:
  Subject: CN=redacted,O=REDACTED
  Serial:  8
  Expires: 2020-10-11 20:39:03

IPA KDC certificate:
  Subject: CN=redacted,O=REDACTED
  Serial:  10
  Expires: 2020-10-11 20:40:01

Enter "yes" to proceed: yes
Proceeding.
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=pki-server cert-fix --ldapi-socket /var/run/slapd-REDACTED.socket --agent-uid ipara --cert subsystem --cert ca_ocsp_signing --cert ca_audit_signing --extra-cert 7 --extra-cert 9 --extra-cert 8 --extra-cert 10
ipapython.ipautil: DEBUG: Process finished, return code=1
ipapython.ipautil: DEBUG: stdout=ERROR: 404 Client Error: Not Found

ipapython.ipautil: DEBUG: stderr=INFO: Loading password config: /etc/pki/pki-tomcat/password.conf
INFO: Fixing the following system certs: ['subsystem', 'ca_ocsp_signing', 'ca_audit_signing']
INFO: Renewing the following additional certs: ['7', '9', '8', '10']
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Stopping the instance to proceed with system cert renewal
INFO: Configuring LDAP password authentication
INFO: Setting pkidbuser password via ldappasswd
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Selftests disabled for subsystems: ca
INFO: Resetting password for uid=ipara,ou=people,o=ipaca
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
INFO: Starting the instance
INFO: Sleeping for 10 seconds to allow server time to start...
INFO: Requesting new cert for subsystem
INFO: Getting subsystem cert info for ca
INFO: Trying to setup a secure connection to CA subsystem.
INFO: Starting new HTTPS connection (1): redacted.redacted
INFO: Stopping the instance
INFO: Selftests enabled for subsystems: ca
INFO: Restoring previous LDAP configuration

Renewed Dogtag subsystem certificate:
  Subject: CN=redacted,O=REDACTED
  Serial:  3
  Expires: 2021-01-20 14:36:10

ipapython.admintool: DEBUG:   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 128, in run
    replicate_dogtag_certs(subject_base, ca_subject_dn, certs)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py", line 251, in replicate_dogtag_certs
    cert = x509.load_certificate_from_file(cert_path)
  File "/usr/lib/python2.7/site-packages/ipalib/x509.py", line 425, in load_certificate_from_file
    with open(filename, mode='rb') as f:

ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: IOError: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/ca_ocsp_signing.crt'
ipapython.admintool: ERROR: The ipa-cert-fix command failed.