There is several certificates which expire on 2016-04-06, i.e., they are expired today and therefore things fails in cascade.

1. Move date back before the date they expire, e.g., 2016-04-01. 
2. restart IPA. Was it successful?
3. run `getcert list` again to see status of the certs

"""
        status: CA_UNREACHABLE
        ca-error: Error 58 connecting to https://euprd-dw-ipa-02.INTERNAL.COM:8443/ca/agent/ca/profileReview: Problem with the local SSL certificate.
"""
should go away

4. if certmonger doesn't renew the certs automatically, resubmit them manually, e.g. `getcert resubmit -i $REQUEST_ID`, where request ID is in `getcert list` output.

5. move date back when all certs are renewed

Hi Petr, here is what I got:
1: Move date back
# date -s "Fri Apr 01 17:13:41 UTC 2016"
Fri Apr  1 17:13:41 UTC 2016

2: Still cannot start IPA:
# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

There is no errors in /var/log/pki/pki-tomcat/catalina*.log, and here is the log in /var/log/messages:
Apr  1 18:51:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:51:35 PM org.apache.catalina.startup.HostConfig deployDescriptor
Apr  1 18:51:35 euprd-dw-ipa-02 server: INFO: Deployment of configuration descriptor /etc/pki/pki-tomcat/Catalina/localhost/ROOT.xml has finished in 961 ms
Apr  1 18:51:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:51:35 PM org.apache.coyote.AbstractProtocol start
Apr  1 18:51:35 euprd-dw-ipa-02 server: INFO: Starting ProtocolHandler ["http-bio-8080"]
Apr  1 18:51:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:51:35 PM org.apache.coyote.AbstractProtocol start
Apr  1 18:51:35 euprd-dw-ipa-02 server: INFO: Starting ProtocolHandler ["http-bio-8443"]
Apr  1 18:51:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:51:35 PM org.apache.coyote.AbstractProtocol start
Apr  1 18:51:35 euprd-dw-ipa-02 server: INFO: Starting ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Apr  1 18:51:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:51:35 PM org.apache.catalina.startup.Catalina start
Apr  1 18:51:35 euprd-dw-ipa-02 server: INFO: Server startup in 2515 ms
Apr  1 18:52:07 euprd-dw-ipa-02 systemd: Starting Cleanup of Temporary Directories...
Apr  1 18:52:07 euprd-dw-ipa-02 systemd: Started Cleanup of Temporary Directories.
Apr  1 18:55:01 euprd-dw-ipa-02 systemd: Started Session 10 of user root.
Apr  1 18:55:01 euprd-dw-ipa-02 systemd: Starting Session 10 of user root.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopping Kerberos 5 KDC...
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopped Kerberos 5 KDC.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopping Kerberos 5 Password-changing and Administration...
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: kadmin.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopped Kerberos 5 Password-changing and Administration.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Unit kadmin.service entered failed state.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: kadmin.service failed.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopping Berkeley Internet Name Domain (DNS) with native PKCS#11...
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: received control channel command 'stop'
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: shutting down: flushing changes
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: stopping command channel on 127.0.0.1#953
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone 0.10.in-addr.arpa/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone 0.0.10.in-addr.arpa/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone 10.0.10.in-addr.arpa/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone 0.2.10.in-addr.arpa/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone 95.17.52.in-addr.arpa/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: zone internal.com/IN: shutting down
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: no longer listening on ::#53
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: no longer listening on 127.0.0.1#53
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: no longer listening on 10.0.10.118#53
Apr  1 18:56:33 euprd-dw-ipa-02 named-pkcs11[5296]: exiting
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopped Berkeley Internet Name Domain (DNS) with native PKCS#11.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopping IPA memcached daemon, increases IPA server performance...
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopped IPA memcached daemon, increases IPA server performance.
Apr  1 18:56:33 euprd-dw-ipa-02 systemd: Stopping The Apache HTTP Server...
Apr  1 18:56:34 euprd-dw-ipa-02 systemd: Stopped The Apache HTTP Server.
Apr  1 18:56:34 euprd-dw-ipa-02 systemd: Stopped target PKI Tomcat Server.
Apr  1 18:56:34 euprd-dw-ipa-02 systemd: Stopping PKI Tomcat Server.
Apr  1 18:56:34 euprd-dw-ipa-02 systemd: Stopping PKI Tomcat Server pki-tomcat...
Apr  1 18:56:34 euprd-dw-ipa-02 systemd: Closed ipa-otpd socket.
Apr  1 18:56:34 euprd-dw-ipa-02 server: Java virtual machine used: /usr/lib/jvm/jre/bin/java
Apr  1 18:56:34 euprd-dw-ipa-02 server: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Apr  1 18:56:34 euprd-dw-ipa-02 server: main class used: org.apache.catalina.startup.Bootstrap
Apr  1 18:56:34 euprd-dw-ipa-02 server: flags used: -DRESTEASY_LIB=/usr/share/java/resteasy-base
Apr  1 18:56:34 euprd-dw-ipa-02 server: options used: -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Apr  1 18:56:34 euprd-dw-ipa-02 server: arguments used: stop
Apr  1 18:56:35 euprd-dw-ipa-02 systemd: Stopped IPA key daemon.
Apr  1 18:56:35 euprd-dw-ipa-02 systemd: Stopping 389 Directory Server INTERNAL-COM....
Apr  1 18:56:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:56:35 PM org.apache.catalina.core.StandardServer await
Apr  1 18:56:35 euprd-dw-ipa-02 server: INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Apr  1 18:56:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:56:35 PM org.apache.coyote.AbstractProtocol pause
Apr  1 18:56:35 euprd-dw-ipa-02 server: INFO: Pausing ProtocolHandler ["http-bio-8080"]
Apr  1 18:56:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:56:35 PM org.apache.coyote.AbstractProtocol pause
Apr  1 18:56:35 euprd-dw-ipa-02 server: INFO: Pausing ProtocolHandler ["http-bio-8443"]
Apr  1 18:56:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:56:35 PM org.apache.coyote.AbstractProtocol pause
Apr  1 18:56:35 euprd-dw-ipa-02 server: INFO: Pausing ProtocolHandler ["ajp-bio-127.0.0.1-8009"]
Apr  1 18:56:35 euprd-dw-ipa-02 server: Apr 01, 2016 6:56:35 PM org.apache.catalina.core.StandardService stopInternal
Apr  1 18:56:35 euprd-dw-ipa-02 server: INFO: Stopping service Catalina
Apr  1 18:56:35 euprd-dw-ipa-02 systemd: Stopped PKI Tomcat Server pki-tomcat.
Apr  1 18:56:37 euprd-dw-ipa-02 systemd: Stopped 389 Directory Server INTERNAL-COM..

But if I "CTRL-C" in in pki-tomcat step, I can get this:
[root@euprd-dw-ipa-02 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful

3. Then `getcert list` will show this after `service certmonger restart`:
[root@euprd-dw-ipa-02 ipa]# getcert list
Number of certificates and requests being tracked: 8.
Request ID '20150702220651':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-INTERNAL-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-INTERNAL-COM/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/dirsrv/slapd-INTERNAL-COM',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=euprd-dw-ipa-02.internal.com,O=INTERNAL.COM
        expires: 2017-07-02 18:20:15 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv INTERNAL-COM
        track: yes
        auto-renew: yes
Request ID '20150702220859':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=euprd-dw-ipa-02.internal.com,O=INTERNAL.COM
        expires: 2017-07-02 18:20:16 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/lib64/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
Request ID '20160426180612':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=CA Audit,O=INTERNAL.COM
        expires: 2016-04-06 00:06:22 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20160426180613':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=OCSP Subsystem,O=INTERNAL.COM
        expires: 2016-04-06 00:06:20 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        eku: id-kp-OCSPSigning
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20160426180614':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=CA Subsystem,O=INTERNAL.COM
        expires: 2016-04-06 00:06:21 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20160426180615':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=Certificate Authority,O=INTERNAL.COM
        expires: 2034-04-17 00:06:19 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20160426180616':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=IPA RA,O=INTERNAL.COM
        expires: 2016-04-06 00:07:26 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
Request ID '20160426180617':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=euprd-dw-ipa-02.internal.com,O=INTERNAL.COM
        expires: 2017-06-21 22:08:02 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
        track: yes
        auto-renew: yes

4: I can do auto renew or manually renew, but the expire date is still 20160406:
[root@euprd-dw-ipa-02 pki-tomcat]# ipa-cacert-manage renew
Renewing CA certificate, please wait
CA certificate successfully renewed
The ipa-cacert-manage command was successful

[root@euprd-dw-ipa-02 pki-tomcat]# getcert resubmit -i 20160426180612
Resubmitting "20160426180612" to "dogtag-ipa-ca-renew-agent".
[root@euprd-dw-ipa-02 pki-tomcat]# getcert resubmit -i 20160426180613
Resubmitting "20160426180613" to "dogtag-ipa-ca-renew-agent".
[root@euprd-dw-ipa-02 pki-tomcat]# getcert resubmit -i 20160426180614
Resubmitting "20160426180614" to "dogtag-ipa-ca-renew-agent".
[root@euprd-dw-ipa-02 pki-tomcat]# getcert resubmit -i 20160426180616
Resubmitting "20160426180616" to "dogtag-ipa-ca-renew-agent".

[root@euprd-dw-ipa-02 pki-tomcat]# getcert list  | grep expire
        expires: 2017-07-02 18:20:15 UTC
        expires: 2017-07-02 18:20:16 UTC
        expires: 2016-04-06 00:06:22 UTC
        expires: 2016-04-06 00:06:20 UTC
        expires: 2016-04-06 00:06:21 UTC
        expires: 2034-04-17 00:06:19 UTC
        expires: 2016-04-06 00:07:26 UTC
        expires: 2017-06-21 22:08:02 UTC

5. If I move date back, some of the certificate status will change back to "SUBMITTING", even after auto or manually renew.
[root@euprd-dw-ipa-02 ~]# hwclock -s
[root@euprd-dw-ipa-02 ~]# date
Wed Apr 27 19:06:02 UTC 2016

For example:
[root@euprd-dw-ipa-02 pki-tomcat]# ipa-cacert-manage renew
Renewing CA certificate, please wait
CA certificate successfully renewed
The ipa-cacert-manage command was successful
[root@euprd-dw-ipa-02 pki-tomcat]# getcert resubmit -i 20160426180616
Resubmitting "20160426180616" to "dogtag-ipa-ca-renew-agent".
[root@euprd-dw-ipa-02 pki-tomcat]# getcert list
...
Request ID '20160426180616':
        status: SUBMITTING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=INTERNAL.COM
        subject: CN=IPA RA,O=INTERNAL.COM
        expires: 2016-04-06 00:07:26 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
        post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
        track: yes
        auto-renew: yes
...

And IPA still cannot restart.

Any further suggestions? Thanks!

Hi Petr, good news!

I used a previous AWS AMI backup of this IPA server to test the above solution and it works! I can change the date then renew the certificate and IPA server can start after change the date back!

[root@euprd-dw-ipa-02 secaops]# getcert list | grep expire
        expires: 2017-07-02 18:20:15 UTC
        expires: 2018-03-23 00:01:26 UTC
        expires: 2018-03-23 00:00:54 UTC
        expires: 2018-03-22 23:59:53 UTC
        expires: 2036-04-01 23:59:51 UTC
        expires: 2017-06-21 22:08:02 UTC
        expires: 2017-07-02 18:20:16 UTC
        expires: 2018-03-23 00:00:04 UTC


Here is the IPA version in this backup:
4.1.0-18

So it worth for you to check why 4.2 failed the auto and manual renew of the certificates. Might be a bug? 

Thanks a lot for your help!

I'm glad that it works.

As for the possible bug, without pki-tomcatd service running(in comment 3) renewal of certs won't work. 

Do you have logs from that time available? We might see there some issue. I'm interested especially in:
* /var/log/pki/pki-tomcat/ca/debug 
* /var/log/pki/pki-tomcat/ca/selftests.log
* journalctl -u pki-tomcatd

We can't do much without the logs.

Hi Petr,

I don't have that VM anymore, but I attached the disk and already sent the logs to you in email. Please check and let me know if anything else.

Thanks again for your help!

Looking at the logs. selftest failed there. From debug log, we can see that there is a validity issue with subsystemCert:

"""
[05/Apr/2016:23:18:11][localhost-startStop-1]: CertUtils: verifySystemCerts() cert tag=subsystem
[05/Apr/2016:23:18:11][localhost-startStop-1]: CertUtils: verifySystemCertByTag(subsystem)
[05/Apr/2016:23:18:11][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(subsystemCert cert-pki-ca,SSLClient)
[05/Apr/2016:23:18:11][localhost-startStop-1]: CertUtils: verifySystemCertByNickname(): calling isCertValid()
[05/Apr/2016:23:18:11][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: subsystemCert cert-pki-ca
[05/Apr/2016:23:18:11][localhost-startStop-1]: SignedAuditEventFactory: create() message=[AuditEvent=CIMC_CERT_VERIFICATION][SubjectID=$System$][Outcome=Failure][CertNickName=subsystemCert cert-pki-ca] CIMC certificate verification
"""

In the info in comment we see that it expires on:
  expires: 2016-04-06 00:06:21 UTC

The log time is 2016-04-05 23:18:11 which is less than 1h before expiration. PKI debug logs shows local time, not UTC time. What is the time difference/time zone?

# date
# date --utc

Hi Petr, this server use UTC and never changed:

[root@euprd-dw-ipa-02 ca]# date
Fri Apr 29 23:22:55 UTC 2016
[root@euprd-dw-ipa-02 ca]# date --utc
Fri Apr 29 23:22:58 UTC 2016

Moving to pki-core component to let PKI developer find out if there is anything to fix.

From my perspective, lookig at self test with lines:

 SystemCertsVerification: system certs verification failure
 SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED!

I'd like to see in the log what is wrong and with which cert. From debug log it seems to me that if fails somewhere around X509CertImpl.checkValidity and probably swallows CertificateExpiredException or CertificateNotYetValidException. I would bet on CertificateExpiredException.

Upstream ticket:
https://fedorahosted.org/pki/ticket/850

Per Bug Triage of 05/03/2016: RHEL 7.4

NOTE: It is believed that this problem may be related to upstream ticket
      https://fedorahosted.org/pki/ticket/850 - JSS certificate validation
      does not pass up exact error from NSS, but for now, we have chosen to
      move this issue to RHEL 7.4.

Upstream ticket:
https://fedorahosted.org/pki/ticket/850

PKI was modified to utilize the new JSS functionality to show which certificate is failing and also the validation error message in the following commit:
* 2c73f1c2721021755d5753f07fa059a018ae9d7f

Hello Petr or who concerns:

I got the same issue in another IPA system, tried everything above but still cannot renew the certificates, do you have any suggestions for how to fix it?

[root@usdev-ops-ipa-01 ~]# rpm -qa | grep ipa
sssd-ipa-1.13.0-40.el7_2.1.x86_64
ipa-admintools-4.2.0-15.el7_2.3.x86_64
python-iniparse-0.4-9.el7.noarch
libipa_hbac-1.13.0-40.el7_2.1.x86_64
python-libipa_hbac-1.13.0-40.el7_2.1.x86_64
ipa-python-4.2.0-15.el7_2.3.x86_64
ipa-client-4.2.0-15.el7_2.3.x86_64
ipa-server-4.2.0-15.el7_2.3.x86_64
ipa-server-dns-4.2.0-15.el7_2.3.x86_64
redhat-access-plugin-ipa-0.9.1-2.el7.noarch
[root@usdev-ops-ipa-01 ~]# getcert list | grep expire
        expires: 2018-01-09 00:40:45 UTC
        expires: 2016-05-03 21:14:05 UTC
        expires: 2016-05-03 21:14:01 UTC
        expires: 2016-05-03 21:14:03 UTC
        expires: 2034-05-14 21:13:57 UTC
        expires: 2017-12-29 00:52:21 UTC
        expires: 2018-01-09 00:40:48 UTC
        expires: 2016-05-03 21:15:03 UTC
[root@usdev-ops-ipa-01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Just to clarify, the patch in comment #16 provides additional information in the selftest log to troubleshoot startup issues (which might be environment issue).

It can be verified as follows:

1. Install a basic CA.
2. Shutdown the CA.
3. Do something to make one of the system certificates invalid (e.g. changing system date past expiration, removing trust flags, removing the certificate).
4. Restart the CA. The CA should fail to start (although Tomcat itself might still start just fine).
5. Check the selftest log. It should contain certificate validation error message provided by JSS/NSS.

Testing based on devel notes comment #21

Test steps:

1. Stop CA instance. Change the system date to verify certificates are expired.
2. Start the CA instance.
3. Check the selftest.log and debug logs.


Test Case 1: Verify the selftest and debug logs.

Selftest logs:

0.localhost-startStop-1 - [01/Jan/2022:00:00:34 EST] [20] [1] CAPresence:  CA is present
0.localhost-startStop-1 - [01/Jan/2022:00:00:34 EST] [20] [1] SystemCertsVerification: system certs verification failure: Certificate ocspSigningCert cert-topology-02-CA CA is invalid: Invalid certificate: (-8181) Peer's Certificate has expired.
0.localhost-startStop-1 - [01/Jan/2022:00:00:34 EST] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED!

debug:

[01/Jan/2022:00:00:34][localhost-startStop-1]: CertUtils: verifySystemCertByNickname() failed: java.lang.Exception: Certificate ocspSigningCert cert-topology-02-CA CA is invalid: Invalid certificate: (-8181) Peer's Certificate has expired.
[01/Jan/2022:00:00:34][localhost-startStop-1]: CertUtils: verifySystemCertsByTag() failed: java.lang.Exception: Certificate ocspSigningCert cert-topology-02-CA CA is invalid: Invalid certificate: (-8181) Peer's Certificate has expired.
[01/Jan/2022:00:00:34][localhost-startStop-1]: SignedAuditEventFactory: create() message created for eventType=CIMC_CERT_VERIFICATION

[01/Jan/2022:00:00:34][localhost-startStop-1]: SignedAuditEventFactory: create() message created for eventType=CIMC_CERT_VERIFICATION

java.lang.Exception: Certificate ocspSigningCert cert-topology-02-CA CA is invalid: Invalid certificate: (-8181) Peer's Certificate has expired.
Caused by: java.security.cert.CertificateException: Invalid certificate: (-8181) Peer's Certificate has expired.
        at org.mozilla.jss.CryptoManager.verifyCertificateNowNative2(Native Method)
        at org.mozilla.jss.CryptoManager.verifyCertificate(CryptoManager.java:1637)
        at com.netscape.cmscore.cert.CertUtils.verifySystemCertByNickname(CertUtils.java:840)
        ... 44 more

Test Case 2: To verify the CA subsystem status.Since CA is not up ideally tomcat should also be stopped or inactive.


[root@host-8-177-84 ca]# systemctl status pki-tomcatd -l
● pki-tomcatd - PKI Tomcat Server topology-02-CA
   Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-01-01 00:12:55 EST; 7min ago
  Process: 17459 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
  Process: 17500 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS)
 Main PID: 17710 (java)
   CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd
           └─17710 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/topology-02-CA -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/topology-02-CA/temp -Djava.util.logging.config.file=/var/lib/pki/topology-02-CA/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/topology-02-CA/conf/catalina.policy org.apache.catalina.startup.Bootstrap start

Jan 01 00:18:08 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Jan 01 00:18:08 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Jan 01 00:18:08 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Jan 01 00:18:08 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: at java.lang.Thread.run(Thread.java:748)
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: ContainerBackgroundProcessor[StandardEngine[Catalina]]: Failed to flush log "/var/lib/pki/topology-02-CA/logs/ca/signedAudit/ca_audit", error: Attempt to log message "/var/lib/pki/topology-02-CA/logs/ca/signedAudit/ca_audit" to closed log file 0.ContainerBackgroundProcessor[StandardEngine[Catalina]] - [01/Jan/2022:00:18:19 EST] [14] [6] [AuditEvent=AUDIT_LOG_SIGNING][SubjectID=$System$][Outcome=Success] signature of audit buffer just flushed: sig: PC7uEu1KZkg8/Q2RyHdl1uExoqrlrS0VkZEqFnWfdzUK23S1zXYfQbj3QC5Y0HtvijLzpsTVwN168+zc1XtQZNzAQtRknZi7eUIyBD5vUgJkd1aRd1VCdvJsQLexm3q1h+N0I64kaLHnESmlp8EEPnrBYl7hD1bEUcb4lCw//YtKKeIlg7lnig89tj0bjhcI8SyqjtO1X8OSbmsMTM8fX0jrpEIhCXE+mgP6Zkj83EssVM+/b66+dh5xDn+PhhyVZQlDwtkGTpXAFH1IXnpjtmcanJ42bkdKsgPfmMQcbwdiBRBmSJN3YZ2PgLLbajM3NfAALtVovHM4iCom7kef/w==
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: SSLAuthenticatorWithFallback: Stopping authenticators
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: SEVERE: The web application [/ca] appears to have started a thread named [LDAPConnThread-22 ldaps://pki1.example.com:3389] but has failed to stop it. This is very likely to create a memory leak.
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: SEVERE: The web application [/ca] appears to have started a thread named [LDAPConnThread-26 ldaps://pki1.example.com:3389] but has failed to stop it. This is very likely to create a memory leak.
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: SEVERE: The web application [/ca] appears to have started a thread named [authorityMonitor] but has failed to stop it. This is very likely to create a memory leak.
Jan 01 00:18:19 host-8-177-84.host.centralci.eng.rdu2.redhat.com server[17710]: SSLAuthenticatorWithFallback: Setting container

Test Case 3: Test to disable the ca subsystem using

pki-server subsystem-disable -i topology-02-CA ca
----------------------------------
Subsystem "ca" is already disabled
----------------------------------
  Subsystem ID: ca
  Instance ID: topology-02-CA
  Enabled: False

so ideally tomcat should also be inactive.

-- > if we run subsystem-enable using

pki-server subsystem-enable -i topology-02-CA ca----------------------
Enabled "ca" subsystem
----------------------
  Subsystem ID: ca
  Instance ID: topology-02-CA
  Enabled: True

It says it enable "CA" not sure how it happens because certificate-validate is failing.

[root@host-8-177-84 ca]# pki-server subsystem-cert-validate -i topology-02-CA ca
  Cert ID: signing
  Nickname: caSigningCert cert-topology-02-CA CA
  Usage: SSLCA
  Token: Internal Key Storage Token
  Status: INVALID

  Cert ID: ocsp_signing
  Nickname: ocspSigningCert cert-topology-02-CA CA
  Usage: StatusResponder
  Token: Internal Key Storage Token
  Status: INVALID

  Cert ID: sslserver
  Nickname: Server-Cert cert-topology-02-CA
  Usage: SSLServer
  Token: Internal Key Storage Token
  Status: INVALID

  Cert ID: subsystem
  Nickname: subsystemCert cert-topology-02-CA
  Usage: SSLClient
  Token: Internal Key Storage Token
  Status: INVALID

  Cert ID: audit_signing
  Nickname: auditSigningCert cert-topology-02-CA CA
  Usage: ObjectSigner
  Token: Internal Key Storage Token
  Status: INVALID
-----------------
Validation failed


Test case 4: debug logs shows:

[01/Jan/2022:00:00:36][localhost-startStop-1]: returnConn: mNumConns now 3
Invalid class name repositorytop
        at com.netscape.cmscore.dbs.DBRegistry.createObject(DBRegistry.java:485)
        at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:167)
        at com.netscape.cmscore.dbs.DBSSession.read(DBSSession.java:137)
        at com.netscape.cmscore.dbs.Repository.getSerialNumber(Repository.java:125)
        at com.netscape.cmscore.dbs.Repository.initCache(Repository.java:244)
        at com.netscape.cmscore.dbs.Repository.checkRanges(Repository.java:460)
        at com.netscape.cmscore.apps.CMSEngine.startup(CMSEngine.java:1377)
        at com.netscape.certsrv.apps.CMS.startup(CMS.java:201)
        at com.netscape.certsrv.apps.CMS.start(CMS.java:1622)
        at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
        at javax.servlet.GenericServlet.init(GenericServlet.java:158)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)

Test Case 5: To test the recovery of CA when all certs are valid.


[root@host-8-177-84 ca]# date
Mon Jan  1 00:00:12 EST 2018
[root@host-8-177-84 ca]# pki-server subsystem-cert-validate -i topology-02-CA ca
  Cert ID: signing
  Nickname: caSigningCert cert-topology-02-CA CA
  Usage: SSLCA
  Token: Internal Key Storage Token
  Status: VALID

  Cert ID: ocsp_signing
  Nickname: ocspSigningCert cert-topology-02-CA CA
  Usage: StatusResponder
  Token: Internal Key Storage Token
  Status: VALID

  Cert ID: sslserver
  Nickname: Server-Cert cert-topology-02-CA
  Usage: SSLServer
  Token: Internal Key Storage Token
  Status: VALID

  Cert ID: subsystem
  Nickname: subsystemCert cert-topology-02-CA
  Usage: SSLClient
  Token: Internal Key Storage Token
  Status: VALID

  Cert ID: audit_signing
  Nickname: auditSigningCert cert-topology-02-CA CA
  Usage: ObjectSigner
  Token: Internal Key Storage Token
  Status: VALID
--------------------
Validation succeeded
--------------------

--> Try to restart CA now.It worked and we are able to submit cert request.

Questions:

1. If you refer test case 2,3 and 4 , i think this behavior is not right.

Just to clarify, Tomcat is an application server (i.e. container) which can run multiple web applications including CA subsystem, KRA subsystem, etc.

For case #2, if a subsystem selftest fails (e.g. due to expired certificate) then only that subsystem should be shutdown. The container and other web applications should not be affected. Ideally there should be a CLI to check individual subsystem status, for example:

  $ pki ca-info
    Status: RUNNING

The systemctl status should be used to check the container status, not the subsystem status.

Similarly, for case #3 the pki-server subsystem-disable is only used to disable a subsystem, not the container, so Tomcat should not be affected.

For case #4, I'm not sure the cause of "Invalid class name repositorytop" error message. It could be just a secondary failure due to expired certificate. If this can be reproduced consistently please open a separate bug.

In case #1 the debug log shows which certificate has a problem and what kind of problem it has, which can help troubleshoot the problem. I think this is sufficient to verify this ticket.

Just FYI, in bug #1454471 all subsystems are now always enabled on startup to avoid confusion. If the selftest fails the subsystem will be disabled automatically like before.

marking this bug verified.Need to verify bug #1454471  also as part of testing

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2110