Bug 1390319
| Summary: | Failed to start pki-tomcatd Service ("ipa-cacert-manage renew" failed?) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Tom Lavigne <tlavigne> |
| Component: | pki-core | Assignee: | RHCS Maintainers <rhcs-maint> |
| Status: | CLOSED ERRATA | QA Contact: | Asha Akkiangady <aakkiang> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 7.2 | CC: | alee, arubin, cfu, edewata, ekeck, ftweedal, jmagne, ksiddiqu, lmgnid, mharmsen, nkinder, pvoborni, rcritten, rpattath, xdong |
| Target Milestone: | rc | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | pki-core-10.3.3-12.el7_3 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1330800 | Environment: | |
| Last Closed: | 2016-12-06 17:04:58 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1330800 | ||
| Bug Blocks: | |||
|
Description
Tom Lavigne
2016-10-31 17:01:42 UTC
Trying to verify following steps on https://bugzilla.redhat.com/show_bug.cgi?id=1330800#c21 : 1. Install a basic CA. 2. Shutdown the CA. 3. Do something to make one of the system certificates invalid (e.g. changing system date past expiration, removing trust flags, removing the certificate). 4. Restart the CA. The CA should fail to start (although Tomcat itself might still start just fine). 5. Check the selftest log. It should contain certificate validation error message provided by JSS/NSS. I had trouble with step 5, selftest log wasn't updated with any error msgs and I tried both changing system date past expiration and removing the certificate. Am I missing something? [root@auto-hv-02-guest05 ~]# rpm -q pki-ca pki-ca-10.3.3-13.el7_3.noarch [root@auto-hv-02-guest05 ~]# certutil -d /etc/pki/pki-tomcat/alias/ -L Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u caSigningCert cert-pki-ca CTu,Cu,Cu Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca u,u,Pu [root@auto-hv-02-guest05 ~]# ipactl stop Stopping ipa-dnskeysyncd Service Stopping ipa-otpd Service Stopping pki-tomcatd Service Stopping ntpd Service Stopping ipa-custodia Service Stopping httpd Service Stopping ipa_memcached Service Stopping named Service Stopping kadmin Service Stopping krb5kdc Service Stopping Directory Service ipa: INFO: The ipactl command was successful [root@auto-hv-02-guest05 ~]# certutil -D -d /etc/pki/pki-tomcat/alias/ -n "caSigningCert cert-pki-ca" [root@auto-hv-02-guest05 ~]# ipactl start Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Starting ipa_memcached Service Starting httpd Service Starting ipa-custodia Service Starting ntpd Service Starting pki-tomcatd Service Failed to start pki-tomcatd Service Shutting down Hint: You can use --ignore-service-failure option for forced start in case that a non-critical service failed Aborting ipactl [root@auto-hv-02-guest05 ~]# grep "verification failure" /var/log/pki/pki-tomcat/ca/selftests.log [root@auto-hv-02-guest05 ~]# (In reply to Xiyang Dong from comment #4) > Trying to verify following steps on > https://bugzilla.redhat.com/show_bug.cgi?id=1330800#c21 : > > 1. Install a basic CA. > 2. Shutdown the CA. > 3. Do something to make one of the system certificates invalid (e.g. > changing system date past expiration, removing trust flags, removing the > certificate). > 4. Restart the CA. The CA should fail to start (although Tomcat itself might > still start just fine). > 5. Check the selftest log. It should contain certificate validation error > message provided by JSS/NSS. > > I had trouble with step 5, selftest log wasn't updated with any error msgs > and I tried both changing system date past expiration and removing the > certificate. > > Am I missing something? > > [root@auto-hv-02-guest05 ~]# rpm -q pki-ca > pki-ca-10.3.3-13.el7_3.noarch > [root@auto-hv-02-guest05 ~]# certutil -d /etc/pki/pki-tomcat/alias/ -L > > Certificate Nickname Trust Attributes > > SSL,S/MIME,JAR/XPI > > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > caSigningCert cert-pki-ca CTu,Cu,Cu > Server-Cert cert-pki-ca u,u,u > auditSigningCert cert-pki-ca u,u,Pu > [root@auto-hv-02-guest05 ~]# ipactl stop > Stopping ipa-dnskeysyncd Service > Stopping ipa-otpd Service > Stopping pki-tomcatd Service > Stopping ntpd Service > Stopping ipa-custodia Service > Stopping httpd Service > Stopping ipa_memcached Service > Stopping named Service > Stopping kadmin Service > Stopping krb5kdc Service > Stopping Directory Service > ipa: INFO: The ipactl command was successful > [root@auto-hv-02-guest05 ~]# certutil -D -d /etc/pki/pki-tomcat/alias/ -n > "caSigningCert cert-pki-ca" > > [root@auto-hv-02-guest05 ~]# ipactl start > Starting Directory Service > Starting krb5kdc Service > Starting kadmin Service > Starting named Service > Starting ipa_memcached Service > Starting httpd Service > Starting ipa-custodia Service > Starting ntpd Service > Starting pki-tomcatd Service > Failed to start pki-tomcatd Service > Shutting down > Hint: You can use --ignore-service-failure option for forced start in case > that a non-critical service failed > Aborting ipactl > [root@auto-hv-02-guest05 ~]# grep "verification failure" > /var/log/pki/pki-tomcat/ca/selftests.log > [root@auto-hv-02-guest05 ~]# I see the same when the caSigning cert is deleted and Ca is restarted [root@mgmt3 ~]# systemctl stop pki-tomcatd@pki-ca. [root@mgmt3 ~]# certutil -D -d /var/lib/pki/pki-ca/alias/ -n "caSigningCert cert-pki-ca CA" [root@mgmt3 ~]# systemctl start pki-tomcatd@pki-ca [root@mgmt3 ~]# certutil -L -d /var/lib/pki/pki-ca/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca CA u,u,Pu ocspSigningCert cert-pki-ca CA u,u,u subsystemCert cert-pki-ca u,u,u There were no updates in the selftest log [root@mgmt3 ~]# tail -f /var/log/pki/pki-ca/ca/selftests.log 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instances 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] CAPresence: CA is present 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SystemCertsVerification: system certs verification success 0.localhost-startStop-1 - [08/Nov/2016:09:17:18 EST] [20] [1] SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at startup! When the trust of caSigning cert was changed instead of deleting it, there was a failure message in the selftest log [root@mgmt3 ~]# certutil -M -d /var/lib/pki/pki-ca/alias/ -n "caSigningCert cert-pki-ca CA" -t ",," [root@mgmt3 ~]# certutil -L -d /var/lib/pki/pki-ca/alias/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Server-Cert cert-pki-ca u,u,u auditSigningCert cert-pki-ca CA u,u,Pu caSigningCert cert-pki-ca CA u,u,u ocspSigningCert cert-pki-ca CA u,u,u subsystemCert cert-pki-ca u,u,u [root@mgmt3 ~]# systemctl start pki-tomcatd@pki-ca [root@mgmt3 ~]# tail -f /var/log/pki/pki-ca/ca/selftests.log 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instances 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] CAPresence: CA is present 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SystemCertsVerification: system certs verification success 0.localhost-startStop-1 - [08/Nov/2016:09:28:16 EST] [20] [1] SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at startup! 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: Initializing self test plugins: 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instances 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] CAPresence: CA is present 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SystemCertsVerification: system certs verification failure: Certificate caSigningCert cert-pki-ca CA is invalid: Invalid certificate: (-8172) Peer's certificate issuer has been marked as not trusted by the user. 0.localhost-startStop-1 - [08/Nov/2016:09:30:46 EST] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! Apparently if the CA signing certificate is removed the server fails to start before the selftest gets executed, so the selftest log does not change. Here is the exception in the debug log:
[08/Nov/2016:19:08:12][localhost-startStop-1]: CRLIssuingPoint:initConfig: mUnexpectedExceptionLoopMax set to 10
java.lang.NullPointerException
at com.netscape.ca.CRLIssuingPoint.initConfig(CRLIssuingPoint.java:747)
at com.netscape.ca.CRLIssuingPoint.init(CRLIssuingPoint.java:480)
at com.netscape.ca.CertificateAuthority.initCRL(CertificateAuthority.java:2152)
at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:590)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1169)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1075)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:581)
at com.netscape.certsrv.apps.CMS.init(CMS.java:187)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1616)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
The missing certificate is actually first detected in SigningUnit.init() but the error was ignored. This issue should be fixed as part of https://fedorahosted.org/pki/ticket/2400.
If the CA signing certificate is expired or not trusted, the selftest does get executed and it will show the corresponding error message:
* Peer's Certificate has expired.
* Peer's certificate issuer has been marked as not trusted by the user.
So I think the patch for this bug is working correctly.
[root@vm-idm-023 ~]# rpm -qi pki-ca Name : pki-ca Version : 10.3.3 Release : 13.el7_3 Architecture: noarch Install Date: Fri 04 Nov 2016 07:14:31 PM IST Group : System Environment/Daemons Size : 2431509 License : GPLv2 Signature : (none) Source RPM : pki-core-10.3.3-13.el7_3.src.rpm Build Date : Fri 04 Nov 2016 07:09:03 AM IST Build Host : ppc-016.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://pki.fedoraproject.org/ Summary : Certificate System - Certificate Authority Verification steps: When the caSigning cert was expired or the caSigning cert was not trusted, the following error messages were seen in the self test log 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: Initializing self test plugins: 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: loading all self test plugin instances 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: loading self test plugins in startup order 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded! 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup: 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] CAPresence: CA is present 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SystemCertsVerification: system certs verification failure: Certificate ocspSigningCert cert-pki-ca CA is invalid: Invalid certificate: (-8181) Peer's Certificate has expired. 0.localhost-startStop-1 - [08/Nov/2021:00:01:27 IST] [20] [1] SelfTestSubsystem: The CRITICAL self test plugin called selftests.container.instance.SystemCertsVerification running at startup FAILED! The results in comment 6 were also observed I am still unable to see update on selftest.log:
[root@auto-hv-02-guest05 ~]# ipactl stop
Stopping ipa-dnskeysyncd Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping ntpd Service
Stopping ipa-custodia Service
Stopping httpd Service
Stopping ipa_memcached Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
Stopping Directory Service
ipa: INFO: The ipactl command was successful
[root@auto-hv-02-guest05 ~]# certutil -L -d /var/lib/pki/pki-tomcat/ca/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
[root@auto-hv-02-guest05 ~]# certutil -M -d /var/lib/pki/pki-tomcat/ca/alias/ -n "caSigningCert cert-pki-ca" -t ",,"
[root@auto-hv-02-guest05 ~]# certutil -L -d /var/lib/pki/pki-tomcat/ca/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
caSigningCert cert-pki-ca u,u,u
[root@auto-hv-02-guest05 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Hint: You can use --ignore-service-failure option for forced start in case that a non-critical service failed
[root@auto-hv-02-guest05 ~]# grep "verification failure" /var/log/pki/pki-tomcat/ca/selftests.log
[root@auto-hv-02-guest05 ~]#
Nor I can see debug log mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1390319#c6
Verified on pki-ca-10.3.3-13.el7_3:
1. Install a basic CA.
2. Shutdown the CA.
3. remove trust flags for audit signing cert
4. Restart the CA.
5. Check the selftest log. It should contain certificate validation error
message provided by JSS/NSS.
[root@auto-hv-02-guest05 ~]# ipactl stop
Stopping ipa-dnskeysyncd Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping ntpd Service
Stopping ipa-custodia Service
Stopping httpd Service
Stopping ipa_memcached Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
Stopping Directory Service
ipa: INFO: The ipactl command was successful
[root@auto-hv-02-guest05 ~]# certutil -L -d /var/lib/pki/pki-tomcat/ca/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u
Server-Cert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
caSigningCert cert-pki-ca CTu,Cu,Cu
[root@auto-hv-02-guest05 ~]# certutil -M -d /var/lib/pki/pki-tomcat/ca/alias/ -n "auditSigningCert cert-pki-ca" -t ",,"
[root@auto-hv-02-guest05 ~]# certutil -L -d /var/lib/pki/pki-tomcat/ca/alias/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca u,u,u
subsystemCert cert-pki-ca u,u,u
Server-Cert cert-pki-ca u,u,u
caSigningCert cert-pki-ca CTu,Cu,Cu
auditSigningCert cert-pki-ca u,u,u
[root@auto-hv-02-guest05 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@auto-hv-02-guest05 ~]# grep "verification failure" /var/log/pki/pki-tomcat/ca/selftests.log
0.localhost-startStop-1 - [09/Nov/2016:12:31:28 EST] [20] [1] SystemCertsVerification: system certs verification failure: Certificate auditSigningCert cert-pki-ca is invalid: Invalid certificate: (-8101) Certificate type not approved for application.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-2881.html |