Bug 1616134

Summary: Unable to install with admin-generated keys
Product: Red Hat Enterprise Linux 7 Reporter: Asha Akkiangady <aakkiang>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED ERRATA QA Contact: Asha Akkiangady <aakkiang>
Severity: high Docs Contact: Marc Muehlfeld <mmuehlfe>
Priority: high    
Version: 7.5CC: cpelland, dpunia, edewata, lmiksik, mharmsen, msauton, rpattath
Target Milestone: rcKeywords: TestCaseProvided, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pki-core-10.5.16-2.el7 Doc Type: Enhancement
Doc Text:
.The `pkispawn` utility now supports using keys created in the NSS database during CA, KRA, and OCSP installations Previously, during a Certificate System installation, the pkispawn utility only supported creating new keys and importing existing keys for system certificates. With this enhancement, pkispawn now supports using keys the administrator generates directly in the NSS database during certificate authority (CA), key recovery authority (KRA), and online certificate status protocol (OCSP) installations.
Story Points: ---
Clone Of:
: 1656297 (view as bug list) Environment:
Last Closed: 2019-08-06 13:07:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1656297    

Description Asha Akkiangady 2018-08-15 03:25:38 UTC
Description of problem:
OCSP pkispawn installation with admin-generated keys instead of pkispawn-generated keys throws null pointer exception. 

Version-Release number of selected component (if applicable):
pki-server-10.5.1-15.el7_5.noarch

How reproducible:
Always

Steps to Reproduce:
The issue is reproducible in both ECC and RSA installations.
1. Install RootCA. Create a CMC SharedToken.
2. Install OCSP step 1 without generating csr. 
3. Generate csr using PKCS10Client command.  Use CMCShared token method to generate certificates. 
4. Configure pkispawn file with csr and cert files generated in step 3 and run OCSP pkispawn step 2. 


Actual results:
pkispawn fails with following error:

pkispawn    : DEBUG    ........... <?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><State>0</State><Type>OCSP</Type><Status>running</Status><Version>10.5.1-14.el7_5</Version></XMLResponse>
pkispawn    : INFO     ....... constructing PKI configuration data.
pki.nssdb   : DEBUG    Command: certutil -L -d /opt/pki-ECC-masterOCSP/ocsp/alias -f /tmp/tmpzcBIlw/password.txt -n PKI OCSP Administrator for ECC-Non-TMS-OCSP -a
pkispawn    : INFO     ....... configuring PKI configuration data.

Installation failed:
<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - java.lang.NullPointerException</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>java.lang.NullPointerException</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
	org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
	org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
	org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
	org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
	org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
	java.security.AccessController.doPrivileged(Native Method)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
</pre></p><p><b>root cause</b> <pre>java.lang.NullPointerException
	org.mozilla.jss.util.Base64OutputStream.write(Base64OutputStream.java:86)
	com.netscape.cmsutil.crypto.CryptoUtil.base64Encode(CryptoUtil.java:1074)
	org.dogtagpki.server.rest.SystemConfigService.processCert(SystemConfigService.java:443)
	org.dogtagpki.server.rest.SystemConfigService.processCerts(SystemConfigService.java:303)
	org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:166)
	org.dogtagpki.server.rest.SystemConfigService.configure(SystemConfigService.java:101)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
	org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
	org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
	org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
	org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
	org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
	java.security.AccessController.doPrivileged(Native Method)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
	org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
	org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
	org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:260)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>

Please check the OCSP logs in /var/log/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/ocsp.

=============================================================
OSCP debug log has this:
14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/logs/autoShutdown.crumb
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: cert not found:auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: Exception:org.mozilla.jss.crypto.ObjectNotFoundException: Certificate not found: auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: done init id=debug
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: initialized debug
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: initSubsystem id=log
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: ready to init id=log
[14/Aug/2018:11:22:00][localhost-startStop-1]: Event filters:
[14/Aug/2018:11:22:00][localhost-startStop-1]:  - RANDOM_GENERATION: (Outcome=Failure)
[14/Aug/2018:11:22:00][localhost-startStop-1]:  - SELFTESTS_EXECUTION: (Outcome=Failure)
[14/Aug/2018:11:22:00][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/logs/ocsp/signedAudit/ocsp_cert-ocsp_audit)
[14/Aug/2018:11:22:00][localhost-startStop-1]: Event filters:
[14/Aug/2018:11:22:00][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/logs/ocsp/system)
[14/Aug/2018:11:22:00][localhost-startStop-1]: Event filters:
[14/Aug/2018:11:22:00][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/logs/ocsp/transactions)
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11/logs/autoShutdown.crumb
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: cert not found:auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
[14/Aug/2018:11:22:00][localhost-startStop-1]: CMSEngine: Exception:org.mozilla.jss.crypto.ObjectNotFoundException: Certificate not found: auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11
==================================================
OCSP's db has audit signing cert imported with correct nickname that was given in pkispawn config "ocsp_audit_signing_ecc_nontms_aakkiang-nocp11":

# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

ocsp_audit_signing_ecc_nontms_aakkiang-nocp11                ,,P  
ocsp_sslserver_ecc_nontms_aakkiang-nocp11                    CTu,Cu,Cu
CA Signing Certificate - Example-rhcs93-ECC-CA               CT,C,C
caSigningCert-ECC-NonTMS-SubCA-aakkiang-nocp11               CT,C,C
===================================================

The problem could be that pkispawn was run with "ocsp_audit_signing_ecc_nontms_aakkiang-nocp11" in the config, but the server was looking for "auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11" instead.

Expected results:
pkispawn of OCSP should succeed. 

Additional info:
A possible workaround that QE can try: running pkispawn with a nickname that matches the one it's looking for, i.e. "auditSigningCert cert-rhcs93-ECC-NonTMS-OCSP-aakkiang-nocp11".

Note: The problem probably happens on all subsystems too, not just ocsp.

Comment 3 Endi Sukma Dewata 2018-08-29 13:24:43 UTC
The patches are available here:
https://github.com/dogtagpki/pki/pull/37

A COPR build is available here:
https://copr.devel.redhat.com/coprs/edewata/pki-10.5/build/22038/

The documentation is available here:
http://www.dogtagpki.org/wiki/Installing_OCSP_with_Custom_Keys

Comment 11 errata-xmlrpc 2019-08-06 13:07:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2228