Bug 1258630

Summary: Upgraded CA lacks ca.sslserver.certreq in CS.cfg
Product: Red Hat Enterprise Linux 7 Reporter: Matthew Harmsen <mharmsen>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED ERRATA QA Contact: Asha Akkiangady <aakkiang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: alee, cheimes, mharmsen, nkinder, rpattath, spoore
Target Milestone: rc   
Target Release: 7.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pki-core-10.2.5-6.el7 Doc Type: Bug Fix
Doc Text:
PKI server stores a copy of the system certificates in each subsystem as a cache. The cache could sometimes become outdated or missing. This update provides a utility to restore the cache data. For detailed instructions on how to check and update the cache, see the following Knowledgebase article: https://access.redhat.com/articles/1993023
 Story Points: ---
Clone Of:
: 1266634 1349710 (view as bug list) Environment:
Last Closed: 2015-11-19 09:23:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1258964, 1266634, 1349710    
Attachments:
Description Flags
pki-edewata-0641-RHEL-1-Added-CLI-to-update-cert-data-and-request-in-CS.cfg.patch
none
pki-edewata-0642-RHEL-1-Added-support-for-secure-database-connection-in-CLI.patch none

Description Matthew Harmsen 2015-08-31 20:27:48 UTC 
I have an IPA deployment that went from 3.3 to 4.2 upgrades over several years. The latest setup has no ca.sslserver.certreq parameter in CS.cfg. This breaks KRA deployment as ipa-kra-install attempts to fetch the certreq and fails, causing null pointer exception.

Set of upgrades:

# for i in 49 47 46 45 30 27 ; do dnf history info $i | grep -A1 pki-base ; done
Upgraded pki-base-10.2.6-4.fc22.noarch     @updates-testing
Upgrade           10.2.7-0.2.fc22.noarch   @vakwetu-dogtag_10.2.7_test_builds
Upgraded pki-base-10.2.6-2.fc22.noarch     @updates-testing
Upgrade           10.2.6-4.fc22.noarch     @updates-testing
Upgraded   pki-base-10.2.6-1.fc22.noarch   @updates-testing
Upgrade             10.2.6-2.fc22.noarch   @updates-testing
Upgraded pki-base-10.2.5-1.fc22.noarch     (unknown)
Upgrade           10.2.6-1.fc22.noarch     @updates-testing
Upgraded pki-base-10.0.5-1.fc19.noarch     @updates-testing/19
Upgrade           10.0.6-1.fc19.noarch     @updates-testing/19
Upgraded pki-base-10.0.4-2.fc19.noarch     (unknown)
Upgrade           10.0.5-1.fc19.noarch     @updates-testing/19
Comment 1 Matthew Harmsen 2015-08-31 20:29:09 UTC 
Edewata wrote:

There should be a copy of the cert request stored in the database, but currently there is no tool to find the exact cert request for a given cert. A manual search probably can be done using ldapsearch. A better way is to use the CLI proposed in ticket #1552.

What is the urgency of this ticket?
Comment 2 Matthew Harmsen 2015-08-31 20:30:05 UTC 
abbra replied:

So I went ahead and generated certificate requests based on the existing certificates using 'openssl x509 -x509toreq'. I needed to create certreq entries for 'Server-Cert cert-pki-ca' and 'subsystemCert cert-pki-ca'.

This allowed me to proceed few steps more. Step [2/7] failed because pki was unable to create ipakra user:

2015-08-12T09:52:52Z DEBUG Starting external process
2015-08-12T09:52:52Z DEBUG args='/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'
2015-08-12T09:52:54Z DEBUG Process finished, return code=255
2015-08-12T09:52:54Z DEBUG stdout=
2015-08-12T09:52:54Z DEBUG stderr=ProcessingException: Unable to invoke request

2015-08-12T09:52:54Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 416, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 406, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/krainstance.py", line 299, in __add_ra_user_to_agent_group
    ipautil.run(args)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 373, in run
    raise CalledProcessError(p.returncode, arg_string, stdout)
CalledProcessError: Command ''/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'' returned non-zero exit status 255

2015-08-12T09:52:54Z DEBUG   [error] CalledProcessError: Command ''/usr/bin/pki' '-d' '/tmp/tmp-AdxN38' '-c' 'XXXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'' returned non-zero exit status 255
2015-08-12T09:52:54Z ERROR 
Your system may be partly configured.
Run ipa-kra-install --uninstall to clean up.

I think upgradability of PKI has to be reworked. We need to ensure upgrade from IPA 3.3 to 4.2 via all intermediate versions is working.
Comment 3 Matthew Harmsen 2015-08-31 20:31:24 UTC 
Edewata added:

Just a note, the cert requests can also be obtained as follows:

    http://pki.fedoraproject.org/wiki/Recovery#ca.sslserver.certreq
    http://pki.fedoraproject.org/wiki/Recovery#ca.subsystem.certreq 

The installation now fails to set up a KRA agent at the "kra-user-add" step. Please follow this instruction to set up the KRA agent manually: http://pki.fedoraproject.org/wiki/IPA_Development#KRA_Agent_Setup

If the manual step also fails, please repeat the last command in verbose mode and post the output, for example:

$ pki -v -c Secret123 -n ipa-ca-agent kra-user-add ipakra --fullName "IPA KRA User"
Comment 4 Matthew Harmsen 2015-08-31 20:32:02 UTC 
abbra responded:



manual creation fails now with following output:

Server URI: http://id.vda.li:8080
Client security database: /tmp/tmp-FFo4P9
Message format: null
Command: kra-user-add ipakra --fullName "IPA KRA User"
Initializing client security database
Logging into security token
Module: kra
HTTP request: GET /kra/rest/account/login HTTP/1.1
  Accept-Encoding: gzip, deflate
  Accept: application/xml
  Host: id.vda.li:8080
  Connection: Keep-Alive
  User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.8.0_51)
HTTP response: HTTP/1.1 302 Found
  Server: Apache-Coyote/1.1
  Cache-Control: private
  Expires: Thu, 01 Jan 1970 00:00:00 GMT
  Location: https://id.vda.li:8443/kra/rest/account/login
  Content-Length: 0
  Date: Thu, 13 Aug 2015 11:56:47 GMT
HTTP redirect: https://id.vda.li:8443/kra/rest/account/login
Client certificate: ipa-ca-agent
HTTP request: GET /kra/rest/account/login HTTP/1.1
  Accept-Encoding: gzip, deflate
  Accept: application/xml
  Host: id.vda.li:8443
  Connection: Keep-Alive
  User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.8.0_51)
Server certificate: CN=id.vda.li,O=VDA.LI
javax.ws.rs.ProcessingException: Unable to invoke request
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:102)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:62)
	at com.sun.proxy.$Proxy23.login(Unknown Source)
	at com.netscape.certsrv.account.AccountClient.login(AccountClient.java:45)
	at com.netscape.certsrv.client.SubsystemClient.login(SubsystemClient.java:49)
	at com.netscape.cmstools.cli.KRACLI.login(KRACLI.java:50)
	at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:54)
	at com.netscape.cmstools.cli.CLI.execute(CLI.java:337)
	at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:557)
	at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:569)
Caused by: java.io.IOException: SocketException cannot write on socket
	at org.mozilla.jss.ssl.SSLSocket.write(SSLSocket.java:1099)
	at org.mozilla.jss.ssl.SSLOutputStream.write(SSLOutputStream.java:56)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:166)
	at org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:272)
	at org.apache.http.impl.AbstractHttpClientConnection.flush(AbstractHttpClientConnection.java:277)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.flush(ManagedClientConnectionImpl.java:181)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:240)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:883)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
	... 11 more

Tomcat does respond on port 8443 and attempt to GET /kra/rest/account/login returns 401 (unauthorized), asking to login.
Comment 5 Matthew Harmsen 2015-08-31 20:33:14 UTC 
edewata asked:

Could you verify that the CA admin certificate is still valid?

$ certutil -L -d ~/.dogtag/nssdb -n ipa-ca-agent

Per IRC discussion the CA admin certificate has actually expired. Currently certmonger does not support tracking certificate in PKCS #12 file. A manual renewal may be necessary.
Comment 6 Matthew Harmsen 2015-08-31 20:34:29 UTC 
abbra replied:

 I've put ca-agent.p12 to NSS database in /etc/pki/tls/private/ipa-ca with pk12util and then ran renewal via certmonger using agent CA interface:

 getcert request -d /etc/pki/tls/private/ipa-ca -p /etc/pki/tls/private/ipa-ca/pwdfile.txt -n ipa-ca-agent  -c 'dogtag-ipa-renew-agent'

However, ipa-kra-install still fails after I copied the renewed certificate back to ca-agent.p12 with pk12util.

2015-08-19T15:29:07Z DEBUG args='/usr/bin/pki' '-d' '/tmp/tmp-rkoP5Q' '-c' 'XXXXXXX' '-n' 'ipa-ca-agent' 'kra-user-add' 'ipakra' '--fullName' 'IPA KRA User'
2015-08-19T15:29:09Z DEBUG Process finished, return code=255
2015-08-19T15:29:09Z DEBUG stdout=
2015-08-19T15:29:09Z DEBUG stderr=PKIException: Unauthorized

KRA instance log says it cannot map the certificate to any user:

19/Aug/2015:15:29:04][http-bio-8443-exec-3]: SourceConfigStore: storing jss.ssl.sslserver.ectype: ECDHE
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.logDebug: Authenticating certificate chain:
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.getAuditUserfromCert: certUID=CN=ipa-ca-agent, O=VDA.LI
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: PKIRealm.logDebug:   CN=ipa-ca-agent, O=VDA.LI
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: started
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: Retrieving client certificate
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuth: Got client certificate
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: Authentication: client certificate found
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: In LdapBoundConnFactory::getConn()
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: masterConn is connected: true
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: getConn: conn is connected true
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: getConn: mNumConns now 2
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: returnConn: mNumConns now 3
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: CertUserDBAuthentication: cannot map certificate to any user
[19/Aug/2015:15:29:09][http-bio-8443-exec-5]: SignedAuditEventFactory: create() message=[AuditEvent=AUTH_FAIL][SubjectID=CN=ipa-ca-agent, O=VDA.LI][Outcome=Failure][AuthMgr=certUserDBAuthMgr][AttemptedCred=CN=ipa-ca-agent, O=VDA.LI] authentication failure
Comment 7 Matthew Harmsen 2015-08-31 20:35:04 UTC 
edewata replied:

Per IRC discussion, the issue can be fixed with the following actions:

    PKI should restore the missing cert requests during upgrade as in comment #2, or remove them from CS.cfg as in ticket #1552.
    PKI should remove extra copies of the CA admin cert (e.g. PEM file, DER file, PKCS #12 file, NSS database) leaving only one master copy. It may require an upgrade script to remove the extra copies from existing installations.
    IPA should move the master copy of CA admin cert to a proper and safe location (e.g. /etc/pki/tls/private/ipa-ca-agent). It may require an upgrade script to move the master copy from /root in existing installations.
    IPA should track the master copy of CA admin cert. It may require certmonger to support tracking a cert stored in PKCS #12 file.
    IPA should use the master copy of CA admin cert to install KRA. It may force renewal before installation. It may require PKI to support importing the admin cert from PKCS #12 file.
    IPA should use -C <password file> instead of -c <password> when invoking the pki tool (​https://fedorahosted.org/freeipa/ticket/5246).
Comment 8 Matthew Harmsen 2015-08-31 20:35:39 UTC 
alee replied:

For reference, the stacktrace containing the certreq exception is located here: ​http://pastebin.test.redhat.com/304244

Now, what it shows is that we throw a null pointer exception in the method updateConfiguration() in SystemConfigService?.java when we try to store the certreq.

After going through the code, my suspicion is that we do not actually need this certreq. The certreq is used initially when generating the request objects when the cert was originally generated.

I believe it used to be needed to display the cert request when cloning. As we no longer support the install panels though, this is no longer a concern.

Basically, I think we can modify updateConfiguration() to store the certreq only if its available, and perhaps to allow the request entry to be removed when the install completes.

This would have to be tested though - especially with cloning.
Comment 10 Matthew Harmsen 2015-08-31 20:36:53 UTC 
edewata added:

IPA has been making some changes:

    using LDAPI to setup KRA agent: ​https://fedorahosted.org/freeipa/ticket/5257
    tracking KRA agent cert: ​https://fedorahosted.org/freeipa/ticket/5253
    removing clear-text password from logs: ​https://fedorahosted.org/freeipa/ticket/5246 

So the only remaining tasks are:

    PKI should remove cert requests from CS.cfg as in comment #12. This requires testing various installation scenarios.
    Optional: PKI should remove extra copies of the CA admin cert (e.g. PEM file, DER file, PKCS #12 file, NSS database) leaving only one master copy.
Comment 12 Matthew Harmsen 2015-09-03 23:18:44 UTC 
TEST PROCEDURE:  Added CLI to update cert data and request in CS.cfg [edewata]

( 1) # pki-server subsystem-cert-find ca

     -----------------
     5 entries matched
     -----------------
       Cert ID: signing
       Nickname: caSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: ocsp_signing
       Nickname: ocspSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: sslserver
       Nickname: Server-Cert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: subsystem
       Nickname: subsystemCert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: audit_signing
       Nickname: auditSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>

     NOTE:  May wish to save off a copy of this to compare with
            the results of step (11) below.

( 2) # systemctl stop pki-tomcatd

( 3) In order to compare this with the result of step (12) below,
     save a copy of '/etc/pki/pki-tomcat/ca/CS.cfg' before removing
     the following lines:

     * ca.audit_signing.cert=<value>
     * ca.audit_signing.certreq=<value>
     * ca.ocsp_signing.cert=<value>
     * ca.ocsp_signing.certreq=<value>
     * ca.signing.cert=<value>
     * ca.signing.certreq=<value>
     * ca.sslserver.cert=<value>
     * ca.sslserver.certreq=<value>
     * ca.subsystem.cert=<value>
     * ca.subsystem.certreq=<value>

( 4) # systemctl start pki-tomcatd

( 5) # pki-server subsystem-cert-find ca

     -----------------
     5 entries matched
     -----------------
       Cert ID: signing
       Nickname: caSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: None
       Request: None
     
       Cert ID: ocsp_signing
       Nickname: ocspSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: None
       Request: None
     
       Cert ID: sslserver
       Nickname: Server-Cert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: None
       Request: None
     
       Cert ID: subsystem
       Nickname: subsystemCert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: None
       Request: None
     
       Cert ID: audit_signing
       Nickname: auditSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: None
       Request: None

( 6) # pki-server subsystem-cert-update ca signing

( 7) # pki-server subsystem-cert-update ca ocsp_signing

( 8) # pki-server subsystem-cert-update ca sslserver

( 9) # pki-server subsystem-cert-update ca subsystem

(10) # pki-server subsystem-cert-update ca audit_signing

(11) # pki-server subsystem-cert-find ca

     -----------------
     5 entries matched
     -----------------
       Cert ID: signing
       Nickname: caSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: ocsp_signing
       Nickname: ocspSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: sslserver
       Nickname: Server-Cert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: subsystem
       Nickname: subsystemCert cert-pki-tomcat
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>
     
       Cert ID: audit_signing
       Nickname: auditSigningCert cert-pki-tomcat CA
       Token: Internal Key Storage Token
       Certificate: <value>
       Request: <value>

     NOTE:  Each <value> should be identical to its
            corresponding <value> in ( 1).

(12) View '/etc/pki/pki-tomcat/ca/CS.cfg':

     * ca.audit_signing.cert=<value>
     * ca.audit_signing.certreq=<value>
     * ca.ocsp_signing.cert=<value>
     * ca.ocsp_signing.certreq=<value>
     * ca.signing.cert=<value>
     * ca.signing.certreq=<value>
     * ca.sslserver.cert=<value>
     * ca.sslserver.certreq=<value>
     * ca.subsystem.cert=<value>
     * ca.subsystem.certreq=<value>

     NOTE: The 'CS.cfg' in steps ( 3) and (12) should be
           identical with regards to these entries.
Comment 13 Endi Sukma Dewata 2015-09-14 23:50:05 UTC 
Created attachment 1073458 [details]
pki-edewata-0641-RHEL-1-Added-CLI-to-update-cert-data-and-request-in-CS.cfg.patch
Comment 14 Endi Sukma Dewata 2015-09-14 23:50:56 UTC 
Created attachment 1073459 [details]
pki-edewata-0642-RHEL-1-Added-support-for-secure-database-connection-in-CLI.patch
Comment 15 Endi Sukma Dewata 2015-09-15 00:00:00 UTC 
Patch #641 provides the basic functionality for updating the cert data and request, but it only works with non-secure DS connection. Patch #642 adds the ability to use secure DS connection as in IPA.

Comment #12 provides the test procedure for patch #641. To test patch #642, enable the SSL connection and client certificate authentication first:
* http://pki.fedoraproject.org/wiki/Enabling_SSL_Connection_with_Internal_Database
* http://pki.fedoraproject.org/wiki/Enabling_Client_Certificate_Authentication_with_Internal_Database
then execute the same test procedure as in comment #12.
Comment 16 Scott Poore 2015-09-16 17:50:57 UTC 
FYI, I saw similar behavior as described in bug #1258964 after moving time forward to test certificate renewal on a new ipa 4.2 install.  I was unable to install KRA due to this after the renewal.
Comment 17 Matthew Harmsen 2015-09-21 21:58:52 UTC 
Checked into 'DOGTAG_10_2_5_RHEL_BRANCH':
* f153bd8a455953698e8af5085cd3cd7b368b1247
Comment 18 Endi Sukma Dewata 2015-09-22 00:11:51 UTC 
Also checked into master:
* bb6b49e0fba2b946c28d1beebfb6d22dfe6d568e
* 2ab352e1756d02695b73a9dbe782d353708ef553

The procedure to restore the missing parameters is documented in the following page: http://pki.fedoraproject.org/wiki/Subsystem_Certificate_Cache
Comment 20 Scott Poore 2015-09-23 16:21:20 UTC 
Was this fix just for upgrades?  Or should it have resolved my issue as well where I wasn't seeing certreq just from install?

With this fix, I tried to verify bug #1258964 again.  I'm still not seeing ca.sslserver.certreq though in CS.cfg with this fix.

My test is IPA server installed with:

ipa-server-4.2.0-11.el7.x86_64
pki-ca-10.2.5-6.el7.noarch

Then I move time forward within 2 weeks of expiration of cert stored in /root/ca-agent.p12:

[root@master ~]# openssl pkcs12 -in /root/ca-agent.p12 -out /root/ca-agent.pem
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
[root@master ~]# openssl x509 -in /root/ca-agent.pem -noout -enddate
notAfter=Sep 12 15:03:38 2017 GMT

[root@master ~]# date 082915032017
Tue Aug 29 15:03:00 CDT 2017

Wait for certs to go into MONITORING again and set time forward to a few days after that expiration:

[root@master ~]# date 091515032017
Fri Sep 15 15:03:00 CDT 2017

Then I try install KRA with ipa-kra-install and that fails.

[root@master ~]# grep ca.sslserver /var/lib/pki/pki-tomcat/ca/conf/CS.cfg
ca.sslserver.cert=MII...truncated for brevity...
ca.sslserver.nickname=Server-Cert cert-pki-ca
ca.sslserver.tokenname=Internal Key Storage Token
[root@master ~]#
Comment 21 Endi Sukma Dewata 2015-09-23 16:42:21 UTC 
The fix provides a tool to recover the missing ca.sslserver.certreq (see link in comment #18). Further investigation is needed to figure out why the ca.sslserver.certreq was missing in the first place.

So to fix the problem you can run the following command before running ipa-kra-install:
$ pki-server subsystem-cert-update ca sslserver
Comment 22 Roshni 2015-09-28 20:17:16 UTC 
[root@cisco-b22m3-01 ~]# rpm -qi ipa-server
Name        : ipa-server
Version     : 4.2.0
Release     : 12.el7
Architecture: x86_64
Install Date: Mon 28 Sep 2015 03:52:49 PM EDT
Group       : System Environment/Base
Size        : 5154390
License     : GPLv3+
Signature   : RSA/SHA256, Thu 24 Sep 2015 01:53:08 AM EDT, Key ID 938a80caf21541eb
Source RPM  : ipa-4.2.0-12.el7.src.rpm
Build Date  : Wed 23 Sep 2015 11:19:36 AM EDT
Build Host  : x86-035.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.freeipa.org/
Summary     : The IPA authentication server

[root@cisco-b22m3-01 ~]# rpm -qi pki-ca
Name        : pki-ca
Version     : 10.2.5
Release     : 6.el7
Architecture: noarch
Install Date: Mon 28 Sep 2015 03:52:48 PM EDT
Group       : System Environment/Daemons
Size        : 2429116
License     : GPLv2
Signature   : RSA/SHA256, Thu 24 Sep 2015 02:23:07 AM EDT, Key ID 938a80caf21541eb
Source RPM  : pki-core-10.2.5-6.el7.src.rpm
Build Date  : Tue 22 Sep 2015 06:02:20 PM EDT
Build Host  : ppc-036.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://pki.fedoraproject.org/
Summary     : Certificate System - Certificate Authority

[root@cisco-b22m3-01 ~]# rpm -qi pki-server
Name        : pki-server
Version     : 10.2.5
Release     : 6.el7
Architecture: noarch
Install Date: Mon 28 Sep 2015 03:52:46 PM EDT
Group       : System Environment/Base
Size        : 4691229
License     : GPLv2
Signature   : RSA/SHA256, Thu 24 Sep 2015 02:23:19 AM EDT, Key ID 938a80caf21541eb
Source RPM  : pki-core-10.2.5-6.el7.src.rpm
Build Date  : Tue 22 Sep 2015 06:02:20 PM EDT
Build Host  : ppc-036.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://pki.fedoraproject.org/
Summary     : Certificate System - PKI Server Framework

Verification steps:

1. ipa-server-install
2. Verify /etc/pki/pki-tomcat/ca/CS.cfg has the following params:

   * ca.audit_signing.cert=<value>
     * ca.audit_signing.certreq=<value>
     * ca.ocsp_signing.cert=<value>
     * ca.ocsp_signing.certreq=<value>
     * ca.signing.cert=<value>
     * ca.signing.certreq=<value>
     * ca.sslserver.cert=<value>
     * ca.sslserver.certreq=<value>
     * ca.subsystem.cert=<value>
     * ca.subsystem.certreq=<value>

3. yum install python-nss
4. successfully executed the steps explained in comment 12
5. ipa-kra-install was successful
Comment 23 errata-xmlrpc 2015-11-19 09:23:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2276.html