Bug 1755223

Summary: Sub-CA key replication failure
Product: Red Hat Enterprise Linux 7 Reporter: Fraser Tweedale <ftweedal>
Component: ipaAssignee: Fraser Tweedale <ftweedal>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: urgent Docs Contact: Florian Delehaye <fdelehay>
Priority: high    
Version: 7.7CC: fdelehay, frenaud, pasik, rcritten, ssidhaye, sumenon, tscherf
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.6.6-8.el7 Doc Type: Bug Fix
Doc Text:
.Sub-CA key replication no longer fails Previously, a change to the credential cache (ccache) behaviour in the Kerberos library caused lightweight Certificate Authority (CA) key replication to fail. This update adapts the IdM lightweight CA key replication client code to the changed ccache behaviour. As a result, the lightweight CA key replication now works correctly.
Story Points: ---
Clone Of:
: 1756913 1756914 (view as bug list) Environment:
Last Closed: 2020-03-31 19:55:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1756913, 1756914, 1829993    

Description Fraser Tweedale 2019-09-25 05:56:51 UTC
Issue

LWCA key replication is failing on RHEL 7.7. Dogtag cannot retrieve the keys. In debug log:

2019-05-30 14:53:03 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: Running ExternalProcessKeyRetriever
2019-05-30 14:53:03 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca 06e99390-24f8-4632-a26c-a01693fb87f
d, server1.ipa.local]
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] SEVERE: Failed to retrieve key from any host.
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] WARNING: KeyRetriever did not return a result.
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: Retrying in 1946 seconds

In journal we see:

GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: KCM:))

In ipa-pki-retrieve-key, credential acquisition (from /etc/pki/pki-tomcat/dogtag.keytab) succeeds but those credentials are not used when binding to LDAP.
Steps to Reproduce

    Create server and CA replica.
    Create LWCA on one CA server.

Actual behavior

LWCA key replicaion failure on the other CA server, as described above.

Furthermore, the ca-show command fails; NPE due to missing keys:

ftweedal% ipa ca-show test1
ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500. 

Debug log:

2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: AuthorityResource.getCert()                                                                                                    
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: content-type: null                                                                                                             
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: accept: [application/pkix-cert]                                                                                                
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: response format: application/pkix-cert                                                                                         
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception                                                                    
org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException                            
        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:77)                                                                                                           
        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:220)          ....
Caused by: java.lang.NullPointerException
        at org.dogtagpki.server.ca.rest.AuthorityService.getCert(AuthorityService.java:147)
    ....

Expected behavior

    no NPE or IPA command failure when executing ca-show and keys are missing.
    Key replication succeeds.


Additional information:

This failure was triggered by a krb5 package update. It was introduced by one of the following patches:

* Mon Dec 17 2018 Robbie Harwood <rharwood> - 1.15.1-36
- Clean up MEMORY ccache behavior to match upstream more closely
- Resolves: #1657890
* Tue Dec 11 2018 Robbie Harwood <rharwood> - 1.15.1-35
- Fix bugs with concurrent use of MEMORY ccaches
- Resolves: #1657890


This update went to 7.7.z and 7.6.z, so we will need to fix IPA on those releases too.


IPA upstream ticket: https://pagure.io/freeipa/issue/7964

Comment 3 Fraser Tweedale 2019-09-25 06:55:46 UTC
Pull request: https://github.com/freeipa/freeipa/pull/3731

Comment 4 Fraser Tweedale 2019-09-27 01:58:42 UTC
ipa-4-6:

* 82a9fe7e655115befbdde10907a5aa7669c35fde Handle missing LWCA certificate or chain
* 436214aea7fd5893525292cb03b3c28cdbc249f2 Fix CustodiaClient ccache handling
* 1f455867f82407c0dfab0b9f123c75ca0d1a0090 CustodiaClient: use ldapi when ldap_uri not specified
* c9d0ba0c355c433ae883cafa3c1e99fea1a85220 CustodiaClient: fix IPASecStore config on ipa-4-7
* e686949dcdc46486061d23d5e18f21e2a2038f58 (HEAD) Bump krb5 min version

Moving to POST

Comment 5 Fraser Tweedale 2019-09-27 05:27:17 UTC
Verification steps:

1. install master

2. install replica with CA

3. create a sub-CA on one server

4. confirm that after a short time (10 seconds ought to do it), the sub-CA signing keys
   appear in the Dogtag NSSDB on the OTHER server
   (certutil -d /etc/pki/pki-tomcat/alias -L)

e.g.

[root@rhel76-0 ~]# certutil -d /etc/pki/pki-tomcat/alias -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
transportCert cert-pki-kra                                   u,u,u
auditSigningCert cert-pki-kra                                u,u,Pu
caSigningCert cert-pki-ca 8e134d37-ab44-4429-88a1-44ee299b4c38 u,u,u      <<<--- look for an entry like this (UUID will differ)
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
storageCert cert-pki-kra                                     u,u,u

Comment 11 Fraser Tweedale 2019-10-02 02:47:32 UTC
Add doc text.

Comment 12 Sudhir Menon 2019-10-04 11:41:49 UTC
Fix is seen. Verified on RHEL7.8

[root@master alias]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 Beta (Maipo)

[root@master alias]# rpm -q ipa-server 389-ds-base krb5-server 
ipa-server-4.6.6-8.el7.x86_64
389-ds-base-1.3.10.1-2.el7.x86_64
krb5-server-1.15.1-46.el7.x86_64

[root@master alias]# certutil -d . -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca 4dbb6f3a-5982-4f57-a178-7aa045831010 u,u,u

Finalize replication settings
Restarting the KDC
[root@replica ~]# cd /var/lib/pki/pki-tomcat/alias/
[root@replica alias]# certutil -d . -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca 4dbb6f3a-5982-4f57-a178-7aa045831010 u,u,u

Comment 18 errata-xmlrpc 2020-03-31 19:55:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1083