RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1755223 - Sub-CA key replication failure
Summary: Sub-CA key replication failure
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.7
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: rc
: ---
Assignee: Fraser Tweedale
QA Contact: ipa-qe
Florian Delehaye
URL:
Whiteboard:
Depends On:
Blocks: 1756913 1756914 1829993
TreeView+ depends on / blocked
 
Reported: 2019-09-25 05:56 UTC by Fraser Tweedale
Modified: 2020-04-30 17:12 UTC (History)
7 users (show)

Fixed In Version: ipa-4.6.6-8.el7
Doc Type: Bug Fix
Doc Text:
.Sub-CA key replication no longer fails Previously, a change to the credential cache (ccache) behaviour in the Kerberos library caused lightweight Certificate Authority (CA) key replication to fail. This update adapts the IdM lightweight CA key replication client code to the changed ccache behaviour. As a result, the lightweight CA key replication now works correctly.
Clone Of:
: 1756913 1756914 (view as bug list)
Environment:
Last Closed: 2020-03-31 19:55:52 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Fedora Pagure freeipa issue 7964 0 None None None 2019-09-25 05:56:50 UTC
Red Hat Product Errata RHBA-2020:1083 0 None None None 2020-03-31 19:56:20 UTC

Description Fraser Tweedale 2019-09-25 05:56:51 UTC 
Issue

LWCA key replication is failing on RHEL 7.7. Dogtag cannot retrieve the keys. In debug log:

2019-05-30 14:53:03 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: Running ExternalProcessKeyRetriever
2019-05-30 14:53:03 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: About to execute command: [/usr/libexec/ipa/ipa-pki-retrieve-key, caSigningCert cert-pki-ca 06e99390-24f8-4632-a26c-a01693fb87f
d, server1.ipa.local]
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] SEVERE: Failed to retrieve key from any host.
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] WARNING: KeyRetriever did not return a result.
2019-05-30 14:53:04 [KeyRetrieverRunner-06e99390-24f8-4632-a26c-a01693fb87fd] FINE: Retrying in 1946 seconds

In journal we see:

GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (No Kerberos credentials available (default cache: KCM:))

In ipa-pki-retrieve-key, credential acquisition (from /etc/pki/pki-tomcat/dogtag.keytab) succeeds but those credentials are not used when binding to LDAP.
Steps to Reproduce

    Create server and CA replica.
    Create LWCA on one CA server.

Actual behavior

LWCA key replicaion failure on the other CA server, as described above.

Furthermore, the ca-show command fails; NPE due to missing keys:

ftweedal% ipa ca-show test1
ipa: ERROR: Request failed with status 500: Non-2xx response from CA REST API: 500. 

Debug log:

2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: AuthorityResource.getCert()                                                                                                    
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: content-type: null                                                                                                             
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: accept: [application/pkix-cert]                                                                                                
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] FINE: MessageFormatInterceptor: response format: application/pkix-cert                                                                                         
2019-05-30 14:56:41 [ajp-nio-127.0.0.1-8009-exec-5] SEVERE: Servlet.service() for servlet [Resteasy] in context with path [/ca] threw exception                                                                    
org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException                            
        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:77)                                                                                                           
        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:220)          ....
Caused by: java.lang.NullPointerException
        at org.dogtagpki.server.ca.rest.AuthorityService.getCert(AuthorityService.java:147)
    ....

Expected behavior

    no NPE or IPA command failure when executing ca-show and keys are missing.
    Key replication succeeds.


Additional information:

This failure was triggered by a krb5 package update. It was introduced by one of the following patches:

* Mon Dec 17 2018 Robbie Harwood <rharwood> - 1.15.1-36
- Clean up MEMORY ccache behavior to match upstream more closely
- Resolves: #1657890
* Tue Dec 11 2018 Robbie Harwood <rharwood> - 1.15.1-35
- Fix bugs with concurrent use of MEMORY ccaches
- Resolves: #1657890


This update went to 7.7.z and 7.6.z, so we will need to fix IPA on those releases too.


IPA upstream ticket: https://pagure.io/freeipa/issue/7964
Comment 3 Fraser Tweedale 2019-09-25 06:55:46 UTC 
Pull request: https://github.com/freeipa/freeipa/pull/3731
Comment 4 Fraser Tweedale 2019-09-27 01:58:42 UTC 
ipa-4-6:

* 82a9fe7e655115befbdde10907a5aa7669c35fde Handle missing LWCA certificate or chain
* 436214aea7fd5893525292cb03b3c28cdbc249f2 Fix CustodiaClient ccache handling
* 1f455867f82407c0dfab0b9f123c75ca0d1a0090 CustodiaClient: use ldapi when ldap_uri not specified
* c9d0ba0c355c433ae883cafa3c1e99fea1a85220 CustodiaClient: fix IPASecStore config on ipa-4-7
* e686949dcdc46486061d23d5e18f21e2a2038f58 (HEAD) Bump krb5 min version

Moving to POST
Comment 5 Fraser Tweedale 2019-09-27 05:27:17 UTC 
Verification steps:

1. install master

2. install replica with CA

3. create a sub-CA on one server

4. confirm that after a short time (10 seconds ought to do it), the sub-CA signing keys
   appear in the Dogtag NSSDB on the OTHER server
   (certutil -d /etc/pki/pki-tomcat/alias -L)

e.g.

[root@rhel76-0 ~]# certutil -d /etc/pki/pki-tomcat/alias -L

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
transportCert cert-pki-kra                                   u,u,u
auditSigningCert cert-pki-kra                                u,u,Pu
caSigningCert cert-pki-ca 8e134d37-ab44-4429-88a1-44ee299b4c38 u,u,u      <<<--- look for an entry like this (UUID will differ)
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
storageCert cert-pki-kra                                     u,u,u
Comment 11 Fraser Tweedale 2019-10-02 02:47:32 UTC 
Add doc text.
Comment 12 Sudhir Menon 2019-10-04 11:41:49 UTC 
Fix is seen. Verified on RHEL7.8

[root@master alias]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 Beta (Maipo)

[root@master alias]# rpm -q ipa-server 389-ds-base krb5-server 
ipa-server-4.6.6-8.el7.x86_64
389-ds-base-1.3.10.1-2.el7.x86_64
krb5-server-1.15.1-46.el7.x86_64

[root@master alias]# certutil -d . -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca 4dbb6f3a-5982-4f57-a178-7aa045831010 u,u,u

Finalize replication settings
Restarting the KDC
[root@replica ~]# cd /var/lib/pki/pki-tomcat/alias/
[root@replica alias]# certutil -d . -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
caSigningCert cert-pki-ca                                    CTu,Cu,Cu
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u
caSigningCert cert-pki-ca 4dbb6f3a-5982-4f57-a178-7aa045831010 u,u,u
Comment 18 errata-xmlrpc 2020-03-31 19:55:52 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1083
Note You need to log in before you can comment on or make changes to this bug.