Bug 1343798
| Summary: | ipa-replica-install fails on adding CA certs to NSS db if multiple versions of the same CA cert is present in LDAP | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | lmgnid | ||||
| Component: | ipa | Assignee: | IPA Maintainers <ipa-maint> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Kaleem <ksiddiqu> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.2 | CC: | frenaud, lmgnid, pasik, pvoborni, rcritten | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-10-19 11:51:32 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Was your CA cert renewed recently? It seems that ipa-replica-install tries to add 2 certs with the same nickname into a slapd nss db. First add is successful, second fails. What does following command return: `ipa cert-find --subject "Certificate Authority"` It will probably return 2 certs? Is one expired? Or low level: ldapsearch -ZZ -h `hostname` -D "cn=Directory Manager" -W -b "cn=certificates,cn=ipa,cn=etc,dc=internal,dc=com" "(&(objectClass=ipaCertificate)(objectClass=pkiCA))" Does the low level output match the above? Hello Petr,
Here is the output of `ipa cert-find --subject "Certificate Authority"`
----------------------
2 certificates matched
----------------------
Serial number (hex): 0x1
Serial number: 1
Status: VALID
Subject: CN=Certificate Authority,O=INTERNAL.COM
Serial number (hex): 0x38FFC000F
Serial number: 15300558863
Status: VALID
Subject: CN=Certificate Authority,O=INTERNAL.COM
----------------------------
Number of entries returned 2
----------------------------
And here is the ldap search output:
# extended LDIF
#
# LDAPv3
# base <cn=certificates,cn=ipa,cn=etc,dc=internal,dc=com> with scope subtree
# filter: (&(objectClass=ipaCertificate)(objectClass=pkiCA))
# requesting: ALL
#
# INTERNAL.COM IPA CA, certificates, ipa, etc, internal.com
dn: cn=INTERNAL.COM IPA CA,cn=certificates,cn=ipa,cn=etc,dc=intern
al,dc=com
ipaConfigString: ipaCa
ipaCertIssuerSerial: CN=Certificate Authority,O=INTERNAL.COM;1
ipaCertIssuerSerial: CN=Certificate Authority,O=INTERNAL.COM;15300
558863
ipaKeyTrust: trusted
cACertificate;binary:: ...
ipaPublicKey:: ...
ipaCertSubject: CN=Certificate Authority,O=INTERNAL.COM
objectClass: ipaCertificate
objectClass: pkiCA
objectClass: ipaKeyPolicy
objectClass: top
cn: INTERNAL.COM IPA CA
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.1
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.2
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.3
ipaKeyExtUsage: 1.3.6.1.5.5.7.3.4
# search result
search: 3
result: 0 Success
# numResponses: 2
# numEntries: 1
BTW, In one of the replica has expired keys and cannot be auto renewed (Not all of the replicas):
[root@usdev-ops-ipa-01 ~]# getcert list | grep expire
expires: 2018-01-09 00:40:45 UTC
expires: 2016-05-03 21:14:05 UTC
expires: 2016-05-03 21:14:01 UTC
expires: 2016-05-03 21:14:03 UTC
expires: 2034-05-14 21:13:57 UTC
expires: 2017-12-29 00:52:21 UTC
expires: 2018-01-09 00:40:48 UTC
expires: 2016-05-03 21:15:03 UTC
As in:
https://bugzilla.redhat.com/show_bug.cgi?id=1343796
And I also got this issue in one of the master: https://bugzilla.redhat.com/show_bug.cgi?id=1344093 Hello Petr or anyone, could you help to check this as it quite urgent for us. Thanks! Why the expired certificates on the replica cannot be renewed? Usually you can just move date back to 2 days before expiration and force certmonger to resubmit the cert (getcert resubmit) Even though that in this case it might not be the case. A lot of things break when certs are expired so it is usually better to renewed them before trying something else. The output you provided in comment 3 doesn't seem to match the output of replica installation. Is there really only one value in cACertificate;binary attribute or only one entry? Workaround might be to temporary leave only one value ipaCertIssuerSerial and cACertificate;binary:: attributes. The serial should match the cert in the cn=INTERNAL.COM IPA CA,cn=certificates,cn=ipa,cn=etc,dc=intern al,dc=com LDAP entry. But then it should be rather changed back. Adding Jan to evaluate if the behavior is correct. To me it seems wrong the different versions of the the same certs are added to cert db with the same nickname. Hi Petr, As in https://bugzilla.redhat.com/show_bug.cgi?id=1343796, I already tried mover date back but still cannot renew the certificate, could you help to check that ticket? Thanks! Follow your workaround, I removed the CA with SN "15300558863" and revoke it with "ipa cert-revoke" command as bellow: Here is the current LDAP CA output (Removed cert bin): # extended LDIF # # LDAPv3 # base <cn=certificates,cn=ipa,cn=etc,dc=internal,dc=com> with scope subtree # filter: (&(objectClass=ipaCertificate)(objectClass=pkiCA)) # requesting: ALL # # INTERNAL.COM IPA CA, certificates, ipa, etc, internal.com dn: cn=INTERNAL.COM IPA CA,cn=certificates,cn=ipa,cn=etc,dc=intern al,dc=com ipaKeyTrust: trusted ipaKeyExtUsage: 1.3.6.1.5.5.7.3.1 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.2 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.3 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.4 ipaConfigString: ipaCa cACertificate;binary:: ... ipaPublicKey:: ... ipaCertSubject: CN=Certificate Authority,O=INTERNAL.COM ipaCertIssuerSerial: CN=Certificate Authority,O=INTERNAL.COM;1 cn: INTERNAL.COM IPA CA objectClass: top objectClass: ipaKeyPolicy objectClass: pkiCA objectClass: ipaCertificate # search result search: 3 result: 0 Success # numResponses: 2 # numEntries: 1 [root@usqa-ops-ipa-01 ~]# ipa cert-find --subject "Certificate Authority" ---------------------- 2 certificates matched ---------------------- Serial number (hex): 0x1 Serial number: 1 Status: VALID Subject: CN=Certificate Authority,O=INTERNAL.COM Serial number (hex): 0x38FFC000F Serial number: 15300558863 Status: REVOKED Subject: CN=Certificate Authority,O=INTERNAL.COM ---------------------------- Number of entries returned 2 ---------------------------- Now I can reinstall one broken IPA serer but still cannot reinstall another broken server, do you have any ideas? THe differences for these 2 servers are: - eupreprd-ops-ipa-01 reinstallation OK, this server had the broken DB issue before as in https://bugzilla.redhat.com/show_bug.cgi?id=1344093 - uspreprd-ops-ipa-01 reinstallation failed, this server had the expire before as in https://bugzilla.redhat.com/show_bug.cgi?id=1343796 - But even after I launched a fresh new redhat 7.2 instance and replace the old uspreprd-ops-ipa-01 with the same name, I still cannot install the replica, logs bellow [root@uspreprd-ops-ipa-01 ~]# ipa-replica-install --ip-address=10.1.0.234 --setup-dns --no-forwarders /home/ec2-user/uspreprd1.gpg ... Configuring Kerberos KDC (krb5kdc). Estimated time: 30 seconds [1/8]: adding sasl mappings to the directory [2/8]: configuring KDC [3/8]: creating a keytab for the directory [4/8]: creating a keytab for the machine [5/8]: adding the password extension to the directory [6/8]: enable GSSAPI for replication [error] RuntimeError: One of the ldap service principals is missing. Replication agreement cannot be converted. Replication error message: Can't acquire busy replica Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR One of the ldap service principals is missing. Replication agreement cannot be converted. Replication error message: Can't acquire busy replica [root@uspreprd-ops-ipa-01 ~]# tail -f /var/log/ipareplica-install.log 2016-06-10T21:54:01Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/krbinstance.py", line 438, in __convert_to_gssapi_replication r_bindpw=self.dm_password) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 1104, in convert_to_gssapi_replication self.gssapi_update_agreements(self.conn, r_conn) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 797, in gssapi_update_agreements self.setup_krb_princs_as_replica_binddns(a, b) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 767, in setup_krb_princs_as_replica_binddns (a_dn, b_dn) = self.get_replica_principal_dns(a, b, retries=100) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 751, in get_replica_principal_dns raise RuntimeError(error) RuntimeError: One of the ldap service principals is missing. Replication agreement cannot be converted. Replication error message: Can't acquire busy replica 2016-06-10T21:54:01Z DEBUG [error] RuntimeError: One of the ldap service principals is missing. Replication agreement cannot be converted. Replication error message: Can't acquire busy replica 2016-06-10T21:54:01Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 311, in run cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 281, in run self.execute() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 303, in execute for nothing in self._executor(): File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 343, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 333, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 87, in run_generator_with_yield_from raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 65, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 524, in _configure executor.next() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 343, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 365, in _handle_exception util.raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 333, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 87, in run_generator_with_yield_from raise_exc_info(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 65, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install for nothing in self._installer(self.parent): File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 879, in main install(self) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 295, in decorated func(installer) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 586, in install krb = install_krb(config, setup_pkinit=not options.no_pkinit) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py", line 93, in install_krb setup_pkinit, pkcs12_info) File "/usr/lib/python2.7/site-packages/ipaserver/install/krbinstance.py", line 214, in create_replica self.start_creation(runtime=30) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 418, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 408, in run_step method() File "/usr/lib/python2.7/site-packages/ipaserver/install/krbinstance.py", line 438, in __convert_to_gssapi_replication r_bindpw=self.dm_password) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 1104, in convert_to_gssapi_replication self.gssapi_update_agreements(self.conn, r_conn) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 797, in gssapi_update_agreements self.setup_krb_princs_as_replica_binddns(a, b) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 767, in setup_krb_princs_as_replica_binddns (a_dn, b_dn) = self.get_replica_principal_dns(a, b, retries=100) File "/usr/lib/python2.7/site-packages/ipaserver/install/replication.py", line 751, in get_replica_principal_dns raise RuntimeError(error) The issue in comment 7 is unrelated to the original issue so I'd not discuss in details it in this bug. It would be good to see ipareplica-install.log though. For investigation of cause of the original issue. Could you provide content of both certificates? If you are concerned about publicity of the data then it could be in a form of private attachment so that it won't be readable outside of Red Hat. Also, do you know how did you create the second cert? Was it by running ipa-cacert-manage command? With what options? For quicker answers I'd advise to open a support case with Red Hat. Additionally, to broaden audience, you can also ask on FreeIPA users list. Hi Petr, I already revoked the 2nd CA and cannot unrevok it now, here are work logs: BTW: I cannot remember exactlly how the 2nd CA was created, it might be created when I try to reinstall the replica with CA and DNS options in one server, which has the expired cert issues [root@usqa-ops-ipa-02 secaops]# ipa cert-find --subject "Certificate Authority" ---------------------- 2 certificates matched ---------------------- Serial number (hex): 0x1 Serial number: 1 Status: VALID Subject: CN=Certificate Authority,O=INTERNAL.COM Serial number (hex): 0x38FFC000F Serial number: 15300558863 Status: REVOKED Subject: CN=Certificate Authority,O=INTERNAL.COM ---------------------------- Number of entries returned 2 ---------------------------- [root@usqa-ops-ipa-02 secaops]# ipa cert-remove-hold 15300558863 Unrevoked: False Error: One or more certificates could not be unrevoked [root@usqa-ops-ipa-02 secaops]# ldapsearch -ZZ -h localhost -D "cn=Directory Manager" -W -b "cn=certificates,cn=ipa,cn=etc,dc=internal,dc=com" "(&(objectClass=ipaCertificate)(objectClass=pkiCA))" Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=certificates,cn=ipa,cn=etc,dc=internal,dc=com> with scope subtree # filter: (&(objectClass=ipaCertificate)(objectClass=pkiCA)) # requesting: ALL # # INTERNAL.COM IPA CA, certificates, ipa, etc, internal.com dn: cn=INTERNAL.COM IPA CA,cn=certificates,cn=ipa,cn=etc,dc=intern al,dc=com ipaKeyTrust: trusted ipaKeyExtUsage: 1.3.6.1.5.5.7.3.1 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.2 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.3 ipaKeyExtUsage: 1.3.6.1.5.5.7.3.4 ipaConfigString: ipaCa cACertificate;binary:: ... ipaCertSubject: CN=Certificate Authority,O=INTERNAL.COM ipaCertIssuerSerial: CN=Certificate Authority,O=INTERNAL.COM;1 cn: INTERNAL.COM IPA CA objectClass: top objectClass: ipaKeyPolicy objectClass: pkiCA objectClass: ipaCertificate # search result search: 3 result: 0 Success # numResponses: 2 # numEntries: 1 And I will PM you the certicate BIN. @Petr, any findings? Thanks! new ipareplica-install.log as written in comment 8 is required to proceed further. Wrt the 2 CA certs - I did not have a change to look at them, but they should not be related to the last error. Regardless the above, bug 1343796 should be resolved first because expired certs are often culprit for various failures. Let us change the BZ subject to the original issue which is really a bug in IPA. Other issues, which may be present in your environment should be discussed separately (either in new BZ or a thread on FreeIPA users list(preferred - broader audience)). Upstream ticket: https://fedorahosted.org/freeipa/ticket/6135 IdM team doesn't have capacity to fix this bug for RHEL 7.4. Moving to next RHEL version. Fixing the bug there will depend on capacity of FreeIPA upstream. Without sufficient justification there is a chance that it will be moved again later. I was not able to reproduce this issue with ipa-server 4.6.4-10.el7.
I tried in domain-level1 with the following scenario:
[master]# ipa-server-install [...]
[master]# getcert resubmit -i $<id_for_IPA_CA>
[master]# ipa-certupdate
[replica]# ipa-replica-install
The replica is successfully installed and the original + renewed IPA CA certs are available in slapd db:
[replica]# certutil -L -d /etc/dirsrv/slapd-DOMAIN-COM
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
DOMAIN.COM IPA CA CT,C,C
DOMAIN.COM IPA CA CT,C,C
Server-Cert u,u,u
Also tried in domain-level 0:
[master]# ipa-server-install --domain-level 0 [...]
[master]# getcert resubmit -i $<id_for_IPA_CA>
[master]# ipa-certupdate
[master]# ipa-replica-prepare replica.domain.com
[replica]# ipa-replica-install <replica file>
The replica is successfully installed and the original + renewed IPA CA certs are available in slapd db:
[replica]# certutil -L -d /etc/dirsrv/slapd-DOMAIN-COM
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
DOMAIN.COM IPA CA CT,C,C
DOMAIN.COM IPA CA CT,C,C
Server-Cert u,u,u
Hence closing as CURRENTRELEASE
|
Created attachment 1165795 [details] ipareplicat-install.log for the failed ipa replicate installation Description of problem: Cannot install the IPA replica Version-Release number of selected component (if applicable): [root@eupreprd-ops-ipa-01 etc]# rpm -qa | grep ipa python-libipa_hbac-1.13.0-40.el7_2.1.x86_64 ipa-server-4.2.0-15.el7_2.3.x86_64 python-iniparse-0.4-9.el7.noarch libipa_hbac-1.13.0-40.el7_2.1.x86_64 ipa-python-4.2.0-15.el7_2.3.x86_64 ipa-client-4.2.0-15.el7_2.3.x86_64 redhat-access-plugin-ipa-0.9.1-2.el7.noarch sssd-ipa-1.13.0-40.el7_2.1.x86_64 ipa-admintools-4.2.0-15.el7_2.3.x86_64 ipa-server-dns-4.2.0-15.el7_2.3.x86_64 How reproducible: Everytime Steps to Reproduce: [root@usqa-ops-ipa-01 ec2-user]# ipa-replica-prepare --ip-address=10.0.10.249 eupreprd-ops-ipa-01.internal.com [root@eupreprd-ops-ipa-01 ec2-user]# ipa-replica-install --ip-address=10.0.10.249 --setup-dns --no-forwarders /home/ec2-user/eupreprd.gpg Actual results: [28/38]: importing CA certificates from LDAP [error] CalledProcessError: Command ''/usr/bin/certutil' '-d' '/etc/dirsrv/slapd-INTERNAL-COM/' '-A' '-n' 'INTERNAL.COM IPA CA' '-t' 'CT,C,C'' returned non-zero exit status 255 Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipa.ipapython.install.cli.install_tool(Replica): ERROR Command ''/usr/bin/certutil' '-d' '/etc/dirsrv/slapd-INTERNAL-COM/' '-A' '-n' 'INTERNAL.COM IPA CA' '-t' 'CT,C,C'' returned non-zero exit status 255 Expected results: Install OK Additional info: ipareplicat-install.log for the failed ipa replicate installation is also attached. And in the master IPA servers, no expired certificates: [root@usqa-ops-ipa-01 ec2-user]# getcert list | grep expire expires: 2017-12-21 22:46:08 UTC expires: 2018-03-26 22:59:53 UTC expires: 2018-03-26 22:58:50 UTC expires: 2018-03-26 22:58:44 UTC expires: 2034-05-14 21:13:57 UTC expires: 2017-12-10 23:21:54 UTC expires: 2017-12-21 22:46:10 UTC expires: 2018-03-26 23:01:58 UTC