Bug 1138777

Summary:	Renewal with no master CA
Product:	Red Hat Enterprise Linux 7	Reporter:	Martin Kosek <mkosek>
Component:	ipa	Assignee:	Martin Kosek <mkosek>
Status:	CLOSED ERRATA	QA Contact:	Namita Soman <nsoman>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	7.0	CC:	jcholast, mkosek, rcritten, spoore
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ipa-4.0.3-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-05 10:13:35 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Martin Kosek 2014-09-05 15:32:54 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/4039

A user has 3 CAs. None of them is configured to do CA renewal. They are all configured to pull updated certificates from LDAP using the certmonger renew_ra_cert CA. It is unclear how it got into this situation. It could be the original master was deleted, or this was the result of some upgrade, but none of the masters is configured to do the actual subsystem renewal. The result was the renewal status was CA_WORKING on all systems.

The user is using IPA 3.0 on RHEL 6.4, so this is a dual 389-ds install.

Fixing this was a relatively straightforward process to pick one to do the renewal, tell certmonger to stop tracking the CA subsystem certificates and then configure certmonger to renew the certificates (it is a difference in the CA script that certmonger users).

So this is problem #1: provide documentation on how to recover from this situation.

This worked ok and all 4 of the required certificates were renewed, but something strange happened.

Only one certificate was added to LDAP, the RA agent cert ipaCert. We never did figure out why.

This is problem #2: see how it can happen that a certificate is renewed and not added to LDAP.

Replication was firing as we saw the ipaCert entry in cn=ca_renewal on all masters and the ou=People entry in the CA DS instance was replicated ok as well. So in short, replication was working in both instances.

To fix this we manually pulled the updated certificates out of the various databases using certutil -L -d /path/to/db -n <nickname> -a. Then we moved the certificates to the non-updated masters and manually added them to their respective database. Note that certutil will pull ALL the certificates out of an NSS database if there is more than one certificate, such as the case for a renewal. This is fine, but it also will only ADD one certificate via certutil -A which means you'll need to edit each file such that only the latest certificate is stored there.

We had to do this for the 3 CA certificates (ocsp, audit, subsystem) in /var/lib/<location>/alias (location is distro-specific) and the ipaCert RA agent certificate in /etc/httpd/alias.

Comment 1 Martin Kosek 2014-09-05 16:14:35 UTC

Fixed upstream:

master:
774140196360c727f11c75622ace488d591ddfba Allow changing CA renewal master in ipa-csreplica-manage.

ipa-4-1:
aae78480220203b1c64c8b3c6b8297868c849110 Allow changing CA renewal master in ipa-csreplica-manage.

ipa-4-0:
8999300894326d104ddf22a97d74d78fdab0984c Allow changing CA renewal master in ipa-csreplica-manage.

Comment 3 Scott Poore 2015-01-13 02:41:46 UTC

Martin,

Any idea today on how we can reproduce this?  Or just test that CA renewal master can be changed with ipa-csreplica-manage?

Thanks,
Scott

Comment 4 Martin Kosek 2015-01-13 11:20:10 UTC

Jan owned the upstream ticket, he should be able to help.

Comment 5 Scott Poore 2015-01-14 19:11:44 UTC

Also, how can I see which host is the master?  There's a set-renewal-master command but, how do I see which one is master?

Thanks,
Scott

Comment 6 Scott Poore 2015-01-15 01:19:42 UTC

nevermind on the last one.  I found it:

ldapsearch -xLLL -D "cn=Directory Manager" -w Secret123 \
    -b cn=CA,cn=$MASTER,cn=masters,cn=ipa,cn=etc,$BASEDN | \
    grep caRenewalMaster|wc -l

if it's the renewal master that will = 1

So, I still need to know if there's any way to reproduce this or just test setup-renewal-master option of ipa-csreplica-manage?

Thanks

Comment 7 Jan Cholasta 2015-01-15 08:18:18 UTC

Now that renewal race conditions are removed, I think it should be enough to test "ipa-csreplica-manage set-renewal-master". Just FYI, "ipa-cacert-manage renew" changes renewal master to the host it is run on (it is necessary to make it work properly).

Comment 8 Scott Poore 2015-01-15 15:55:32 UTC

Verified.

Version ::

ipa-server-4.1.0-13.el7.x86_64

Results ::


[root@rhel7-1 ~]# ldapsearch -xLLL -D "cn=Directory Manager" -w Secret123     -b cn=masters,cn=ipa,cn=etc,$BASEDN |egrep "cn=CA|caRenewalMaster"
dn: cn=CA,cn=rhel7-1.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
ipaConfigString: caRenewalMaster
dn: cn=CA,cn=rhel7-3.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-4.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-5.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com

[root@rhel7-1 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master 
rhel7-1.example.com is already the renewal master

[root@rhel7-1 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master rhel7-2.example.com
Failed to set renewal master to rhel7-2.example.com: no such entry

[root@rhel7-1 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master rhel7-3.example.com
rhel7-3.example.com is now the renewal master

[root@rhel7-1 ~]# ldapsearch -xLLL -D "cn=Directory Manager" -w Secret123     -b cn=masters,cn=ipa,cn=etc,$BASEDN |egrep "cn=CA|caRenewalMaster"
dn: cn=CA,cn=rhel7-1.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-3.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
ipaConfigString: caRenewalMaster
dn: cn=CA,cn=rhel7-4.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-5.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com

[root@rhel7-1 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master rhel7-5.example.comrhel7-5.example.com is now the renewal master

[root@rhel7-1 ~]# ldapsearch -xLLL -D "cn=Directory Manager" -w Secret123     -b cn=masters,cn=ipa,cn=etc,$BASEDN |egrep "cn=CA|caRenewalMaster"
dn: cn=CA,cn=rhel7-1.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-3.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-4.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-5.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
ipaConfigString: caRenewalMaster

[root@rhel7-1 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master rhel7-1.example.comrhel7-1.example.com is now the renewal master

[root@rhel7-1 ~]# ldapsearch -xLLL -D "cn=Directory Manager" -w Secret123     -b cn=masters,cn=ipa,cn=etc,$BASEDN |egrep "cn=CA|caRenewalMaster"
dn: cn=CA,cn=rhel7-1.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
ipaConfigString: caRenewalMaster
dn: cn=CA,cn=rhel7-3.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-4.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com
dn: cn=CA,cn=rhel7-5.example.com,cn=masters,cn=ipa,cn=etc,dc=example,dc=com

[root@rhel7-3 ~]# ipa-csreplica-manage -p Secret123 set-renewal-master
rhel7-3.example.com is now the renewal master

Comment 10 errata-xmlrpc 2015-03-05 10:13:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0442.html