Bug 1040018

Summary:	Automatic CA subsystem certificate renewal is broken on CA clones
Product:	Red Hat Enterprise Linux 7	Reporter:	Dmitri Pal <dpal>
Component:	ipa	Assignee:	Martin Kosek <mkosek>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Namita Soman <nsoman>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	7.0	CC:	mkosek, rcritten, spoore
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ipa-3.3.3-10.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1040009	Environment:
Last Closed:	2014-06-13 10:48:12 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1040009, 1049532
Bug Blocks:

Description Dmitri Pal 2013-12-10 14:08:56 UTC

+++ This bug was initially created as a clone of Bug #1040009 +++

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/4064

On CA clones, certmonger uses the `dogtag-ipa-retrieve-agent-submit` CA helper script to retrieve renewed CA subsystem certificates from master CA. Certmonger expects the script to write the certificate in PEM format to its standard output. The script does that, but prepends an extra "\033[?1034h" to the output, causing certmonger to fail to parse the certificate.

The erroneous output is caused by a bug in readline: http://lists.gnu.org/archive/html/bug-readline/2013-06/msg00000.html, https://bugzilla.redhat.com/show_bug.cgi?id=880393. The Python readline module is not imported in `dogtag-ipa-retrieve-agent-submit` itself, but in some module it imports.

The workaround is to set the `TERM` environment variable to some terminal type which does not support the meta-key capability (such as vt100) before importing modules in `dogtag-ipa-retrieve-agent-submit`.

Comment 1 Martin Kosek 2014-01-02 14:50:22 UTC

Fixed upstream:
master: https://fedorahosted.org/freeipa/changeset/1357eade4c5086e6c837a49f3008616317f88e5f
ipa-3-3: https://fedorahosted.org/freeipa/changeset/854dbb8ff9e898b289c9464567f141421880a050

Comment 3 Martin Kosek 2014-01-07 16:19:32 UTC

It was found out that PKI service did not start properly after the renewal, this is being solved upstream:

https://fedorahosted.org/freeipa/ticket/4092

Comment 4 Martin Kosek 2014-01-08 09:02:23 UTC

Ticket 4092 was fixed upstream:

master: 911f5e9eb76099f8e5cfcff1232c1b10ad05b45a
ipa-3-3: edccf59d8018349bc3596e017a660dcb83034932

Comment 5 Martin Kosek 2014-01-08 09:04:01 UTC

Linking to related SELinux issue preventing renewal with enforced SELinux - Bug 1049532.

Comment 6 Scott Poore 2014-02-10 22:38:43 UTC

Martin,

Does this have to be verified on an IPA Replica with CA installed or can it be verified on an IPA "Master" (single server environment)?

Is it enough to do a getcert resubmit  and confirm the expiration or do we need to go all the way to change the date and watch the auto-renewal system resubmit?

Thanks,
Scott

Comment 7 Rob Crittenden 2014-02-11 13:59:44 UTC

Scott, needs another master to fully test this, along with switching dates, etc.

Only one master does the actual renewal. It stuffs the result into LDAP which the other masters get the updated certificates from. It is this process that was going sideways. ASCII garbage was being included when we fetched the certificate from LDAP and provided it to certmonger.

Comment 8 Scott Poore 2014-02-11 14:28:04 UTC

When you say another master, you mean another IPA server created with --setup-ca for the same domain?

Comment 9 Rob Crittenden 2014-02-11 14:34:01 UTC

Correct. The renewal is done differently on the master initiating the renewal and the other masters and this bug presented on the other masters.

Comment 10 Scott Poore 2014-02-11 17:37:11 UTC

Verified.

Version ::

ipa-server-3.3.3-17.el7.x86_64

Results ::

resubmit test:

[root@master ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211000124':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='906281234271'
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-renew-agent
        issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
        subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
        expires: 2016-02-01 00:00:45 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes


[root@replica1 ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211143326':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='480267000059'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 00:00:45 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

[root@master ~]# getcert resubmit -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Resubmitting "20140211000124" to "dogtag-ipa-renew-agent".

...waited for status to go back to MONITORING...

[root@master ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211000124':
        status: MONITORING
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='906281234271'
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-renew-agent
        issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
        subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
        expires: 2016-02-01 15:32:03 UTC
        key usage: digitalSignature,nonRepudiation
        pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
        post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
        track: yes
        auto-renew: yes


...Then on replica, run resubmit and check...

[root@replica1 ~]# getcert resubmit -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Resubmitting "20140211143326" to "dogtag-ipa-retrieve-agent-submit".

...wait for status to change to MONITORING...

[root@replica1 ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211143326':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='480267000059'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 15:32:03 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

...finally, confirm certs match:

[root@replica1 ~]# ssh $MASTER "certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n 'auditSigningCert cert-pki-ca'" > cert.master
root.example.test's password: 

[root@replica1 ~]# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n 'auditSigningCert cert-pki-ca' > cert.replica

[root@replica1 ~]# diff cert.master cert.replica
[root@replica1 ~]# 

Now, automatic renewal with time change...

This was run after a VM rebuild.

ON MASTER:

# ipactl stop
Stopping Directory Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping httpd Service
Stopping ipa_memcached Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
ipa: INFO: The ipactl command was successful

# hostname
master.ipa1.example.test

[root@master ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211161725':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='419558786185'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-renew-agent
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 16:16:45 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

[root@master ~]# date 010415002016
Mon Jan  4 15:00:00 CST 2016

[root@master ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
ipa: INFO: The ipactl command was successful

ON REPLICA:

# ipactl stop
Stopping Directory Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping httpd Service
Stopping ipa_memcached Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service

[root@replica1 ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211164557':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='288546002948'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 16:16:45 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

[root@replica1 ~]# date 010415002016
Mon Jan  4 15:00:00 CST 2016

[root@replica1 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Starting ipa-otpd Service
ipa: INFO: The ipactl command was successful

ON MASTER:

[root@master ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211161725':
	status: NOTIFYING_VALIDITY
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='419558786185'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-renew-agent
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 16:16:45 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

[root@master ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211161725':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='419558786185'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-renew-agent
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2017-12-24 21:14:20 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

ON REPLICA:

[root@replica1 ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211164557':
	status: NOTIFYING_VALIDITY
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='288546002948'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2016-02-01 16:16:45 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

[root@replica1 ~]# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
Number of certificates and requests being tracked: 7.
Request ID '20140211164557':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='288546002948'
	certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=IPA1.EXAMPLE.TEST
	subject: CN=CA Audit,O=IPA1.EXAMPLE.TEST
	expires: 2017-12-24 21:14:20 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes

And to confirm that certs are the same:

[root@replica1 ~]# ssh $MASTER "certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n 'auditSigningCert cert-pki-ca'" > cert.master
root.example.test's password: 

[root@replica1 ~]# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n 'auditSigningCert cert-pki-ca' > cert.replica

[root@replica1 ~]# diff cert.master cert.replica

Comment 12 Ludek Smid 2014-06-13 10:48:12 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.