Bug 1040009

Summary:	Automatic CA subsystem certificate renewal is broken on CA clones
Product:	Red Hat Enterprise Linux 6	Reporter:	Dmitri Pal <dpal>
Component:	ipa	Assignee:	Martin Kosek <mkosek>
Status:	CLOSED ERRATA	QA Contact:	Namita Soman <nsoman>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	6.4	CC:	jcholast, jnansi, ksiddiqu, mkosek, rcritten
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ipa-3.0.0-38.el6	Doc Type:	Bug Fix
Doc Text:	Cause: A bug in the python readline module causes a stray escape sequence to be prepended to the output of the script which certmonger uses to acquire renewed certificates on CA clones. Consequence: Certmonger fails to parse the output of the script and certificate renewal fails. Fix: Work around the bug in the python readline module. Result: Certmonger can successfully parse the output of the script and complete the renewal.	Story Points:	---
Clone Of:
Clones:	1040018 (view as bug list)		Environment:
Last Closed:	2014-10-14 07:32:29 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1040018, 1061410

Description Dmitri Pal 2013-12-10 13:48:20 UTC

This bug is created as a clone of upstream ticket:
https://fedorahosted.org/freeipa/ticket/4064

On CA clones, certmonger uses the `dogtag-ipa-retrieve-agent-submit` CA helper script to retrieve renewed CA subsystem certificates from master CA. Certmonger expects the script to write the certificate in PEM format to its standard output. The script does that, but prepends an extra "\033[?1034h" to the output, causing certmonger to fail to parse the certificate.

The erroneous output is caused by a bug in readline: http://lists.gnu.org/archive/html/bug-readline/2013-06/msg00000.html, https://bugzilla.redhat.com/show_bug.cgi?id=880393. The Python readline module is not imported in `dogtag-ipa-retrieve-agent-submit` itself, but in some module it imports.

The workaround is to set the `TERM` environment variable to some terminal type which does not support the meta-key capability (such as vt100) before importing modules in `dogtag-ipa-retrieve-agent-submit`.

Comment 1 Martin Kosek 2014-01-02 14:48:58 UTC

Fixed upstream:
master: https://fedorahosted.org/freeipa/changeset/1357eade4c5086e6c837a49f3008616317f88e5f
ipa-3-3: https://fedorahosted.org/freeipa/changeset/854dbb8ff9e898b289c9464567f141421880a050

Comment 2 Namita Soman 2014-01-06 16:39:46 UTC

Please provide steps to verify.

Comment 3 Martin Kosek 2014-01-07 11:52:05 UTC

I think there are 2 approaches:

1) End to end testing (more difficult approach though more complete):
- Install IPA server
- Install IPA replica with CA
- move system time with both close to the end of cert validity, check if certs renew (based on Yi's work)

2) Certmonger resubmit testing (easier, I was testing it today):
- Install IPA server
- Install IPA replica with CA
- On IPA server, resubmit some of the CA subsystem certificates, for example audit cert:
# getcert resubmit -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
- On IPA server, wait until the certificate is generated and status is back to MONITORING with:
# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
- On IPA replica we will try to force renewal of this certificate. Run:
# getcert resubmit -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
- On IPA replica, wait until the certificate is renewed and status is back to MONITORING with:
# getcert list -d '/etc/pki/pki-tomcat/alias' -n 'auditSigningCert cert-pki-ca'
- The new certificate should be now updated, see
# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n 'auditSigningCert cert-pki-ca'
- PKI service should also be running

With old IPA version, the resubmission of certificate on IPA replica will not be successful, certmonger may either crash or report unsuccessful retrieval of the certificate

Jan, please comment if I missed anything

Comment 5 Kaleem 2014-07-25 12:27:40 UTC

Verified using second method mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1040009#c3

IPA version:
===========
[root@hp-bl280cg6-01 ipa-ca-install]# rpm -q ipa-server
ipa-server-3.0.0-42.el6.x86_64
[root@hp-bl280cg6-01 ipa-ca-install]# 

Snip from automation log:
=========================

On Master
=========

Resubmitting "20140725114312" to "dogtag-ipa-renew-agent".
:: [   PASS   ] :: Resubmiting the auditSigningCert for renewal (Expected 0, got 0)
:: [   PASS   ] :: Checking the status of certificate (Expected 0, got 0)
Number of certificates and requests being tracked: 8.
Request ID '20140725114312':
	status: SUBMITTING
	stuck: no
	key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='824831973245'
	certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-renew-agent
	issuer: CN=Certificate Authority,O=TESTRELM.TEST
	subject: CN=CA Audit,O=TESTRELM.TEST
	expires: 2016-07-14 11:42:32 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes
:: [   PASS   ] :: Running 'cat /tmp/bz_1040009.txt' (Expected 0, got 0)
:: [   PASS   ] :: File '/tmp/bz_1040009.txt' should contain 'SUBMITTING' 
:: [   PASS   ] :: Running 'sleep 120' (Expected 0, got 0)
:: [   PASS   ] :: Checking the status of certificate (Expected 0, got 0)
Number of certificates and requests being tracked: 8.
Request ID '20140725114312':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='824831973245'
	certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-renew-agent
	issuer: CN=Certificate Authority,O=TESTRELM.TEST
	subject: CN=CA Audit,O=TESTRELM.TEST
	expires: 2016-07-14 12:06:23 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes
:: [   PASS   ] :: Running 'cat /tmp/bz_1040009.txt' (Expected 0, got 0)
:: [   PASS   ] :: File '/tmp/bz_1040009.txt' should contain 'MONITORING'

On Replica
==========

Resubmitting "20140725115536" to "dogtag-ipa-retrieve-agent-submit".
:: [   PASS   ] :: Resubmiting the auditSigningCert for renewal (Expected 0, got 0)
:: [   PASS   ] :: Checking the status of certificate (Expected 0, got 0)
Number of certificates and requests being tracked: 8.
Request ID '20140725115536':
	status: SUBMITTING
	stuck: no
	key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='655437621144'
	certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=TESTRELM.TEST
	subject: CN=CA Audit,O=TESTRELM.TEST
	expires: 2016-07-14 11:42:32 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes
:: [   PASS   ] :: Running 'cat /tmp/bz_1040009.txt' (Expected 0, got 0)
:: [   PASS   ] :: File '/tmp/bz_1040009.txt' should contain 'SUBMITTING' 
Fri Jul 25 08:13:44 EDT 2014
:: [   PASS   ] :: Running 'sleep 300' (Expected 0, got 0)
:: [   PASS   ] :: Checking the status of certificate (Expected 0, got 0)
Number of certificates and requests being tracked: 8.
Request ID '20140725115536':
	status: MONITORING
	stuck: no
	key pair storage: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin='655437621144'
	certificate: type=NSSDB,location='/var/lib/pki-ca/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
	CA: dogtag-ipa-retrieve-agent-submit
	issuer: CN=Certificate Authority,O=TESTRELM.TEST
	subject: CN=CA Audit,O=TESTRELM.TEST
	expires: 2016-07-14 12:06:23 UTC
	key usage: digitalSignature,nonRepudiation
	pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad
	post-save command: /usr/lib64/ipa/certmonger/restart_pkicad "auditSigningCert cert-pki-ca"
	track: yes
	auto-renew: yes
:: [   PASS   ] :: Running 'cat /tmp/bz_1040009.txt' (Expected 0, got 0)
:: [   PASS   ] :: File '/tmp/bz_1040009.txt' should contain 'MONITORING'

Comment 6 errata-xmlrpc 2014-10-14 07:32:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1383.html