1788907 – Renewed certs are not picked up by IPA CAs

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1788907 - Renewed certs are not picked up by IPA CAs

Summary: Renewed certs are not picked up by IPA CAs

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	ipa
Sub Component:
Version:	7.7
Hardware:	All
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Florence Blanc-Renaud
QA Contact:	ipa-qe
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1788833 1790663
TreeView+	depends on / blocked

Reported:	2020-01-08 11:07 UTC by Ahmed Nazmy
Modified:	2023-12-15 17:09 UTC (History)
CC List:	5 users (show)
Fixed In Version:	ipa-4.6.6-12.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1790663 (view as bug list)
Environment:
Last Closed:	2020-09-29 19:58:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	FREEIPA-9462	None	None	None	2023-02-16 07:56:30 UTC
Red Hat Knowledge Base (Solution)	4773021	None	None	None	2020-05-27 07:31:08 UTC
Red Hat Product Errata	RHSA-2020:3936	None	None	None	2020-09-29 19:59:23 UTC

Description Ahmed Nazmy 2020-01-08 11:07:58 UTC

Description of problem:

When the CA subsystem below associated certs renewal is due:
~~~
'subsystemCert cert-pki-ca'
'ocspSigningCert cert-pki-ca'
'auditSigningCert cert-pki-ca'
'/var/lib/ipa/ra-agent.pem'
~~~

A non-renewal master CA might submit a renewal request before the renewal master actually updating the certs. This is expected. 

But, if certmonger on this node is stopped/interrupted for any reason while the resubmit request is in "CA_WORKING" state it will not be able to track changes to this cert anymore.

Version-Release number of selected component (if applicable):

certmonger-0.78.4-11.el7.x86_64
ipa-server-4.6.5-11.el7_7.3.x86_64

How reproducible:

Always

Steps to Reproduce:

1- Install an IPA master + one or more CA replicas, self signed certs.

2- On a non-renewal master resubmit a cert request:
  # getcert resubmit -f /var/lib/ipa/ra-agent.pem

3- while the submission is in "CA_WORKING" state, stop then start certmonger

4- Now we'll get invalid cookie state for "ra-agent.pem"
   # getcert list -f /var/lib/ipa/ra-agent.pem
    [...]
    ca-error: Invalid cookie: u''
    [...]

5- This CA will not pickup any changes to this cert anymore.


Actual results:

Newly generated certs by the renewal master are not picked by affected CAs. Breaking operations related to certs. For example, ipa host-add/host-del

Expected results:

Certmonger track certs in between reboots/interruptions and while in "CA_WORKING" state.

Comment 4 Rob Crittenden 2020-01-08 20:05:18 UTC

I can reproduce it.

The cookie after the resubmit in this case contains: "{"profile": "caServerCert", "cookie": ""}"

I'm still wrapping my head around the usage of cookie but it is an attempt to maintain state for a renewal. There are two states that I can see: retrieve and store, along with a number of attempts. I haven't yet been able to trace through all the code but pretty obviously there is a path that skips setting this state which dooms future renewals from working.

Comment 7 Rob Crittenden 2020-01-09 20:06:17 UTC

Upstream ticket:
https://pagure.io/freeipa/issue/8164

Comment 8 Rob Crittenden 2020-01-09 21:40:52 UTC

The cookie value is not needed when renewing certs via LDAP so relax the requirement for one. If one is needed it is enforced at the time.

Upstream PR https://github.com/freeipa/freeipa/pull/4111

Comment 9 Rob Crittenden 2020-01-13 20:42:40 UTC

Fixed upstream
master:
https://pagure.io/freeipa/c/b5b9efeb57c010443c33c6f14f831abdbd804e78

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/3d7d58d8214f3c899c0afd1a3a6a6678f38b7b39

Fixed upstream
ipa-4-7:
https://pagure.io/freeipa/c/e5983600bfc0f143c3a6732be6532e48d9faaf15

Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/73d415b72da8a57a2369a55b1533b45f36daf544

Comment 10 Florence Blanc-Renaud 2020-01-28 09:57:11 UTC

RHEL-7.8 is already near the end of a Development Phase and development is being wrapped up. This bug is being moved to RHEL 7.9.
If you believe this particular bug should be reconsidered for 7.8, please let us know.

Comment 13 Rob Crittenden 2020-03-30 19:09:13 UTC

Upstream test

Fixed upstream
master:
https://pagure.io/freeipa/c/58ad7b74eb4136ff8cd10ad6caf463df7403f5b3

Comment 14 Rob Crittenden 2020-03-31 15:30:21 UTC

ipa-4-8:
https://pagure.io/freeipa/c/77409e2be01984ff9bdd61989a94845cf9206116

Comment 15 Rob Crittenden 2020-04-06 12:14:21 UTC

Upstream test

Fixed upstream
ipa-4-6:
https://pagure.io/freeipa/c/f77c2f122802aad38aa12e5fc6c9cc49aaa49dfa
https://pagure.io/freeipa/c/4235ccba769653fc0d906d4c79bdde5dbfcdad74

Comment 16 Sumedh Sidhaye 2020-04-16 09:38:42 UTC

Reproducer : pre-patch systems

[root@master ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@replica ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

master

[root@cloud-qe-19 ~]# rpm -q ipa-server certmonger 
ipa-server-4.6.5-11.el7.x86_64
certmonger-0.78.4-11.el7.x86_64
[root@cloud-qe-19 ~]# 


replica

[root@cloud-qe-21 ~]# rpm -q ipa-server certmonger
ipa-server-4.6.5-11.el7.x86_64
certmonger-0.78.4-11.el7.x86_64



[root@replica ~]# getcert list -f /var/lib/ipa/ra-agent.pem
Number of certificates and requests being tracked: 9.
Request ID '20200305081550':
	status: CA_WORKING
	stuck: no
	key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
	certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
	CA: dogtag-ipa-ca-renew-agent
	issuer: CN=Certificate Authority,O=TESTREALM.TEST
	subject: CN=IPA RA,O=TESTREALM.TEST
	expires: 2022-02-23 05:19:20 UTC
	key usage: digitalSignature,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
	post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
	track: yes
	auto-renew: yes
[root@replica ~]# systemctl stop certmonger; sleep 5; systemctl start certmonger

[root@replica ~]#tail /var/log/messages

Mar  5 03:20:01 cloud-qe-21 systemd: Started Session 32 of user root.
Mar  5 03:28:47 cloud-qe-21 ntpd[3845]: 0.0.0.0 0512 02 freq_set ntpd 0.000 PPM
Mar  5 03:28:47 cloud-qe-21 ntpd[3845]: 0.0.0.0 0515 05 clock_sync
Mar  5 03:33:47 cloud-qe-21 dogtag-ipa-ca-renew-agent-submit: Updated certificate not available
Mar  5 03:34:40 cloud-qe-21 systemd: Stopping Certificate monitoring and PKI enrollment...
Mar  5 03:34:40 cloud-qe-21 systemd: Stopped Certificate monitoring and PKI enrollment.
Mar  5 03:34:52 cloud-qe-21 systemd: Starting Certificate monitoring and PKI enrollment...
Mar  5 03:34:53 cloud-qe-21 systemd: Started Certificate monitoring and PKI enrollment.
Mar  5 03:35:14 cloud-qe-21 certmonger: 2020-03-05 03:35:14 [7978] Invalid cookie: u''


Verification using automation: test_integration/test_cert.py::TestCertmongerInterruption::test_certmomger_tracks_renewed_certs_during_interruption

Builds used in verification:
ipa-client-4.6.8-1.el7.x86_64
ipa-client-common-4.6.8-1.el7.noarch
ipa-common-4.6.8-1.el7.noarch
ipa-server-4.6.8-1.el7.x86_64
ipa-server-common-4.6.8-1.el7.noarch
ipa-server-dns-4.6.8-1.el7.noarch


============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-3.10.1, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python2
cachedir: .pytest_cache
metadata: {'Python': '2.7.5', 'Platform': 'Linux-3.10.0-1133.el7.x86_64-x86_64-with-redhat-7.9-Maipo', 'Packages': {'py': '1.8.1', 'pytest': '3.10.1', 'pluggy': '0.13.1'}, 'Plugins': {u'html': u'1.22.1', u'multihost': u'1.1', u'sourceorder': u'0.5', u'metadata': u'1.8.0'}}
rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile:
plugins: metadata-1.8.0, html-1.22.1, multihost-1.1, sourceorder-0.5
collecting ... collected 1 item

test_integration/test_cert.py::TestCertmongerInterruption::test_certmomger_tracks_renewed_certs_during_interruptions PASSED [100%]

---------------- generated xml file: /home/cloud-user/junit.xml ----------------
----------- generated html file: file:///home/cloud-user/report.html -----------
========================== 1 passed in 907.43 seconds ==========================

Comment 18 errata-xmlrpc 2020-09-29 19:58:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: ipa security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3936

Note You need to log in before you can comment on or make changes to this bug.