Bug 1788907 - Renewed certs are not picked up by IPA CAs
Summary: Renewed certs are not picked up by IPA CAs
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.7
Hardware: All
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Florence Blanc-Renaud
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On:
Blocks: 1788833 1790663
TreeView+ depends on / blocked
 
Reported: 2020-01-08 11:07 UTC by Ahmed Nazmy
Modified: 2020-09-20 13:30 UTC (History)
5 users (show)

Fixed In Version: ipa-4.6.6-12.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1790663 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4773021 None None None 2020-05-27 07:31:08 UTC

Description Ahmed Nazmy 2020-01-08 11:07:58 UTC
Description of problem:

When the CA subsystem below associated certs renewal is due:
~~~
'subsystemCert cert-pki-ca'
'ocspSigningCert cert-pki-ca'
'auditSigningCert cert-pki-ca'
'/var/lib/ipa/ra-agent.pem'
~~~

A non-renewal master CA might submit a renewal request before the renewal master actually updating the certs. This is expected. 

But, if certmonger on this node is stopped/interrupted for any reason while the resubmit request is in "CA_WORKING" state it will not be able to track changes to this cert anymore.

Version-Release number of selected component (if applicable):

certmonger-0.78.4-11.el7.x86_64
ipa-server-4.6.5-11.el7_7.3.x86_64

How reproducible:

Always

Steps to Reproduce:

1- Install an IPA master + one or more CA replicas, self signed certs.

2- On a non-renewal master resubmit a cert request:
  # getcert resubmit -f /var/lib/ipa/ra-agent.pem

3- while the submission is in "CA_WORKING" state, stop then start certmonger

4- Now we'll get invalid cookie state for "ra-agent.pem"
   # getcert list -f /var/lib/ipa/ra-agent.pem
    [...]
    ca-error: Invalid cookie: u''
    [...]

5- This CA will not pickup any changes to this cert anymore.


Actual results:

Newly generated certs by the renewal master are not picked by affected CAs. Breaking operations related to certs. For example, ipa host-add/host-del

Expected results:

Certmonger track certs in between reboots/interruptions and while in "CA_WORKING" state.

Comment 4 Rob Crittenden 2020-01-08 20:05:18 UTC
I can reproduce it.

The cookie after the resubmit in this case contains: "{"profile": "caServerCert", "cookie": ""}"

I'm still wrapping my head around the usage of cookie but it is an attempt to maintain state for a renewal. There are two states that I can see: retrieve and store, along with a number of attempts. I haven't yet been able to trace through all the code but pretty obviously there is a path that skips setting this state which dooms future renewals from working.

Comment 7 Rob Crittenden 2020-01-09 20:06:17 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/8164

Comment 8 Rob Crittenden 2020-01-09 21:40:52 UTC
The cookie value is not needed when renewing certs via LDAP so relax the requirement for one. If one is needed it is enforced at the time.

Upstream PR https://github.com/freeipa/freeipa/pull/4111

Comment 10 Florence Blanc-Renaud 2020-01-28 09:57:11 UTC
RHEL-7.8 is already near the end of a Development Phase and development is being wrapped up. This bug is being moved to RHEL 7.9.
If you believe this particular bug should be reconsidered for 7.8, please let us know.

Comment 13 Rob Crittenden 2020-03-30 19:09:13 UTC
Upstream test

Fixed upstream
master:
https://pagure.io/freeipa/c/58ad7b74eb4136ff8cd10ad6caf463df7403f5b3

Comment 16 Sumedh Sidhaye 2020-04-16 09:38:42 UTC
Reproducer : pre-patch systems

[root@master ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@replica ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

master

[root@cloud-qe-19 ~]# rpm -q ipa-server certmonger 
ipa-server-4.6.5-11.el7.x86_64
certmonger-0.78.4-11.el7.x86_64
[root@cloud-qe-19 ~]# 


replica

[root@cloud-qe-21 ~]# rpm -q ipa-server certmonger
ipa-server-4.6.5-11.el7.x86_64
certmonger-0.78.4-11.el7.x86_64



[root@replica ~]# getcert list -f /var/lib/ipa/ra-agent.pem
Number of certificates and requests being tracked: 9.
Request ID '20200305081550':
	status: CA_WORKING
	stuck: no
	key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
	certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
	CA: dogtag-ipa-ca-renew-agent
	issuer: CN=Certificate Authority,O=TESTREALM.TEST
	subject: CN=IPA RA,O=TESTREALM.TEST
	expires: 2022-02-23 05:19:20 UTC
	key usage: digitalSignature,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
	post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
	track: yes
	auto-renew: yes
[root@replica ~]# systemctl stop certmonger; sleep 5; systemctl start certmonger

[root@replica ~]#tail /var/log/messages

Mar  5 03:20:01 cloud-qe-21 systemd: Started Session 32 of user root.
Mar  5 03:28:47 cloud-qe-21 ntpd[3845]: 0.0.0.0 0512 02 freq_set ntpd 0.000 PPM
Mar  5 03:28:47 cloud-qe-21 ntpd[3845]: 0.0.0.0 0515 05 clock_sync
Mar  5 03:33:47 cloud-qe-21 dogtag-ipa-ca-renew-agent-submit: Updated certificate not available
Mar  5 03:34:40 cloud-qe-21 systemd: Stopping Certificate monitoring and PKI enrollment...
Mar  5 03:34:40 cloud-qe-21 systemd: Stopped Certificate monitoring and PKI enrollment.
Mar  5 03:34:52 cloud-qe-21 systemd: Starting Certificate monitoring and PKI enrollment...
Mar  5 03:34:53 cloud-qe-21 systemd: Started Certificate monitoring and PKI enrollment.
Mar  5 03:35:14 cloud-qe-21 certmonger: 2020-03-05 03:35:14 [7978] Invalid cookie: u''


Verification using automation: test_integration/test_cert.py::TestCertmongerInterruption::test_certmomger_tracks_renewed_certs_during_interruption

Builds used in verification:
ipa-client-4.6.8-1.el7.x86_64
ipa-client-common-4.6.8-1.el7.noarch
ipa-common-4.6.8-1.el7.noarch
ipa-server-4.6.8-1.el7.x86_64
ipa-server-common-4.6.8-1.el7.noarch
ipa-server-dns-4.6.8-1.el7.noarch


============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-3.10.1, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python2
cachedir: .pytest_cache
metadata: {'Python': '2.7.5', 'Platform': 'Linux-3.10.0-1133.el7.x86_64-x86_64-with-redhat-7.9-Maipo', 'Packages': {'py': '1.8.1', 'pytest': '3.10.1', 'pluggy': '0.13.1'}, 'Plugins': {u'html': u'1.22.1', u'multihost': u'1.1', u'sourceorder': u'0.5', u'metadata': u'1.8.0'}}
rootdir: /usr/lib/python2.7/site-packages/ipatests, inifile:
plugins: metadata-1.8.0, html-1.22.1, multihost-1.1, sourceorder-0.5
collecting ... collected 1 item

test_integration/test_cert.py::TestCertmongerInterruption::test_certmomger_tracks_renewed_certs_during_interruptions PASSED [100%]

---------------- generated xml file: /home/cloud-user/junit.xml ----------------
----------- generated html file: file:///home/cloud-user/report.html -----------
========================== 1 passed in 907.43 seconds ==========================


Note You need to log in before you can comment on or make changes to this bug.