I am moving this issue to pki-server component as the error happens inside the tool 'pki-server cert-fix' which is used internally by ipa-cert-fix.

I found upstream PKI tickets to retrieve the certreq and cert from LDAP directly
instead of storing a redundant copy in CS.cfg:
https://pagure.io/dogtagpki/issue/1552
https://pagure.io/dogtagpki/issue/1598

IMO, there are 2 ways to fix this BZ:

Option 1 [Low risk]:
====================
Retrieve the cert request value from /var/lib/certmonger/requests and stick it into CS.cfg

This needs to be implemented by ipa-cert-fix

Option 2 [High risk]:
=====================
Remove the *.certreq and *.cert from CS.cfg and retrieve it from LDAP. This means, we need to
handle different installation scenarios.

I have to dig deeper to see the impact. This must be implemented in PKI and tested *thoroughly*


Question to Marco: is there an empty value in *.sslserver.certreq or the entire key/value pair is
missing from CS.cfg?

Dinesh: this is probably a symptom of an older bug that caused the .certreq directives
to disappear: https://pagure.io/freeipa/issue/7288.

As for how to fix, I am OK with Option 1.  Because we are root we could construct
a CSR ourselves, too.  Option 2 is too much effort to make reliable.

I'm being hit by this. In anwser to Dinesh's question in #14, here is what I have for certreq in my CS.cfg:

[root@localhost ca]# grep certreq  CS.cfg
ca.audit_signing.certreq=None
ca.ocsp_signing.certreq=None
ca.subsystem.certreq=None

(In reply to Dinesh Prasanth from comment #14)
> I found upstream PKI tickets to retrieve the certreq and cert from LDAP
> directly
> instead of storing a redundant copy in CS.cfg:
> https://pagure.io/dogtagpki/issue/1552
> https://pagure.io/dogtagpki/issue/1598
> 
> IMO, there are 2 ways to fix this BZ:
> 
> Option 1 [Low risk]:
> ====================
> Retrieve the cert request value from /var/lib/certmonger/requests and stick
> it into CS.cfg
> 
> This needs to be implemented by ipa-cert-fix
> 
> Option 2 [High risk]:
> =====================
> Remove the *.certreq and *.cert from CS.cfg and retrieve it from LDAP. This
> means, we need to
> handle different installation scenarios.
> 
> I have to dig deeper to see the impact. This must be implemented in PKI and
> tested *thoroughly*
> 
> 
> Question to Marco: is there an empty value in *.sslserver.certreq or the
> entire key/value pair is
> missing from CS.cfg?

In each of the 3 occurrences I've seen, the entire key/value pair is missing from CS.cfg.

(In reply to Matt Mossholder from comment #16)
> I'm being hit by this. In anwser to Dinesh's question in #14, here is what I
> have for certreq in my CS.cfg:
> 
> [root@localhost ca]# grep certreq  CS.cfg
> ca.audit_signing.certreq=None
> ca.ocsp_signing.certreq=None
> ca.subsystem.certreq=None

Note that you are missing the ca.sslserver.certreq .... This is the same as Marco observed.


(In reply to Marco Rhodes from comment #17)
> In each of the 3 occurrences I've seen, the entire key/value pair is missing
> from CS.cfg.

As Fraser pointed out in #c15, this looks like the symptom of an older bug that was already fixed.

As per #comment 15, I am reassigning this bug to IPA.

Another possibility to remediate this issue is to fix up CS.cfg during ipa-server-upgrade.
IMO this is more desirable than adding extra complexity to ipa-cert-fix.

Thank you taking your time and submitting this request for Red Hat Enterprise Linux 7. Unfortunately, this bug cannot be kept even as a stretch goal and was postponed to RHEL8.

Marco,

Do you happen to have reproducer steps? We were wondering if setting ca.sslserver.certreq=None is sufficient to workaround this bug.


If so, we'd move the component to pki-core and fix this in Dogtag.

Otherwise, we'd probably keep this in IPA, as pki-server cert-fix is meant to be run offline (and wouldn't necessarily have LDAP access). 


Thanks!

Reproducer steps:

1. install ipa server with --no-ntp --no-dnssec-validation (for test purpose, as the test will manipulate the date), then stop the services: ipactl stop
2. move the date post expiration: date -s +2years
3. edit /etc/pki/pki-tomcat/ca/CS.cfg and delete the line ca.sslserver.certreq=MII...
4. restart the services: ipactl start -> the command fails as pki fails to start
5. force-restart the services (389ds must be running in order to launch ipa-cert-fix): ipactl start --ignore-service-failures
6. launch ipa-cert-fix -> the command fails because it doesn't find the CSR


Summary of what we know so far:
===============================
1. How do we get into this situation:
In older versions of IPA, a known issue (https://pagure.io/freeipa/issue/7288) resulted in removal of the directive ca.sslserver.certreq from /etc/pki/pki-tomcat/ca/CS.cfg after the renewal of "Server-Cert cert-pki-ca".

2. What is the consequence:
When ipa-cert-fix is launched and the directive is missing, the command fails. The command is internally calling "pki-server cert-fix", that should produce a file containing the renewed cert in /etc/pki/pki-tomcat/certs/sslserver.crt, but:
- pki-server cert-fix fails if the directive is missing.
- ipa-cert-fix does properly not handle the error returned by pki-server cert-fix and goes on, trying to read the file sslserver.crt that wasn't created.


Alex suggested to check the behavior of ipa-cert-fix if we manually write "ca.sslserver.certreq=None" to the CS.cfg file, but this doesn't fix the problem: "pki-server cert-fix" also exits with 1 but sslserver.crt is created as an empty file and ipa-cert-fix fails when trying to initialize the cert from the file content.

IMO the fix should be done on IPA side. We can either add an upgrade plugin that re-adds ca.sslserver.certreq into CS.cfg or modify ipa-cert-fix for it to read the CSR from LDAP (ipa-cert-fix already requires the Directory Server to be running when it is launched). We also need to properly handle when "pki-server cert-fix" exits on error.

Upstream ticket:
https://pagure.io/freeipa/issue/8618

Fixed upstream
master:
https://pagure.io/freeipa/c/b8ece644e849f8341544bcadba8b19fd888dd90d
https://pagure.io/freeipa/c/1a988ba96d7563d66e47db0d6bcc13cad7a85149
https://pagure.io/freeipa/c/98711e8edf052211e5ccc3c1ac3c2367c6f23aff


TestCase provided upstream in a new file test_integration/test_ipa_cert_fix.py

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/f36e518b5704b02b81a4b80a1b84c429594cf5ce
https://pagure.io/freeipa/c/eb711f781322657b0b3d77332f2462ecfb27db95
https://pagure.io/freeipa/c/7f2be8a45a1d4baff0074cf4d8c446e3d08db795

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/c9e0aa31cbd614ef7da5e94aeca3c292219d364a
https://pagure.io/freeipa/c/8f53cc36723e5ef84fc2c3cc143d8e0f2c7f283b
https://pagure.io/freeipa/c/c1207c34a768ed5eb11a161c989484183281892b

Pre-verified using compose: rhel-8.4.0-mbs/9665-1253-idm/
Using version : 
2021-01-29T07:48:00     name: ipa-server
2021-01-29T07:48:00     release: 1.module+el8.4.0+9665+c9815399
2021-01-29T07:48:00     source: rpm
2021-01-29T07:48:00     version: 4.9.1

============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /usr/libexec/platform-python
cachedir: /home/cloud-user/.pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-279.el8.x86_64-x86_64-with-redhat-8.4-Ootpa', 'Packages': {'pytest': '3.10.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '1.22.1', 'multihost': '3.0', 'sourceorder': '0.5'}}
rootdir: /usr/lib/python3.6/site-packages/ipatests, inifile:
plugins: metadata-1.11.0, html-1.22.1, multihost-3.0, sourceorder-0.5
collecting ... collected 1 item

test_integration/test_ipa_cert_fix.py::TestIpaCertFix::test_missing_csr PASSED [100%]

---------------- generated xml file: /home/cloud-user/junit.xml ----------------
----------- generated html file: file:///home/cloud-user/report.html -----------
========================= 1 passed in 1019.04 seconds ==========================

Verified using nightly compose and using version:

2021-02-02T13:14:19+0000     name: ipa-server
2021-02-02T13:14:19+0000     release: 1.module+el8.4.0+9665+c9815399
2021-02-02T13:14:19+0000     source: rpm
2021-02-02T13:14:19+0000     version: 4.9.1

Test log:
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /usr/libexec/platform-python
cachedir: /home/cloud-user/.pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-280.el8.x86_64-x86_64-with-redhat-8.4-Ootpa', 'Packages': {'pytest': '3.10.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '1.22.1', 'multihost': '3.0', 'sourceorder': '0.5'}}
rootdir: /usr/lib/python3.6/site-packages/ipatests, inifile:
plugins: metadata-1.11.0, html-1.22.1, multihost-3.0, sourceorder-0.5
collecting ... collected 1 item

test_integration/test_ipa_cert_fix.py::TestIpaCertFix::test_missing_csr PASSED [100%]

---------------- generated xml file: /home/cloud-user/junit.xml ----------------
----------- generated html file: file:///home/cloud-user/report.html -----------
========================= 1 passed in 1127.29 seconds ==========================

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: idm:DL1 and idm:client security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1846