Bug 1779984
| Summary: | The ipa-cert-fix command failed. [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/27-renewed.crt' | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | amitkuma | |
| Component: | ipa | Assignee: | Fraser Tweedale <ftweedal> | |
| Status: | CLOSED ERRATA | QA Contact: | ipa-qe <ipa-qe> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | high | |||
| Version: | 8.1 | CC: | abroy, apeddire, edewata, frenaud, ftweedal, ksiddiqu, myusuf, pasik, pcech, rcritten, sorlov, ssidhaye, tscherf, twoerner | |
| Target Milestone: | rc | Keywords: | Triaged | |
| Target Release: | 8.0 | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | ipa-4.9.3-1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1930586 (view as bug list) | Environment: | ||
| Last Closed: | 2021-11-09 18:21:19 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1930586 | |||
|
Description
amitkuma
2019-12-05 07:36:20 UTC
*** Bug 1779987 has been marked as a duplicate of this bug. *** *** Bug 1779999 has been marked as a duplicate of this bug. *** (In reply to amitkuma from comment #0) ~snip~ > > Request ID '20191204141343': > status: MONITORING > stuck: no > key pair storage: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB',pin set > certificate: > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert > cert-pki-ca',token='NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=Certificate Authority,O=IPA.TEST > subject: CN=Certificate Authority,O=IPA.TEST > expires: 2039-12-04 09:12:31 EST Your CA expires on 2039. ~snip~ > [root@master ~]# date --set="Tue Nov 13 15:23:34 PDT 2090" > Mon Nov 13 17:23:34 EST 2090 You are running the cert-fix tool in 2090, at a point where CA is expired. The prerequisite to run cert-fix tool is that you need to have a valid CA signing certificate. At this point, you have failed to meet prerequiste > [root@master ~]# date --set="Tue Nov 13 15:23:34 PDT 2090"^C > (reverse-i-search)`cert': service ^Crtmonger restart > [root@master ~]# ip-acer^C > [root@master ~]# ipa-cert-fix > > WARNING > > ipa-cert-fix is intended for recovery when expired certificates > prevent the normal operation of FreeIPA. It should ONLY be used > in such scenarios, and backup of the system, especially certificates > and keys, is STRONGLY RECOMMENDED. > > > The following certificates will be renewed: > > Dogtag sslserver certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 29 > Expires: 2027-11-03 21:24:34 The tool identifies (from local nssdb) that sslserver cert is expired. > > Dogtag subsystem certificate: > Subject: CN=CA Subsystem,O=IPA.TEST > Serial: 28 > Expires: 2027-11-03 21:24:23 > > Dogtag ca_ocsp_signing certificate: > Subject: CN=OCSP Subsystem,O=IPA.TEST > Serial: 30 > Expires: 2027-11-03 21:24:15 > > Dogtag ca_audit_signing certificate: > Subject: CN=CA Audit,O=IPA.TEST > Serial: 26 > Expires: 2027-11-03 21:24:22 > > IPA IPA RA certificate: > Subject: CN=IPA RA,O=IPA.TEST > Serial: 27 > Expires: 2027-11-03 21:24:24 > > IPA Apache HTTPS certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 24 > Expires: 2027-11-14 22:29:32 > > IPA LDAP certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 25 > Expires: 2027-11-14 22:29:44 > > IPA KDC certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 23 > Expires: 2027-11-14 22:29:23 > > Enter "yes" to proceed: yes > Proceeding. > Renewed Dogtag sslserver certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 29 > Expires: 2091-02-13 22:24:04 > > Renewed Dogtag subsystem certificate: > Subject: CN=CA Subsystem,O=IPA.TEST > Serial: 16 > Expires: 2027-11-03 21:26:07 > > Renewed Dogtag ca_ocsp_signing certificate: > Subject: CN=OCSP Subsystem,O=IPA.TEST > Serial: 17 > Expires: 2027-11-03 21:26:08 > > Renewed Dogtag ca_audit_signing certificate: > Subject: CN=CA Audit,O=IPA.TEST > Serial: 18 > Expires: 2027-11-03 21:26:09 > > [Errno 2] No such file or directory: > '/etc/pki/pki-tomcat/certs/27-renewed.crt' This is expected because expired CA cannot sign a cert. > The ipa-cert-fix command failed. ~snip~ > > # ipa-cert-fix -v Second time the tool is executed > ipapython.admintool: DEBUG: Not logging to a file > ipalib.install.sysrestore: DEBUG: Loading StateFile from > '/var/lib/ipa/sysrestore/sysrestore.state' > ipalib.install.sysrestore: DEBUG: Loading Index file from > '/var/lib/ipa/sysrestore/sysrestore.index' > ipaserver.install.installutils: DEBUG: httpd is configured > ipaserver.install.installutils: DEBUG: kadmin is configured > ipaserver.install.installutils: DEBUG: dirsrv is configured > ipaserver.install.installutils: DEBUG: pki-tomcatd is configured > ipaserver.install.installutils: DEBUG: install is not configured > ipaserver.install.installutils: DEBUG: krb5kdc is configured > ipaserver.install.installutils: DEBUG: named is configured > ipaserver.install.installutils: DEBUG: filestore has files > ipapython.ipautil: DEBUG: Starting external process > ipapython.ipautil: DEBUG: args=['pki-server', 'cert-fix', '--help'] > ipapython.ipautil: DEBUG: Process finished, return code=0 > ipapython.ipautil: DEBUG: stdout=Usage: pki-server cert-fix [OPTIONS] > > --cert <Cert ID> Fix specified system cert (default: all > certs). > --extra-cert <Serial> Also renew cert with given serial number. > --agent-uid <String> UID of Dogtag agent user > --ldapi-socket <Path> Path to DS LDAPI socket > --ldap-url <URL> LDAP URL (mutually exclusive to > --ldapi-socket) > -i, --instance <instance ID> Instance ID (default: pki-tomcat). > -p, --port <port number> Secure port number (default: 8443). > -v, --verbose Run in verbose mode. > --debug Run in debug mode. > --help Show help message. > > > ipapython.ipautil: DEBUG: stderr= > ipalib.plugable: DEBUG: importing all plugin modules in ipaserver.plugins... ~snip~ > 'sql:/etc/pki/pki-tomcat/alias', '-L', '-n', 'Server-Cert cert-pki-ca', > '-a', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt'] > ipapython.ipautil: DEBUG: Process finished, return code=0 > ipapython.ipautil: DEBUG: stdout=-----BEGIN CERTIFICATE----- > MIIDpzCCAg+gAwIBAgIBHTANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhJUEEu > VEVTVDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MCIYDzIwOTAxMTEz > MjIyNDA0WhgPMjA5MTAyMTMyMjI0MDRaMC0xETAPBgNVBAoMCElQQS5URVNUMRgw > FgYDVQQDDA9tYXN0ZXIuaXBhLnRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw > ggEKAoIBAQC4mD9RpmAgwr+7o2PiIsIqAySEiFWQI2/H3ZVPwTElKgPLZrxIfnCm > UKB2N586BrOjjU45eAYcHJZMHoLjELsUpHhfdqPrjACJqWiZjS4beWnMaxmJpVwv > nGMaSjA4l/Fk6CEC8vxZHFhkNKJIQzB0PNrQKUZSeLr8RQOcGyUvZtRsicnIcbj3 > OK4d5rhkL8ajiroqmyWhwRHlapmN82EgEm48Fa6GOyLh0tHYd67kJSmhjfUsUzso > IzSi4iMHbBgDRswKWRSQfV3OPEdakHl01vSMo2TBQqxEERoZw6bXnU5pLKKHFa3+ > 1+LgRKMRbg8GvPBa+lD4MWmS0beHjxx3AgMBAAGjSDBGMB8GA1UdIwQYMBaAFJfL > G1HRhskWb2mHp7o0OGpstD47MBMGA1UdJQQMMAoGCCsGAQUFBwMBMA4GA1UdDwEB > /wQEAwIE8DANBgkqhkiG9w0BAQsFAAOCAYEAsboOlOYSI4GQNK0akiRn2cvBPFxx > 9t/KvtcWfsQKhgM+vOvd1Zr0qytd2o1gAmc/I+l0tVElnBbmEHCqSBoYy7CKOqwQ > frn2Sa8l1OkyP9dtn2rSMAEWdniEu7zPpj70+yT0b5c+f5tE9ZRPPw3SgkCd3n02 > IJzBtdTCUt2kfk3RobQABSBfh6h1g4unwt/TWGHqPqP6fm2zamdzY4LVisYYTxd/ > Q9GBDy6N0cnEPyGUhEIFLka9A7HNsL8hy5pd3HTYJK7d9zm57NZlqISy66v7vHwI > 3p7dggI/lR/X2JOZIDItgXybsTPzkRwBtmTTzPF+8DBWQ8Y55dN5HRhT6M32aJ3B > nsv784zK/0FOieJJ4ajpKbudlZohVJTKQeoEMozovT9h9q3HE8lDkGP2xukFbHvl > B1jiX9JdMSTzSu4TWNkNwTurMv9Otuq0CfzQ4H4GNh3XVfr8totgq1my+zokTjzD > l0jqJzkOQDWV+gjB8jMxWQfiK9/76JYYgZd5 > -----END CERTIFICATE----- This is a temporary sslserver cert that was created while you ran the cert-fix for the first time. This cert expires on: February 13, 2091 . This cert WILL NOT appear in LDAP. The tool now thinks that sslserver cert is valid and removes it from its renewal list. ~snip~ > WARNING > > ipa-cert-fix is intended for recovery when expired certificates > prevent the normal operation of FreeIPA. It should ONLY be used > in such scenarios, and backup of the system, especially certificates > and keys, is STRONGLY RECOMMENDED. > > > The following certificates will be renewed: > > Dogtag subsystem certificate: > Subject: CN=CA Subsystem,O=IPA.TEST > Serial: 28 > Expires: 2027-11-03 21:24:23 > > Dogtag ca_ocsp_signing certificate: > Subject: CN=OCSP Subsystem,O=IPA.TEST > Serial: 30 > Expires: 2027-11-03 21:24:15 > > Dogtag ca_audit_signing certificate: > Subject: CN=CA Audit,O=IPA.TEST > Serial: 26 > Expires: 2027-11-03 21:24:22 > > IPA IPA RA certificate: > Subject: CN=IPA RA,O=IPA.TEST > Serial: 27 > Expires: 2027-11-03 21:24:24 > > IPA Apache HTTPS certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 24 > Expires: 2027-11-14 22:29:32 > > IPA LDAP certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 25 > Expires: 2027-11-14 22:29:44 > > IPA KDC certificate: > Subject: CN=master.ipa.test,O=IPA.TEST > Serial: 23 > Expires: 2027-11-14 22:29:23 sslserver is not on this list. > > Enter "yes" to proceed: yes > Proceeding. > ipapython.ipautil: DEBUG: Starting external process > ipapython.ipautil: DEBUG: args=['pki-server', 'cert-fix', '--ldapi-socket', > '/var/run/slapd-IPA-TEST.socket', '--agent-uid', 'ipara', '--cert', > 'subsystem', '--cert', 'ca_ocsp_signing', '--cert', 'ca_audit_signing', > '--extra-cert', '27', '--extra-cert', '24', '--extra-cert', '25', > '--extra-cert', '23'] sslserver is not on this list. ~snip~ NOTE: If this tool is executed with a valid CA signing cert for the third time on the same machine, the renewals of other certs may succeed but sslserver cert will not get renewed and so the CA won't start. This issue - as reported - is ultimately caused by missing certificates, resulting in Dogtag not starting properly. That problem itself is out of scope for ipa-cert-fix to address, but I will investigate how our tools can respond better to this sort of failure. In particular: 1. `pki-server cert-fix' appears to exit cleanly even though it was unable to renew the certificates. This behaviour is incorrect. The expected behaviour in this scenario is to return nonzero. `ipa-cert-fix' should then detect that `pki-server cert-fix' failed, and not attempt to locate renewed certificates but instead report the error. 2. `ipa-cert-fix' should have better handling of the case where the expected renewed certificates are not present. Although this indicative of an error in `pki-server cert-fix', we should fail with an error message, not a traceback. I am proceeding with the above. Upstream PR: https://github.com/freeipa/freeipa/pull/5579 This change improves the handling of Dogtag `pki-server cert-fix` errors in `ipa-cert-fix`. The actual cause of the error in this BZ is in Dogtag, and is tracked by https://bugzilla.redhat.com/show_bug.cgi?id=1930586. Upstream ticket: https://pagure.io/freeipa/issue/8721 master PR: https://github.com/freeipa/freeipa/pull/5579 ipa-4-9 PR: https://github.com/freeipa/freeipa/pull/5590 master: 8c2c6f8 (HEAD) ipa-cert-fix: improve handling of 'pki-server cert-fix' failure ipa-4-9: f2b1b5b (HEAD) ipa-cert-fix: improve handling of 'pki-server cert-fix' failure Moving to POST. Fixed upstream tests: master: https://pagure.io/freeipa/c/16057898af69ee795c9c2871ce5936a49d108e1c Upstream test cases test_missing_startup and test_expired_CA_cert are not in RHEL package yet, so for verification I am manually executing test suite test_integration/test_ipa_cert_fix.py from upstream master branch against latest nightly compose with ipa-server-4.9.5-1.module+el8.5.0+11410+91a33fe4.x86_64
The test "test_missing_startup" failed:
================================================================================================================= FAILURES ==================================================================================================================
____________________________________________________________________________________________________ TestIpaCertFix.test_missing_startup ____________________________________________________________________________________________________
self = <ipatests.test_integration.test_ipa_cert_fix.TestIpaCertFix object at 0x7f050b591048>, expire_cert_critical = <function expire_cert_critical.<locals>._expire_cert_critical at 0x7f050c84c2f0>
def test_missing_startup(self, expire_cert_critical):
"""
Test ipa-cert-fix fails when startup directive is missing from CS.cfg
This test checks that if 'selftests.container.order.startup' directive
is missing from CS.cfg, ipa-cert-fix fails and throw proper error
message. It also checks that underlying command 'pki-server cert-fix'
should fail to renew the cert.
related: https://pagure.io/freeipa/issue/8721
"""
expire_cert_critical(self.master)
# pki must be stopped in order to edit CS.cfg
self.master.run_command(['ipactl', 'stop'])
self.master.run_command([
'sed', '-i', r'/selftests\.container\.order\.startup/d',
paths.CA_CS_CFG_PATH
])
# dirsrv needs to be up in order to run ipa-cert-fix
self.master.run_command(['ipactl', 'start',
'--ignore-service-failures'])
result = self.master.run_command(['ipa-cert-fix', '-v'],
stdin_text='yes\n',
raiseonerr=False)
err_msg1 = "ERROR: 'selftests.container.order.startup'"
# check that pki-server cert-fix command fails
err_msg2 = ("ERROR: CalledProcessError(Command "
"['pki-server', 'cert-fix'")
> assert err_msg1 and err_msg2 in result.stderr_text
E assert ("ERROR: 'selftests.container.order.startup'" and "ERROR: CalledProcessError(Command ['pki-server', 'cert-fix'" in "ipapython.admintool: DEBUG: Not logging to a file\nipalib.sysrestore: DEBUG: Loading StateFile from '/var/lib/ipa/sys...r=ipa: INFO: The ipactl command was successful\n\nipapython.admintool: INFO: The ipa-cert-fix command was successful\n")
E + where "ipapython.admintool: DEBUG: Not logging to a file\nipalib.sysrestore: DEBUG: Loading StateFile from '/var/lib/ipa/sys...r=ipa: INFO: The ipactl command was successful\n\nipapython.admintool: INFO: The ipa-cert-fix command was successful\n" = <pytest_multihost.transport.SSHCommand object at 0x7f050b53ff98>.stderr_text
test_integration/test_ipa_cert_fix.py:235: AssertionError
The test `test_missing_startup`, which was added in commit https://pagure.io/freeipa/c/16057898af69ee795c9c2871ce5936a49d108e1c, is not valid. The behaviour of Dogtag where it fails to successfully execute `pki-server cert-fix` when the CS.cfg parameter was missing was a bug. The bug was fixed by commit https://github.com/dogtagpki/pki/pull/3466/. Therefore, I think that the whole `test_missing_startup` test should be be deleted. Or, it should be changed to verify that the missing CS.cfg parameter does NOT cause `pki-server cert-fix` (and `ipa-cert-fix`) to fail. But this might not be robust, in case the test ends up being run in an environment with an older version of Dogtag that does not have the fix. tl;dr `test_missing_startup` is expecting failure when it should not. I leave to IPA developers to work out how best to proceed. Clearing NEEDINFO. Fixed upstream test: ipa-4-9: https://pagure.io/freeipa/c/02c0da3ef74948579106aab4b669f6e64dd60b24 @myusuf the test code needs to be fixed:
if (tasks.get_pki_version(self.master)
< tasks.parse_version('10.11.0')):
assert (err_msg1 in result.stderr_text
and err_msg2 in result.stderr_text)
else:
> assert warn_msg in result.stdout_text
It should look for the message in stderr_text, not in stdout_text.
Moving back the BZ to ON_QA as the issue is in the test, not in the fix. PR raised to fix the test code: https://github.com/freeipa/freeipa/pull/5938 Test fixed in ipa master: https://pagure.io/freeipa/c/bb38fbca911276b2486120a46d1cd75f4e358ef9 version: ipa-server-4.9.6-4.module+el8.5.0+11912+1b4496cf.x86_64.rpm
2021-08-04T10:35:19+0000 ============================= test session starts ==============================
2021-08-04T10:35:19+0000 platform linux -- Python 3.9.6, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
2021-08-04T10:35:19+0000 cachedir: .pytest_cache
2021-08-04T10:35:19+0000 metadata: {'Python': '3.9.6', 'Platform': 'Linux-5.13.6-200.fc34.x86_64-x86_64-with-glibc2.33', 'Packages': {'pytest': '3.10.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'html': '1.22.1', 'metadata': '1.11.0', 'multihost': '3.0', 'sourceorder': '0.5'}}
2021-08-04T10:35:19+0000 rootdir: /tmp/wp/freeipa, inifile: tox.ini
2021-08-04T10:35:19+0000 plugins: html-1.22.1, metadata-1.11.0, multihost-3.0, sourceorder-0.5
2021-08-04T10:35:19+0000 collecting ... collected 5 items / 3 deselected
2021-08-04T10:35:19+0000
2021-08-04T10:47:59+0000 ipatests/test_integration/test_ipa_cert_fix.py::TestIpaCertFix::test_missing_startup PASSED [ 50%]
2021-08-04T10:57:37+0000 ipatests/test_integration/test_ipa_cert_fix.py::TestIpaCertFix::test_expired_CA_cert PASSED [100%]
2021-08-04T10:57:37+0000
Automation passed. Hence marking as verified. Above report.html is attached for reference.
Test fixed in ipa: ipa-4-9: https://pagure.io/freeipa/c/96dd8ac1cd2e7fb8177d83e7ba5c6d79f4216ea3 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ipa bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4230 |