1641119 – CC: CA/OCSP startup fail on SystemCertsVerification if enableOCSP is true

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1641119 - CC: CA/OCSP startup fail on SystemCertsVerification if enableOCSP is true

Summary: CC: CA/OCSP startup fail on SystemCertsVerification if enableOCSP is true

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	pki-core
Sub Component:
Version:	7.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Jack Magne
QA Contact:	Asha Akkiangady
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1657922
TreeView+	depends on / blocked

Reported:	2018-10-19 16:44 UTC by Christina Fu
Modified:	2019-08-06 13:07 UTC (History)
CC List:	5 users (show)
Fixed In Version:	pki-core-10.5.16-2.el7
Doc Type:	Bug Fix
Doc Text:	Cause: This issue arises when restarting either a CA or OCSP subsystem that has been configured for OCSP checking. Consequence: When such a server is restarted the selftests designed to test the validity of the subsystem certs in a given subystem fail due to the presence of OCSP checking. Fix: The fix is to modify the the selftests for CA and OCSP only to perform a simpler set of cert validity tests, which do not cause problems. This new behavior now takes place by default. The original behavior can be forced with the following CS.cfg value: selftests.plugin.SystemCertsVerification.FullCAandOCSPVerify=true With this setting the original behavior will take place. This would only be desired in cases where OCSP checking has not been configured, and the full cert validity tests are desired. Result:
Clone Of:
Clones:	1657922 (view as bug list)
Environment:
Last Closed:	2019-08-06 13:07:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:2228	0	None	None	None	2019-08-06 13:07:40 UTC

Description Christina Fu 2018-10-19 16:44:41 UTC

This came out from supporting Gossamer (CC lab) on getting OCSP installed.

We found out that CA (and possibly OCSP) won't start if enableOCSP is true and the url it follows points to itself, and our workaround is to tell people to install external OCSP.  Gossamer had real concern about this. While I agreed with the concern (it is a weird workaround), I stated that when push comes to shove and if there is no way around, we just might have to go with it, as I am not aware of anything in the PP or relevant RFCs that would prevent that.  But I agree that we should still try to investigate and see if there is a better solution to that.

I conducted some tests, and I suspect I know what happens.  Selftests are
executed before the server is up, so it's not because the CA can't do
loopback (as ocsp requests are on non-tls), but rather being the fact that
the server is not even up to receive any requests.

We need to find out if there are ways to get around this.  Some ideas come to
mind:
 * find out if there is a way to "relocate" the selftest code to make it work
 and yet satisfy the requirements
 * maybe rewrite the selftest so that during startup, we check internally
 (CRL?) instead of going with the enableOCSP setting?

Comment 2 Christina Fu 2018-10-19 18:17:30 UTC

an update from discussion with Gossamer this morning:

They believe that we could redefine selftests so that they don't have to do revocation checking.
We could special-case CA/OCSP so they do just expiration checks and/or some other  easier-to-check "vital signs" of a CA or OCSP.

This should minimize the work required for us.

Comment 3 Jack Magne 2018-12-07 01:11:42 UTC

The checking for this has been in master:

commit c7b8771119f2e055bdca322afd7986cb653ccbd3
Author: jmagne <jmagne>
Date:   Thu Nov 8 17:07:40 2018 -0800

    Resolve: Bug 1641119 - CC: CA/OCSP startup fail on SystemCertsVerification if enableOCSP is true. (#87)
    
    The approach taken by this patch is quite simple. The SystemCertsVerification self test has been modified to
    optionally act differently when verifying the system certs of both ca and ocsp instances.
    
    Previously, the test would do a full cert verification , which results in an ocsp check being done at the nss level, if ocsp has been enabled in the server.xml. The past result was to have the server hang on startup , due to the fact that an ocsp check of a given cert would loop back to the ca or ocsp server itself to do the work. In the case of the self test /startup scenario, the server will not be sufficiently ready to field such a request, thus resulting in a hang situation.
    
    This fix modifies the cert checks for ca and ocsp to ONLY do a validity test for each cert.
    
    The code has created an optional parameter than can force our of this behaviour if the admin absolutely wants to:
    
    selftests.plugin.SystemCertsVerification.FullCAandOCSPVerify= true
    
    IF, the admin wants the test to behave as it did before. This may be the case where we know ocsp is not configured for the ca or ocsp itself.
    
    The value, is false by default and is false if the line is not present.
    
    The simple validity test is all that gets done at this point but could be modified to do more in the future.
    We already have a validity test for just the CA singing and OCSP signing certs. I felt it was cleaner to just leave those in place unchanged, safely leaving the original wiring in place.

Comment 4 Jack Magne 2018-12-07 23:11:35 UTC

Branch checkin:

commit 3eab287365d83a167fff7ec1287bd70647e93757 (HEAD -> DOGTAG_10_5_BRANCH, origin/DOGTAG_10_5_BRANCH)
Author: jmagne <jmagne>
Date:   Thu Nov 8 17:07:40 2018 -0800

    Resolve: Bug 1641119 - CC: CA/OCSP startup fail on SystemCertsVerification if enableOCSP is true. (#87)
    
    The approach taken by this patch is quite simple. The SystemCertsVerification self test has been modified to
    optionally act differently when verifying the system certs of both ca and ocsp instances.
    
    Previously, the test would do a full cert verification , which results in an ocsp check being done at the nss level, if ocsp has been enabled in the server.xml. The past result was to have the server hang on startup , due to the fact that an ocsp check of a given cert would loop back to the ca or ocsp server itself to do the work. In the case of the self test /startup scenario, the server will not be sufficiently ready to field such a request, thus resulting in a hang situation.
    
    This fix modifies the cert checks for ca and ocsp to ONLY do a validity test for each cert.
    
    The code has created an optional parameter than can force our of this behaviour if the admin absolutely wants to:
    
    selftests.plugin.SystemCertsVerification.FullCAandOCSPVerify= true
    
    IF, the admin wants the test to behave as it did before. This may be the case where we know ocsp is not configured for the ca or ocsp itself.
    
    The value, is false by default and is false if the line is not present.

Patch built and tested on f27 vm.

Comment 8 Amol K 2019-06-14 07:27:41 UTC

I tested this bugzilla on 10.5.16-2.el7

I tried the steps which are mentioned in comment #6. 

- Installation of standalone CA, KRA and OCSP, after restart observed logs.
- Setup selftests.plugin.SystemCertsVerification.FullCAandOCSPVerify=true in subsystem's CS.cfg file and restart the subsystem.
- I could see that there are different logs.

Which is working as expected.
Verifying this bugzilla.

Comment 11 errata-xmlrpc 2019-08-06 13:07:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2228

Note You need to log in before you can comment on or make changes to this bug.