Description of problem: I'm having trouble viewing source.redhat.com in Epiphany. There are TLS certificate validation failures for igloo-prod.azureedge.net when loading the homepage. The problem is exposed because I made some changes in glib-networking 2.70 to enable GnuTLS's support for validating stapled OCSP responses. I asked Simo about this and he thinks the OCSP check requires SHA-1. I'm not sure what to do about this. Version-Release number of selected component (if applicable): crypto-policies-20210819-1.gitd0fdcfb.fc35, gnutls-3.7.2-2.fc35 How reproducible: Always Steps to Reproduce: 1. $ gnutls-cli igloo-prod.azureedge.net Actual results: After running 'sudo update-crypto-policies --set DEFAULT', I see: - Status: The certificate is NOT trusted. The received OCSP status response is invalid. *** PKI verification of server certificate failed... *** Fatal error: Error in the certificate. Expected results: If I run 'sudo update-crypto-policies --set LEGACY', then I get a successful connection and can use source.redhat.com normally. Additional info: I missed this until today because I usually use Epiphany Technology Preview, which doesn't use Fedora crypto policies because we never upstreamed them.
This is the same issue we've seen with .net utilities choking on a microsoft domain when using wget. It seem like GnuTLS is either the only library enforcing OCSP checks always or is the only one that blocks use of SHA-1 in OCSP, somethign that apparently is still widespread. We may need to consider a policy exception to block SHA-1 in cert signatures but allow it in OCSP :-/
(In reply to Simo Sorce from comment #2) > It seem like GnuTLS is either the only library enforcing OCSP checks always > or is the only one that blocks use of SHA-1 in OCSP, somethign that > apparently is still widespread. Behavior varies based on how you use GnuTLS. Since glib-networking 2.70, GNOME now tries to use gnutls_certificate_verify_peers3() whenever possible. This function verifies only stapled OCSP responses, which I (possibly incorrectly) assumed were rarely used. If not possible to use gnutls_certificate_verify_peers3(), then we fall back to gnutls_x509_crt_check_hostname(), which does no revocation checks. (This is what we always did prior to glib-networking 2.70.) So to be clear, glib-networking does not communicate with an actual OCSP server: the failure happens when processing a stapled response. (I *suspect* gnutls-cli does exactly the same, but did not check.) > We may need to consider a policy exception to block SHA-1 in cert signatures > but allow it in OCSP :-/ Maybe indeed.
(In reply to Michael Catanzaro from comment #3) > If not possible to use gnutls_certificate_verify_peers3(), then we fall back to gnutls_x509_crt_check_hostname() Oops, I meant we fall back to gnutls_x509_trust_list_verify_crt().
Hi, any thoughts on this? If not fixed, I will have to force GNOME to set GnuTLS priority string manually to bypass the Fedora crypto policy, and that's not cool. Will be doing a glib-net release very soon.
Daiki, Alex, should we consider allowing SHA-1 Signature verification for OCSP for now ? It seem like we are getting real life issues, and as much as I'd like to get those servers fixed, it does no good to have people switch to LEGACY policy for everything when only OCSP is affected.
I lack the expertise to answer how much does it compromise the overall security. If we really need a knob for that, I can plumb one, though I'm not a fan of relaxing the policy mid-release. "Force GNOME to set GnuTLS priority string manually to bypass the Fedora crypto policy" is both 1) drastic, because you can append to the priority string and 2) likely not going to work because of https://fedoraproject.org/wiki/Changes/GnutlsAllowlisting
From the internal GnuTLS implementation point of view, there are currently 3 security levels regarding signatures: _SECURE (for everything else), _INSECURE_FOR_CERTS (for everything except certs), and _INSECURE (nothing). Perhaps we could add a middle ground, between the first two, say _INSECURE_FOR_TLS and change the meaning of _SECURE to cover the PKIX use-cases?
Honestly I think marking SHA-1 as _INSECURE_FOR_CERTS is good enough, because that is what all web browsers do on every platform other than RHEL/Fedora, and matching other platforms reduces risk for incompatibility. But introducing a new security level seems fine to me....
That'd be silently mass-relaxing the "Disable SHA1 support for use in signatures" change announced in https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 requirement, so, no. Another thing we might need to undertake is to take a closer look at how gnutls handles OCSP to make sure that signature hash restrictions doesn't make it rejecting a connection that would otherwise be accepted if the server just skipped sending the OCSP staple. All in all, I'm afraid this isn't something that can be resolved swiftly given how multiple components, several public interfaces and OS-global defaults are involved, so you might consider rolling the Epiphany-side change back meanwhile, especially if we're still talking F35 =(
(In reply to Alexander Sosedkin from comment #10) > All in all, I'm afraid this isn't something that can be resolved swiftly > given how multiple components, several public interfaces and OS-global > defaults are involved, so you might consider rolling the Epiphany-side > change back meanwhile, especially if we're still talking F35 =( To be clear: there's nothing to change in Epiphany. I have to change glib-networking to stop using gnutls_certificate_verify_peers3() and go back to gnutls_x509_trust_list_verify_crt(). This will disable certificate revocation checks, key usage checks, basic constraints and name constraints checks for all desktop components, not just Epiphany. But I agree that's a reasonable suggestion, because the glib-networking change to use gnutls_certificate_verify_peers3() was the change that exposed this issue, so it makes sense to revert it for now. But I'll do that only in a downstream Fedora patch, because this is a Fedora-specific problem and the change is good for upstream.
(In reply to Alexander Sosedkin from comment #10) > Another thing we might need to undertake is to take a closer look at how > gnutls handles OCSP to make sure that signature hash restrictions doesn't > make it rejecting a connection that would otherwise be accepted if the > server just skipped sending the OCSP staple. That'd be great, as it would certainly fix this issue.
This sounds like a similar problem we had with NSS. When validating pkix changes, you need to include policy in the chain selection code. We would find a valid chain with an old SHA-1 signed intermediate and, then try to validate that chain. Once we rejected the SHA-1 intermediate we could properly process the OCSP request. I wonder if a similiar thing isn't happening here (we all thought the OCSP request was SHA-1 signed when it wasn't).
I doubt that's happening in this case, because source.redhat.com uses a very simple certificate chain with no complex path building required. We've had similar mistakes in both glib-networking and GnuTLS in the past, though. glib-networking isn't in the business of certificate path building anymore, and I've reported https://gitlab.com/gnutls/gnutls/-/issues/1286 to suggest improvements there (though I imagine it would be a tough project).
(In reply to Michael Catanzaro from comment #11) > But I agree that's a reasonable suggestion, because the glib-networking > change to use gnutls_certificate_verify_peers3() was the change that exposed > this issue, so it makes sense to revert it for now. But I'll do that only in > a downstream Fedora patch, because this is a Fedora-specific problem and the > change is good for upstream. I've added a patch sabotage-certificate-revocation.patch to Fedora's glib-networking package, and am releasing a workaround update now (glib-networking-2.70.1-1.fc35). Will revert that sabotage once we have a GnuTLS-level solution for this.
FEDORA-2021-5c39828ca6 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-5c39828ca6
FEDORA-2021-5c39828ca6 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-5c39828ca6` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-5c39828ca6 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
When I configured bodhi to not close this bug when the update is pushed, I would have expected it to not mess with the bug status at all....
Two possible solutions: (a) Just allow SHA-1 for OCSP responses. This seems best to me. The web PKI ecosystem doesn't seem too concerned with improving OCSP, and moving faster than the web seems unwise, so it might be a while before it's possible to be stricter here. (b) Don't allow SHA-1, but also do not fail certificate validation if the response signature is rejected, just ignore it to allow the connection to complete. In this case, the check for SHA-1 actually *reduces* security because it would cause a valid response that revokes the certificate to be ignored. If banning SHA-1 *reduces* security relative to allowing SHA-1, then we've probably done something wrong, so I'm skeptical of this approach.
> I doubt that's happening in this case, because source.redhat.com uses a very simple certificate chain with no complex path building required. It's not the source.redhat.com chain that's at issue, it's the chain the the OCSP server that is signing the OCSP response. Currently NSS rejects any SHA-1 *signature* in the OCSP chain but firefox can get to source.redhat.com. NSS isn't rejecting SHA-1 used to identify the revoked cert (which is what everyone thought the problem was). You really need to actually debug the gnutls chain live to determine where it is actually failing before we can really understand how to counteract it.
FEDORA-2021-5c39828ca6 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
(In reply to Bob Relyea from comment #20) > > I doubt that's happening in this case, because source.redhat.com uses a very simple certificate chain with no complex path building required. > > It's not the source.redhat.com chain that's at issue, it's the chain the the > OCSP server that is signing the OCSP response. Currently NSS rejects any > SHA-1 *signature* in the OCSP chain but firefox can get to > source.redhat.com. NSS isn't rejecting SHA-1 used to identify the revoked > cert (which is what everyone thought the problem was). You really need to > actually debug the gnutls chain live to determine where it is actually > failing before we can really understand how to counteract it. We have dug into that before in bug 2003363 and the reason why Firefox can get to source.redhat.com is, I suppose, the side-effect of SHA-1 re-enablement for Firefox add-ons: https://src.fedoraproject.org/rpms/firefox/blob/rawhide/f/firefox-nss-addon-hack.patch Note that moz::pkix uses VFY_VerifyDigestDirect for verifying OCSP response signatures.
(In reply to Daiki Ueno from comment #22) > We have dug into that before in bug 2003363 Um, I'm almost certain this is the exact same issue, so I'm just going to go ahead and mark it as a duplicate. Feel free to undo that if you think I'm wrong about that for some reason. *** This bug has been marked as a duplicate of bug 2003363 ***
(In reply to Michael Catanzaro from comment #19) > Two possible solutions: > > (a) Just allow SHA-1 for OCSP responses. This seems best to me. The web PKI > ecosystem doesn't seem too concerned with improving OCSP, and moving faster > than the web seems unwise, so it might be a while before it's possible to be > stricter here. > > (b) Don't allow SHA-1, but also do not fail certificate validation if the > response signature is rejected, just ignore it to allow the connection to > complete. In this case, the check for SHA-1 actually *reduces* security > because it would cause a valid response that revokes the certificate to be > ignored. If banning SHA-1 *reduces* security relative to allowing SHA-1, > then we've probably done something wrong, so I'm skeptical of this approach. How about a third possibility: the certificate is re-issued in accordance with the CA/Browser Forum Baseline Requirements? The CA/BF deprecated SHA-1 in 2015 or so. Browsers like Chrome[1,2] and Firefox[3] implemented the policy in 2017 or so. Why is a SHA-1 certificate in the wild in 2021? [1] https://security.googleblog.com/2014/09/gradually-sunsetting-sha-1.html [2] https://security.googleblog.com/2015/12/an-update-on-sha-1-certificates-in.html [3] https://blog.mozilla.org/security/2017/02/23/the-end-of-sha-1-on-the-public-web/
This is about OCSP. We've already established this problem is specific to GnuTLS on Fedora and RHEL. No other software rejects it.
(In reply to Michael Catanzaro from comment #25) > This is about OCSP. > > We've already established this problem is specific to GnuTLS on Fedora and > RHEL. No other software rejects it. My bad, you are right. According to RFC 6277, Section 3, Updates to Mandatory and Optional Cryptographic Algorithms: OLD: Clients that request OCSP services SHALL be capable of processing responses signed used DSA keys identified by the DSA sig-alg-oid specified in section 7.2.2 of [RFC2459]. Clients SHOULD also be capable of processing RSA signatures as specified in section 7.2.1 of [RFC2459]. OCSP responders SHALL support the SHA1 hashing algorithm. NEW: Clients that request OCSP services SHALL be capable of processing responses signed using RSA with SHA-1 (identified by sha1WithRSAEncryption OID specified in [RFC3279]) and RSA with SHA-256 (identified by sha256WithRSAEncryption OID specified in [RFC4055]). Clients SHOULD also be capable of processing responses signed using DSA keys (identified by the id-dsa-with- sha1 OID specified in [RFC3279]). Clients MAY support other algorithms.