Bug 1905613

Summary: mod_ssl does not like valid certificate chain
Product: Red Hat Enterprise Linux 8 Reporter: Joe Madden <joe.madden>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED ERRATA QA Contact: Branislav Náter <bnater>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.3CC: bnater, jorton, luhliari, vpolasek
Target Milestone: rcKeywords: Triaged
Target Release: 8.5   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: httpd-2.4-8050020210712211742.b4937e53 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 18:43:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Server Cert
none
Intermediate
none
root ca none

Description Joe Madden 2020-12-08 16:13:01 UTC
Created attachment 1737668 [details]
Server Cert

Created attachment 1737668 [details]
Server Cert

Description of problem:
Rebuilding a webserver from redhat 6 to redhat 8, we came across an issue where mod_ssl/httpd does not like a valid certificate chain and will refuse to start.

Version-Release number of selected component (if applicable):
httpd-tools-2.4.37-30.module+el8.3.0+7001+0766b9e7.x86_64
httpd-2.4.37-30.module+el8.3.0+7001+0766b9e7.x86_64
httpd-manual-2.4.37-30.module+el8.3.0+7001+0766b9e7.noarch
redhat-logos-httpd-81.1-1.el8.noarch
rphttpd-filesystem-2.4.37-30.module+el8.3.0+7001+0766b9e7.noarch
mhttpd-devel-2.4.37-30.module+el8.3.0+7001+0766b9e7.x86_64
mod_ssl-2.4.37-30.module+el8.3.0+7001+0766b9e7.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Install httpd
2. Configure SSL like so:
        # SSL config
        SSLEngine On
        SSLOptions +StrictRequire
        SSLVerifyClient none
        SSLProxyEngine Off
        SSLInsecureRenegotiation Off
        SSLHonorCipherOrder On
        SSLProtocol TLSv1.2
        SSLCipherSuite !ADH:!RC4-SHA:+HIGH:!MEDIUM:!LOW:!SSLv2:!EXPORT:ALL
        SSLCertificateFile /etc/httpd/certs/rccstatus.crt
        SSLCertificateKeyFile /etc/httpd/certs/rccstatus.key
        SSLCertificateChainFile /etc/httpd/certs/rfd-intermediate.crt
3. restart httpd - Fails,

Actual results:
error_log:
[Tue Dec 08 15:58:43.641806 2020] [ssl:emerg] [pid 4574:tid 139812245698880] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/httpd/rfd/ssl_error_log for more information
AH00016: Configuration Failed

virtual host error log:
[Tue Dec 08 15:58:43.641412 2020] [ssl:info] [pid 4574:tid 139812245698880] AH01914: Configuring server www.rccstatus.org.uk:443 for SSL protocol
[Tue Dec 08 15:58:43.641800 2020] [ssl:emerg] [pid 4574:tid 139812245698880] AH01903: Failed to configure CA certificate chain!


Expected results:
apache to restart normaly

Additional info:
I've checked the certificates with an online certitifcate chain checker, openssl and they come back as okay:

certs] openssl verify -CAfile <(cat rfd-ca.crt rfd-intermediate.crt) rccstatus.crt
rccstatus.crt: OK

Attached the certificates - I've tried adding the root ca into the chain, makes no difference. I have other chain certificates working as expected but httpd just doesn't appear to like this one for some reason.

If i comment out the SSLCertificateChainFile the service will start, but ovbisouly does not provide the intemediate to clients which is an issue.

Comment 1 Joe Madden 2020-12-08 16:13:22 UTC
Created attachment 1737669 [details]
Intermediate

Comment 2 Joe Madden 2020-12-08 16:13:43 UTC
Created attachment 1737670 [details]
root ca

Comment 3 Joe Madden 2020-12-08 16:32:23 UTC
Further update to this - I've checked other virtual hosts i had to put on and the follow intermediates fail:


        Issuer: C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
        Validity
            Not Before: Feb 20 10:00:00 2014 GMT
            Not After : Feb 20 10:00:00 2024 GMT
        Subject: C = BE, O = GlobalSign nv-sa, CN = GlobalSign Domain Validation CA - SHA256 - G2


         Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
        Validity
            Not Before: Nov 21 00:00:00 2018 GMT
            Not After : Nov 21 00:00:00 2028 GMT
        Subject: C = BE, O = GlobalSign nv-sa, CN = GlobalSign RSA OV SSL CA 2018
        Subject Public Key Info:

        Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
        Validity
            Not Before: Mar  8 12:00:00 2013 GMT
            Not After : Mar  8 12:00:00 2023 GMT
        Subject: C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA

I though maybe it was an issue with root ca, but i think this rules it out.


This one works fine - I can't for the life of me figure out why 

        Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
        Validity
            Not Before: Nov  6 12:23:52 2017 GMT
            Not After : Nov  6 12:23:52 2027 GMT
        Subject: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = Thawte RSA CA 2018

Comment 4 Joe Madden 2020-12-09 13:06:36 UTC
I've managed to fix this issue.

At build time we run this to apply the pci-dss profile:

oscap xccdf eval --remediate --fetch-remote-resources --profile pci-dss --report /root/oscap_usgcb_remediation_report.html /usr/share/xml/scap/ssg/content/ssg-rhel8-xccdf.xml

This performs the following remediation which adds a new include into /etc/pki/tls/openssl.cnf:

[ default_modules ]

ssl_conf = ssl_module

[ ssl_module ]

system_default = crypto_policy

[ crypto_policy ]

.include /etc/crypto-policies/back-ends/openssl.config


.include /etc/crypto-policies/back-ends/opensslcnf.config

This is the added line:

.include /etc/crypto-policies/back-ends/openssl.config

which links to :

/etc/crypto-policies/back-ends/openssl.config -> /usr/share/crypto-policies/DEFAULT/openssl.txt


The other include link links to:

/etc/crypto-policies/back-ends/opensslcnf.config -> /usr/share/crypto-policies/DEFAULT/opensslcnf.txt



The Broken addition:

 cat /usr/share/crypto-policies/DEFAULT/openssl.txt
@SECLEVEL=2:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8

the other file:

 cat /usr/share/crypto-policies/DEFAULT/opensslcnf.txt
CipherString = @SECLEVEL=2:kEECDH:kRSA:kEDH:kPSK:kDHEPSK:kECDHEPSK:-aDSS:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:-ARIA:-AESCCM8
Ciphersuites = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256
MinProtocol = TLSv1.2
MaxProtocol = TLSv1.3
SignatureAlgorithms = ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ed25519:ed448:rsa_pss_pss_sha256:rsa_pss_rsae_sha256:rsa_pss_pss_sha384:rsa_pss_rsae_sha384:rsa_pss_pss_sha512:rsa_pss_rsae_sha512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1


I'm guessing either openssl doesn't like the two includes or the broke addition breaks openssl, but in turn that breaks mod_ssl.

Not sure where to report this if i'm honest.

Thanks,

Comment 5 Joe Orton 2020-12-14 09:19:08 UTC
Adding OpenSSL and scap-security-guide owners, not immediately obvious how to triage this.

I'm assuming that the OpenSSL profile applied excludes the signature algorithm used by one of the CAs, but mod_ssl is not reporting enough information to be sure, which itself is a bug we can look at.  Not sure if that configuration is intentional or not, so maybe scap or OpenSSL folks can help?

Comment 6 Joe Madden 2020-12-14 09:22:04 UTC
As the end user, a log entry would have been useful from mod_ssl to say it was rejected because of reasons.

That said, i haven't looked at what the scap configuration has done yet and figured out what it didn't like about the certificate.

Comment 7 Joe Orton 2020-12-14 09:28:47 UTC
Yeah, agreed.  Could you file a support ticket for this please so we can investigate further?

Comment 8 Joe Madden 2020-12-14 09:46:51 UTC
I would but I can't unfortunately where on a self support tier :(

Happy to provide anything you need though if you get someone to contact me.

Thanks,

Comment 9 Joe Orton 2020-12-15 09:54:03 UTC
Fixed upstream: http://svn.apache.org/viewvc?view=revision&revision=1884452

Comment 10 Tomas Mraz 2020-12-15 10:14:46 UTC
@vpolasek The addition of .include /etc/crypto-policies/back-ends/openssl.config is totally wrong. The openssl.config is not supposed to be included, only the opensslcnf.config is.

The weak signature algorithms in certificates can be blocked by both the @SECLEVEL=2 or the signaturealgorithms setting.

Comment 11 Tomas Mraz 2020-12-15 10:19:53 UTC
However unless the broken include of the openssl.config somehow messes things up I do not see any reason why this certificate chain would not be acceptable for the settings in the opensslcnf.config.

Comment 12 Tomas Mraz 2020-12-15 10:23:01 UTC
Can you please try
openssl verify -auth_level 2 -CAfile <(cat rfd-ca.crt rfd-intermediate.crt) rccstatus.crt

Comment 13 Joe Orton 2020-12-15 10:24:43 UTC
There is also a possible spurious failure in this code path due to mod_ssl not clearing the OpenSSL error stack, if mod_md is also in use here.

https://bz.apache.org/bugzilla/show_bug.cgi?id=62880

Comment 14 Tomas Mraz 2020-12-15 11:23:31 UTC
Yeah, that is the most probable culprit - and the incorrect include triggers this.

Comment 16 Joe Madden 2020-12-16 10:59:03 UTC
I checked the certificates for you - They came back as okay on Auth 2

 openssl verify -auth_level 2 -CAfile <(cat rfd-ca.crt rfd-intermediate.crt) rccstatus.crt
rccstatus.crt: OK

Comment 17 Tomas Mraz 2020-12-16 12:48:32 UTC
So yeah, the broken include (which would be otherwise just ignored) triggers the issue from the comment #13.

Comment 37 errata-xmlrpc 2021-11-09 18:43:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: httpd:2.4 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4257