Bug 1771979

Summary: NSS no longer sends intermediate client certificates in TLS if the intermediate certificates are not explicitly trusted
Product: Red Hat Enterprise Linux 7 Reporter: Graham Leggett <minfrin>
Component: 389-ds-baseAssignee: nss-nspr-maint <nss-nspr-maint>
Status: CLOSED WONTFIX QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: high Docs Contact:
Priority: medium    
Version: 7.7CC: hkario, rrelyea, ssorce
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-17 07:31:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Graham Leggett 2019-11-13 10:28:46 UTC
Description of problem:

NSS no longer sends intermediate client certificates in a TLS handshake despite these being available if the intermediate certificates are not explicitly trusted in the certificate database.

As a result, previously working 389ds replication connections over TLS with client certificates fail where they worked previously.

Version-Release number of selected component (if applicable):

nss-3.44.0-4.el7.x86_64

How reproducible:


Steps to Reproduce:
1. Deploy a 389ds master/slave config with replication via TLS and client certs
2. Upgrade NSS

Actual results:

The master 389ds server no longer sends intermediate certs in SSL handshake.

Expected results:

The master 389ds server sends intermediate certs in SSL handshake.

Additional info:

Setting the trust settings on the intermediate certs to "trusted" works around the problem. It is no longer enough for the root certificate to be explicitly trusted.

Comment 2 Bob Relyea 2019-11-27 17:13:03 UTC
Could send send the sample cert chain thats failing (we just need the public certs, not the private key).

Thanks,
bob

Comment 3 Alicja Kario 2019-11-27 17:23:50 UTC
also, does the client trust the root CA that the client's certificate chains up to?

Comment 4 Graham Leggett 2020-01-02 12:41:35 UTC
Not able to send the certificate chain, as the chain is private. Also, the client does trust the root CA, that's how it all worked in the past.

What stopped working was intermediate certificates being offered by NSS. In the past, the intermediates needed to just be present and valid, and that was enough. Now, since a recent update, the intermediates need to be present, valid *and* explicitly marked trusted in the NSS database.

Comment 5 Alicja Kario 2020-01-02 15:17:08 UTC
(In reply to Graham Leggett from comment #4)
> Not able to send the certificate chain, as the chain is private.

could you then recreate a copy of it with no PII using the same tools? please also identify the fields that differ (except the subject, issuer, signature value and subject public key value)

> Also, the
> client does trust the root CA, that's how it all worked in the past.
> 
> What stopped working was intermediate certificates being offered by NSS. In
> the past, the intermediates needed to just be present and valid, and that
> was enough. Now, since a recent update, the intermediates need to be
> present, valid *and* explicitly marked trusted in the NSS database.

this is not behaviour we are able to reproduce with
nspr-4.21.0-1.el7.ppc64le
nspr-devel-4.21.0-1.el7.ppc64le
nss-3.44.0-7.el7_7.ppc64le
nss-debuginfo-3.44.0-7.el7_7.ppc64le
nss-devel-3.44.0-7.el7_7.ppc64le
nss-pem-1.0.3-7.el7.ppc64le
nss-pkcs11-devel-3.44.0-7.el7_7.ppc64le
nss-softokn-3.44.0-8.el7_7.ppc64le
nss-softokn-debuginfo-3.44.0-8.el7_7.ppc64le
nss-softokn-devel-3.44.0-8.el7_7.ppc64le
nss-softokn-freebl-3.44.0-8.el7_7.ppc64le
nss-softokn-freebl-devel-3.44.0-8.el7_7.ppc64le
nss-sysinit-3.44.0-7.el7_7.ppc64le
nss-tools-3.44.0-7.el7_7.ppc64le
nss-util-3.44.0-4.el7_7.ppc64le
nss-util-debuginfo-3.44.0-4.el7_7.ppc64le
nss-util-devel-3.44.0-4.el7_7.ppc64le

full reproducer that first creates the described PKI and then uses them for TLS connection:

cd /tmp
wget https://raw.githubusercontent.com/redhat-qe-security/certgen/master/certgen/lib.sh
source lib.sh
x509KeyGen ca &&
x509KeyGen client-ca &&
x509KeyGen server-ca &&
x509KeyGen client &&
x509KeyGen server &&
x509SelfSign ca &&
x509CertSign -t CA --CA ca client-ca &&
x509CertSign -t CA --CA ca --DN 'O=Other intermediate CA' server-ca &&
x509CertSign --CA server-ca server &&
x509CertSign --CA client-ca -t webclient client

mkdir server-db
certutil -N -d sql:server-db --empty-password
certutil -A -d sql:server-db -i $(x509Cert ca) -n ca -t 'cTC,cTC,' -a
certutil -A -d sql:server-db -i $(x509Cert server-ca) -n server-ca -t ',,' -a
pk12util -d sql:server-db -i $(x509Key --pkcs12 --with-cert server) -W ''

mkdir client-db
certutil -N -d sql:client-db --empty-password
certutil -A -d sql:client-db -i $(x509Cert ca) -n ca -t 'cTC,cTC,' -a
certutil -A -d sql:client-db -i $(x509Cert client-ca) -n client-ca -t ',,' -a
pk12util -d sql:client-db -i $(x509Key --pkcs12 --with-cert client) -W ''


In one terminal:
/usr/lib64/nss/unsupported-tools/selfserv -d sql:server-db/ -n server -rr -p 4433 -v


In other terminal:
/usr/lib64/nss/unsupported-tools/tstclnt -d sql:client-db/ -h localhost -p 4433 <<<"GET / HTTP/1.0
"


I'm getting the following from server:

selfserv: About to call accept.



selfserv: About to call accept.
selfserv: Subject: CN=John Smith
selfserv: Issuer : O=Example intermediate CA
selfserv: -- SSL3: Certificate Validated.
selfserv: 0 cache hits; 1 cache misses, 0 cache not reusable
          0 stateless resumes, 0 ticket parse failures
selfserv: SSL version 3.4 using 128-bit AES-GCM with 128-bit AEAD MAC
selfserv: Server Auth: 2048-bit TLS 1.3, Key Exchange: 255-bit TLS 1.3
          Compression: NULL, Extended Master Secret: Yes
selfserv: subject DN: CN=John Smith
selfserv: issuer  DN: O=Example intermediate CA


And the following from client:

subject DN: CN=localhost
issuer  DN: O=Other intermediate CA
0 cache hits; 0 cache misses, 0 cache not reusable
0 stateless resumes
Received 0 Cert Status items (OCSP stapled data)
HTTP/1.0 200 OK
Server: Generic Web Server
Date: Tue, 26 Aug 1997 22:10:05 GMT
Content-type: text/plain

GET / HTTP/1.0

EOF


So the intermediates are sent by both the client and server, and trusted by the other side.

Comment 6 Alicja Kario 2020-03-23 11:48:24 UTC
Without additional information or a reproducer we are unable to diagnose and fix this issue.

If you encounter similar problem, please open a new bug and provide the necessary steps to reproduce it.

Comment 7 Graham Leggett 2020-11-17 23:05:51 UTC
Just rammed into this problem again, and following a trail of reverse engineering code from sssd to openldap I think I've got to the bottom of the problem.

In RHEL7.5, the crypto library behind openldap was changed from NSS to openssl, and testing did not pick up a behaviour change.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/openldap

"The OpenLDAP suite in Red Hat Enterprise Linux 7.5 and later no longer uses Mozilla implementation of Network Security Services (NSS). Instead, it uses the OpenSSL."

In the case of the NSS code, intermediate certificates stored on the client side will be filled in as usual where needed. In the openssl code the behaviour is different as the SSL_CTX_use_certificate_file() call is used instead of the SSL_CTX_use_certificate_chain_file() call that is recommended by openssl.

https://github.com/openldap/openldap/blame/b06f5b0493937fc28f2cc86df1d7f464aa4504d8/libraries/libldap/tls_o.c#L387

The SSL_CTX_use_certificate_file() call loads in the first certificate only, and ignores all subsequent certificates in the chain.

Counter-intuitively openldap requires you to specify intermediate certificates as trusted root certificates, and not treated as intermediates like the SSL_CTX_use_certificate_chain_file() does.

I suspect what is happening here is that without being explicitly trusted, the intermediate certs aren't exported into the hack that 389ds uses to marry the NSS-based server to the openssl-based client. With the intermediates not being exported, SSL connections are no longer trusted, and they break.

While 389ds tries to use multiple SSL libraries at the same time, there will just be chaos.

This bug looks like a 389ds bug, not an NSS one.

Comment 9 Graham Leggett 2020-11-18 10:24:43 UTC
Related issue in openldap server: https://bugzilla.redhat.com/show_bug.cgi?id=1578438

Issue at CentOS: https://forums.centos.org/viewtopic.php?t=67042

Comment 12 RHEL Program Management 2021-05-17 07:31:06 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.