Could send send the sample cert chain thats failing (we just need the public certs, not the private key).

Thanks,
bob

also, does the client trust the root CA that the client's certificate chains up to?

Not able to send the certificate chain, as the chain is private. Also, the client does trust the root CA, that's how it all worked in the past.

What stopped working was intermediate certificates being offered by NSS. In the past, the intermediates needed to just be present and valid, and that was enough. Now, since a recent update, the intermediates need to be present, valid *and* explicitly marked trusted in the NSS database.

(In reply to Graham Leggett from comment #4)
> Not able to send the certificate chain, as the chain is private.

could you then recreate a copy of it with no PII using the same tools? please also identify the fields that differ (except the subject, issuer, signature value and subject public key value)

> Also, the
> client does trust the root CA, that's how it all worked in the past.
> 
> What stopped working was intermediate certificates being offered by NSS. In
> the past, the intermediates needed to just be present and valid, and that
> was enough. Now, since a recent update, the intermediates need to be
> present, valid *and* explicitly marked trusted in the NSS database.

this is not behaviour we are able to reproduce with
nspr-4.21.0-1.el7.ppc64le
nspr-devel-4.21.0-1.el7.ppc64le
nss-3.44.0-7.el7_7.ppc64le
nss-debuginfo-3.44.0-7.el7_7.ppc64le
nss-devel-3.44.0-7.el7_7.ppc64le
nss-pem-1.0.3-7.el7.ppc64le
nss-pkcs11-devel-3.44.0-7.el7_7.ppc64le
nss-softokn-3.44.0-8.el7_7.ppc64le
nss-softokn-debuginfo-3.44.0-8.el7_7.ppc64le
nss-softokn-devel-3.44.0-8.el7_7.ppc64le
nss-softokn-freebl-3.44.0-8.el7_7.ppc64le
nss-softokn-freebl-devel-3.44.0-8.el7_7.ppc64le
nss-sysinit-3.44.0-7.el7_7.ppc64le
nss-tools-3.44.0-7.el7_7.ppc64le
nss-util-3.44.0-4.el7_7.ppc64le
nss-util-debuginfo-3.44.0-4.el7_7.ppc64le
nss-util-devel-3.44.0-4.el7_7.ppc64le

full reproducer that first creates the described PKI and then uses them for TLS connection:

cd /tmp
wget https://raw.githubusercontent.com/redhat-qe-security/certgen/master/certgen/lib.sh
source lib.sh
x509KeyGen ca &&
x509KeyGen client-ca &&
x509KeyGen server-ca &&
x509KeyGen client &&
x509KeyGen server &&
x509SelfSign ca &&
x509CertSign -t CA --CA ca client-ca &&
x509CertSign -t CA --CA ca --DN 'O=Other intermediate CA' server-ca &&
x509CertSign --CA server-ca server &&
x509CertSign --CA client-ca -t webclient client

mkdir server-db
certutil -N -d sql:server-db --empty-password
certutil -A -d sql:server-db -i $(x509Cert ca) -n ca -t 'cTC,cTC,' -a
certutil -A -d sql:server-db -i $(x509Cert server-ca) -n server-ca -t ',,' -a
pk12util -d sql:server-db -i $(x509Key --pkcs12 --with-cert server) -W ''

mkdir client-db
certutil -N -d sql:client-db --empty-password
certutil -A -d sql:client-db -i $(x509Cert ca) -n ca -t 'cTC,cTC,' -a
certutil -A -d sql:client-db -i $(x509Cert client-ca) -n client-ca -t ',,' -a
pk12util -d sql:client-db -i $(x509Key --pkcs12 --with-cert client) -W ''


In one terminal:
/usr/lib64/nss/unsupported-tools/selfserv -d sql:server-db/ -n server -rr -p 4433 -v


In other terminal:
/usr/lib64/nss/unsupported-tools/tstclnt -d sql:client-db/ -h localhost -p 4433 <<<"GET / HTTP/1.0
"


I'm getting the following from server:

selfserv: About to call accept.



selfserv: About to call accept.
selfserv: Subject: CN=John Smith
selfserv: Issuer : O=Example intermediate CA
selfserv: -- SSL3: Certificate Validated.
selfserv: 0 cache hits; 1 cache misses, 0 cache not reusable
          0 stateless resumes, 0 ticket parse failures
selfserv: SSL version 3.4 using 128-bit AES-GCM with 128-bit AEAD MAC
selfserv: Server Auth: 2048-bit TLS 1.3, Key Exchange: 255-bit TLS 1.3
          Compression: NULL, Extended Master Secret: Yes
selfserv: subject DN: CN=John Smith
selfserv: issuer  DN: O=Example intermediate CA


And the following from client:

subject DN: CN=localhost
issuer  DN: O=Other intermediate CA
0 cache hits; 0 cache misses, 0 cache not reusable
0 stateless resumes
Received 0 Cert Status items (OCSP stapled data)
HTTP/1.0 200 OK
Server: Generic Web Server
Date: Tue, 26 Aug 1997 22:10:05 GMT
Content-type: text/plain

GET / HTTP/1.0

EOF


So the intermediates are sent by both the client and server, and trusted by the other side.

Without additional information or a reproducer we are unable to diagnose and fix this issue.

If you encounter similar problem, please open a new bug and provide the necessary steps to reproduce it.

Just rammed into this problem again, and following a trail of reverse engineering code from sssd to openldap I think I've got to the bottom of the problem.

In RHEL7.5, the crypto library behind openldap was changed from NSS to openssl, and testing did not pick up a behaviour change.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/openldap

"The OpenLDAP suite in Red Hat Enterprise Linux 7.5 and later no longer uses Mozilla implementation of Network Security Services (NSS). Instead, it uses the OpenSSL."

In the case of the NSS code, intermediate certificates stored on the client side will be filled in as usual where needed. In the openssl code the behaviour is different as the SSL_CTX_use_certificate_file() call is used instead of the SSL_CTX_use_certificate_chain_file() call that is recommended by openssl.

https://github.com/openldap/openldap/blame/b06f5b0493937fc28f2cc86df1d7f464aa4504d8/libraries/libldap/tls_o.c#L387

The SSL_CTX_use_certificate_file() call loads in the first certificate only, and ignores all subsequent certificates in the chain.

Counter-intuitively openldap requires you to specify intermediate certificates as trusted root certificates, and not treated as intermediates like the SSL_CTX_use_certificate_chain_file() does.

I suspect what is happening here is that without being explicitly trusted, the intermediate certs aren't exported into the hack that 389ds uses to marry the NSS-based server to the openssl-based client. With the intermediates not being exported, SSL connections are no longer trusted, and they break.

While 389ds tries to use multiple SSL libraries at the same time, there will just be chaos.

This bug looks like a 389ds bug, not an NSS one.

Related issue in openldap server: https://bugzilla.redhat.com/show_bug.cgi?id=1578438

Issue at CentOS: https://forums.centos.org/viewtopic.php?t=67042

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.