RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1578438 - openldap TLS is not sending the intermediate cert after update
Summary: openldap TLS is not sending the intermediate cert after update
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openldap
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Matus Honek
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-15 15:23 UTC by mcguppy
Modified: 2021-12-10 16:09 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-17 13:34:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
CentOS 14805 0 None None None 2018-05-15 15:23:17 UTC

Internal Links: 1898924

Description mcguppy 2018-05-15 15:23:17 UTC
Description of problem: During OS update, openldap was updated from 2.4.44-5 to 2.4.44-13. Since then openLDAP server is not sending out any more the intermediate certificate


Version-Release number of selected component (if applicable): openldap.x86_64 2.4.44-5.el7


How reproducible: always


Steps to Reproduce:
1. install openldap in version 2.4.44-5
2. create a server SSL certificate signed by a intermediate certificate
3. bundle the server and the intermediate certificate together with the key into a PKCS12 cert: openssl pkcs12 -export -out server-fqdn.intermediate-chain.crt.bundle.pkcs12 -inkey server-fqdn.key.pem -in server-fqdn.intermediate-chain.crt.bundle.pem -certfile root-ca.crt.pem
4. import the PKCS12 cert into the NSS DB in /etc/openldap/certs: pk12util -d /etc/openldap/certs -i server-fqdn.intermediate-chain.crt.bundle.pkcs12
5. import the CA cert into /etc/pki/ca-trust/source/anchors/ and recreate the ca-trust by: update-ca-trust extract
6. configure openldap to have the following TLS attributes: olcTLSCACertificatePath: /etc/openldap/certs olcTLSCertificateFile: "The nick name of the server cert" olcTLSCertificateKeyFile: /etc/openldap/certs/password 
7. enable ldaps by adding "ldaps:///" in the file /etc/sysconfig/slapd to the argument SLAPD_URLS
8. restart openldap by systemctl restart slapd
9. check ldapsearch by using ldaps: ldapsearch -D <user DN of Manager> -W -b "<base-DN>" cn=<searching-object> -H ldaps://<server-fqdn> --> after typing in the password, the searched object is presented (this search is also possible from any LDAP client, that trust in the CA cert)
10. check SSL by: openssl s_client -connect <server-fqdn>:636 -showcerts --> the whole SSL cert chain (server-cert, intermediate cert, root CA cert) is presented with a result of "Verify return code: 0 (ok)"
11. update openldap from 2.4.44-5 to the newest version in CentOS 7 (2.4.44-13)
12. check ldapsearch by using ldaps: ldapsearch -D <user DN of Manager> -W -b "<base-DN>" cn=<searching-object> -H ldaps://<server-fqdn> --> after typing in the password you get the error: "ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)"
13. check SSL by: openssl s_client -connect <server-fqdn>:636 -showcerts --> only the server cert is presented (the intermediate cert is missing) --> Error message: Verify return code: 21 (unable to verify the first certificate)


Actual results: openssl s_client -connect <server-fqdn>:636 -showcerts --> only the server cert is presented (the intermediate cert is missing) --> Error message: Verify return code: 21 (unable to verify the first certificate)


Expected results: openssl s_client -connect <server-fqdn>:636 -showcerts --> the whole SSL cert chain (server-cert, intermediate cert, root CA cert) is presented with a result of "Verify return code: 0 (ok)"


Additional info: This problem has been seen on CentOS 7.5 and reported to CentOS bug tracking system (see Add External Bug). On advice of the CentOS Forum  https://www.centos.org/forums/viewtopic.php?f=48&t=67042 I also have to report it here.

Comment 2 mcguppy 2018-05-16 09:03:49 UTC
I found two solutions for this issue:

-----

Solution 1 (with NSS):
Modify the trustargs of the intermediate certificate in the NSS cert store of the openldap server:

certutil -d /etc/openldap/certs -M -t "CT,," -n "Intermediate cert  nick name"

This adds the to trust args:
C - Trusted CA (implies c)
T - trusted CA for client authentication (ssl server only)

------

Solution 2 (with OS CA trust):
copy the intermediate certificate on the openldap server into the directory
/etc/pki/ca-trust/source/anchors/  and perform a

update-ca-trust extract

------

After restarting openldap, intermediate certificate is delivered again from the server for both solutions.

Somehow it sound logic to have this trustargs set on the intermediate certificate, but I am still wondering, why it was working for the last two years without them?
So I am not sure, if it is really a bug or just a miss configuration on my side, which was working for a long time but now it isn't any more.

Comment 3 Matus Honek 2018-05-16 13:28:39 UTC
Foremost, please, take into account that now when compiled with OpenSSL in RHEL-7.5, if you have configured at least one of CACertFile or CACertDir, then the system-wide ca-trust is not used. You can fix this by pointing to it directly by either of these options. This is the default behaviour when compiled with OpenSSL and we'd like to keep as close to upstream as possible.

TBH, I could not reproduce. I'm a bit confused what the actual state of your configuration is on both sides (client and server). You can have a look into /tmp/openldap-tlsmc-*/cacerts folder (the one that belongs to your setup, README files inside would give you a hint) for what actually was extracted.

OpenSSL utils work only upon PEM files from system-wide store and those explicitly passed to them. OpenLDAP works additionally upon NSS DB from which it takes only the certs marked with at least C or T flags, and only those it uses further on for CA purposes.

Ad Solution 1: Were the flags C and T previously unset for the intermediate cert? If so, that is the issue as per the paragraph just above.

Ad Solution 2: This would work for openssl s_client but not for ldapsearch set up with an explicit TLS_CACERT or TLS_CACERTDIR. Is that the case?

Comment 4 mcguppy 2018-05-16 19:40:23 UTC
My setup is as follows:
On all my systems (server and client), the ROOT CA is put to /etc/pki/ca-trust/source/anchors/ and then extracted by 'update-ca-trust extract'. 

On my ldap clients I always use the CA file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

On the openldap server side I have to deliver the server certificate and the intermediate certificate. These two certs together with the key of the server cert, I bundeled into a PKCS12 format and then I imported this into the NSS store at /etc/openldap/certs
With this import, the C and T flags were not set on the intermediate certificate (no flags at all on it). On the server cert I had the flags "u,u,u".

My slapd.d config looks like this:

dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/openldap/slapd.args
olcPidFile: /var/run/openldap/slapd.pid
olcTLSCACertificatePath: /etc/openldap/certs
olcTLSCertificateKeyFile: /etc/openldap/certs/password
structuralObjectClass: olcGlobal
entryUUID: 091e7b9a-c539-1037-9b51-07aa765fcaee
creatorsName: cn=config
createTimestamp: 20180326120035Z
olcSizeLimit: -1
olcTLSCertificateFile: "dev-ad-user-sync-srv.ffhs.local - FFHS LOCAL"

This setup has worked for a long time without the trustargs set on the intermediate cert until the update to CentOS 7.5

But I understand, this is not a bug. It is a configuration error.

My target is to get rid of this NSS and switch to OpenSSL.
Not quite sure, how to get to this point with very low downtime.
My openLDAP Provider node was not autoupdated and like this this is the only node that is working currently. The openLDAP consumer nodes are all on CentOS 7.5 on which ldaps is not working.

I am sorry to think this could be ab bug.
Thank you very much for your help.
Kind regards, Stefan

Comment 5 Matus Honek 2018-05-17 11:47:36 UTC
I see. We had to pick what would be trusted. In this specific case the intermediate cert might have had the flags unset intentionally in order to not trust it. We chose the safer for better.

Well, in RHEL-7 the NSS DBs should be supported till the end of life, so you should be fine to keep using this setup. However the PEM files configuration is preferred now for (at least) it makes things less confusing.

When trying to change the setup to PEM files, it should be as easy as changing paths for the olcTLS* attributes to the according directories. Keep in mind that if NSS DB is present in the olcTLSCACertificatePath directory then it will be used solely, no additional files will be loaded from that directory.

The way I would go is to make the system-wide ca-trust contain all the CA certs I need, get the user cert and private key into a PEM file, then just dropping olcTLSCACertificatePath (so that system-wide ca-trust is used) and pointing olcCertificateFile/KeyFile to appropriate locations. I believe no restart should be required in this case, the only downtime would probably be due to internal reinitialization of slapd, thus active connections might be closed. While modifying your TLS configuration, please take into account this potential issue which would exhibit itself should you set up an incorrect configuration: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.5_release_notes/#BZ1524193

Hope this helps.
Matus

Comment 6 mcguppy 2018-05-17 12:38:12 UTC
Hi Matus

Thank you very much for your help.
I was able to switch openLDAP from NSS to openssl this morning on a testsystem.
My config looks like this right now:

# config
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /var/run/openldap/slapd.args
olcLogLevel: stats
olcPidFile: /var/run/openldap/slapd.pid
olcSizeLimit: -1
olcTLSCACertificateFile: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
olcTLSCertificateFile: /etc/pki/tls/certs/dev-ad-user-sync-srv.ffhs.local.crt.
 pem
olcTLSCertificateKeyFile: /etc/pki/tls/private/dev-ad-user-sync-srv.ffhs.local
 .key.pem

If I correctly understand you, the argument olcTLSCACertificateFile can be deleted and then the system wide ca-trust is used. I will try this.

I have to say, it wasn't that easy to change this, as openLDAP very often complained "server is not willing to do.....".
One have to do the steps in a correct order and I also had to stop it once and start it with only ldapi./// only to set the olcCertificateFile after I deleted it first (I was not able to just modify it).

One mistake I run into was the file permission on the key file as it was only readable by root. I set an ACL on it to grand read access fro user 'ldap'. 

It would be very helpful if openLDAP would be a bit more precise in the error messages, when you try something that is not possible. 
I got many times just this error: "ldap_modify: Other (e.g., implementation specific) error (80)".

Another question: What do you advise to do with the NSS DB in /etc/openldap/certs after all is switched to openssl? Can I just delete the files in it or even the whole directory? Or could it be one day, that during an update, this will be recreated and will end up in some strange side effects?

What I was really surprised this morning when setting up an completely new CentOS 7.5 and then install an openldap server, still NSS was configured. I thought this will show me a clean openldap server with openssl for TLS as a reference, how things should be correctly configured, but I this was not the case.

But at least, after my re configuring, there is no /tmp/openldap-tlsmc-*/cacerts folder any more created, so it looks okay so far.

As said, I will work now on a playbook to switch from NSS to openssl for my productive systems and then I hope, all goes well.

Thanks again for your great help.
Kind regards, Stefan

Comment 7 Matus Honek 2018-05-17 13:34:14 UTC
(In reply to mcguppy from comment #6)
> If I correctly understand you, the argument olcTLSCACertificateFile can be
> deleted and then the system wide ca-trust is used. I will try this.
Exactly.

> I have to say, it wasn't that easy to change this, as openLDAP very often
> complained "server is not willing to do.....".
> One have to do the steps in a correct order and I also had to stop it once
> and start it with only ldapi./// only to set the olcCertificateFile after I
> deleted it first (I was not able to just modify it).
That's what I meant by pointing you to the link (although it is not as explicit as it could be). Modifying cn=config is far less forgiving than plaintext slapd.conf is, unfortunately. You have to change all the TLS attrs you want to change, at once, atomically, like:
~~~
dn: cn=config
changeType: modify
replace: olcTLSx...
olcTLSx...: my_value_for_x
-
replace: olcTLSy...
olcTLSy...: my_value_for_y
~~~
The TLS context recreates from the new values right after you change them and if incorrect it leads to a rather difficult situation sometimes.

> It would be very helpful if openLDAP would be a bit more precise in the
> error messages, when you try something that is not possible. 
> I got many times just this error: "ldap_modify: Other (e.g., implementation
> specific) error (80)".
This is limited by RFCs, and security reasons as well - a precise error may some reveal vulnerability of the server to an attacker. Of course, a lot could be improved when it comes to being user friendly, as always.

> Another question: What do you advise to do with the NSS DB in
> /etc/openldap/certs after all is switched to openssl? Can I just delete the
> files in it or even the whole directory? Or could it be one day, that during
> an update, this will be recreated and will end up in some strange side
> effects?
AFAICT, the database should not get recreated unless both slapd.conf and slapd.d are missing in the /etc/openldap directory. Actually, we estimate most people were using PEM files instead even before.

> What I was really surprised this morning when setting up an completely new
> CentOS 7.5 and then install an openldap server, still NSS was configured. I
> thought this will show me a clean openldap server with openssl for TLS as a
> reference, how things should be correctly configured, but I this was not the
> case.
We cannot change user facing design that much so this is a courtesy to sysadmins. They might be expecting some state and were we to change this it would bring them obstacles in form of a need to alter deployment scripts (some have e.g. automatic deployment scripts for dynamic provisioning).

> But at least, after my re configuring, there is no
> /tmp/openldap-tlsmc-*/cacerts folder any more created, so it looks okay so
> far.
Correct. They don't get created if there was no NSS DB configured.

Glad to help. Now, I'm going close this bug as NOTABUG but feel free to follow up should you have more of related issues.


Note You need to log in before you can comment on or make changes to this bug.