Description of problem: After upgrading to F14, ldapsearch no longer communicates with LDAP directories when specifying ssl, i.e. ldapsearch -H ldaps://server -b dc=whtever,dc=com uid=myname Using -d 1 on the command line shows: TLS: could not initialize moznss using security dir /etc/pki/tls/certs I tried a variety of things, cleaning out the cert dir, moving it around, nothing seems to work. Open to suggestions how to track this down. strace did not help me at all. Version-Release number of selected component (if applicable): openldap-clients.i686 2.4.22-7.fc14 How reproducible: Always, see above. Actual results: see above. Expected results: Be able to search LDAP directories using SSL Additional info:
I think it's just the error message that is bad. The way openldap in F-14 works is that if you specify a cacertdir, it tries to use that as a Mozilla NSS cert/key database. If that is not possible, it falls back to using that directory as a directory of PEM CA cert files. Can you post the entire output of -d 1 as an attachment to this bug?
Created attachment 458189 [details] Output from '-d 1' Attached. Names and numbers changed to protect the innocent... Both /etc/ldap.conf and /etc/openldap/ldap.conf contain TLS_CACERTDIR /etc/pki/tls/certs
Try this: ls -al /etc/pki/tls/certs This worked for me: su - root LDAPTLS_CACERTDIR=/etc/pki/tls/certs ldapsearch -Z -H ldap://myhostname .... This did not: # do not su, use my normal user account LDAPTLS_CACERTDIR=/etc/pki/tls/certs ldapsearch -Z -H ldap://myhostname .... If I do this: su - root -c "chmod 644 /etc/pki/tls/certs/slapd.pem" Then the ldapsearch works. What is your setting for TLS_REQCERT in /etc/openldap/ldap.conf?
Grumble just lost all my comments due to a firefox hickup. Let me start again. So this will probably be a bit shorter than it was before. None of this works, regardless of being root, regular user, or the value of LDAPTLS_CACERTDIR. The error message remains the same: # echo $LDAPTLS_CACERTDIR /etc/pki/tls/certs TLS: could not initialize moznss using security dir /etc/pki/tls/certs - error -8174:Unknown code ___f 18. TLS certificate verification: subject: -unknown-, issuer: -unknown-, cipher: -unknown-, security level: off, secret key bits: 0, total key bits: 0, cache hits: 0, cache misses: 0, cache not reusable: 0 TLS certificate verification: bad If I set TLS_REQCERT to never, the query does work, but again, the error message (using -d 1) remains the same. I re-checked the cert, regenerated the link for the cert, etc. # ls -la $LDAPTLS_CACERTDIR total 12 drwxr-xr-x 2 root root 4096 Nov 6 09:01 . drwxr-xr-x. 6 root root 4096 Nov 6 08:58 .. lrwxrwxrwx 1 root root 7 Nov 6 09:01 901f571b.0 -> cert.pem -rw-r--r-- 1 root root 1793 Jun 4 16:04 cert.pem
ok, so it's not a permissions problem - could be a problem with the cert that NSS doesn't like How did you generate cert.pem i.e. did you use a CA? What openssl command did you use? Do openssl x509 -in $LDAPTLS_CACERTDIR/cert.pem -text and attach the output to this bug. Be sure to first obscure any sensitive information.
This cert is from our internal corporate CA. Not sure how they actually generate these, I can find out if need be. I've used those certs all over the place though and this all worked prior to upgrading, so I don't think it's a certificate issue. Perhaps it's a certificate chain issue? The error message is clearly totally useless in it's current form. Any way I can get something more descriptive? I'll attached an annotated version of the openssl output shortly. I can mail you the actual cert if you like...
Created attachment 458782 [details] output from openssl -text with names altered and encoded data removed
(In reply to comment #6) > This cert is from our internal corporate CA. Not sure how they actually > generate these, I can find out if need be. I've used those certs all over the > place though and this all worked prior to upgrading, so I don't think it's a > certificate issue. Perhaps it's a certificate chain issue? The error message is > clearly totally useless in it's current form. Any way I can get something more > descriptive? Unfortunately NSS is rather terse in it's errors. Looking at https://bugzilla.redhat.com/attachment.cgi?id=458189 > TLS: could not initialize moznss using security dir /etc/pki/tls/certs - error -8174:Unknown code ___f 18. Please ignore this error. This is not the problem. OpenLDAP tries to use the given dir first as a NSS key/cert database, but since we are using PEM files, it will fallback and assume the directory contains PEM files and get the messages we see below. > TLS: could not add the certificate (null) - error -8192:Unknown code ___f 0. -8192 means SEC_ERROR_IO - usually some sort of I/O error - unable to open file, etc. There are 5 of these messages in the log, which to me indicates that NSS attempted and failed to open/read 5 files - are there/were there more files in the $LDAPTLS_CACERTDIR? Perhaps symlinks pointing to non-existent files? > TLS certificate verification: Error, -8179: Unknown code ___f 13 -8179 means SEC_ERROR_UNKNOWN_ISSUER i.e. when we got the peer server cert, and checked the issuer, we could not determine it. This usually indicates a missing CA, could be a missing subordinate CA, which could also indicate a certificate chain issue. > > I'll attached an annotated version of the openssl output shortly. I can mail > you the actual cert if you like... No, that output is fine. It looks like the top level self signed CA cert of your organization - do you have any subordinate CAs that could have issued your server cert? Can you do the openssl x509 -in /path/to/server-cert.pem -text with the directory server certificate? That way we can see what the Subject is and what the Issuer is.
OK, some of these may have been since I tried a whole bunch of different things, such as grabbing the directory with the full client chain, etc. So let me get back to the basics, only have the cert in that dir and re-check.
OK, recreated the certs directory somewhere else, and did all of this as root: root > mkdir /root/certs root > export LDAPTLS_CACERTDIR=/root/certs root > cd $LDAPTLS_CACERTDIR root > cp -p /etc/pki/tls/certs/cert.pem ./cert.pem root > ln -s cert.pem `openssl x509 -hash -noout -in cert.pem`.0 root > ls -la $LDAPTLS_CACERTDIR total 12 drwxr-xr-x 2 root root 4096 Nov 8 14:13 . dr-xr-x---. 13 root root 4096 Nov 8 14:12 .. lrwxrwxrwx 1 root root 8 Nov 8 14:13 901f571b.0 -> cert.pem -rw-r--r-- 1 root root 1793 Jun 4 16:04 cert.pem root > ldapsearch -d 1 -x -H ldaps://ldap.bigcorporation.com -b dc=bigcorporation,dc=com uid=userid sn ldap_url_parse_ext(ldaps://ldap.bigcorporation.com) ldap_create ldap_url_parse_ext(ldaps://ldap.bigcorporation.com:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.bigcorporation.com:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 10.129.112.62:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 TLS: could not initialize moznss using security dir /root/certs - error -8174:Unknown code ___f 18. TLS certificate verification: subject: -unknown-, issuer: -unknown-, cipher: -unknown-, security level: off, secret key bits: 0, total key bits: 0, cache hits: 0, cache misses: 0, cache not reusable: 0 TLS certificate verification: bad TLS certificate verification: Error, -8179: Unknown code ___f 13 TLS: error: connect - force handshake failure -1 - error -8179:Unknown code ___f 13 TLS: can't connect: . ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) Using strace, I see it actually reading the /root/certs/cert.pem file (as well as the linked file): open("/root/certs/cert.pem", O_RDONLY|O_LARGEFILE) = 5 fstat64(5, {st_mode=S_IFREG|0644, st_size=1793, ...}) = 0 read(5, "-----BEGIN CERTIFICATE-----\nMIIE"..., 1793) = 1793 close(5) = 0 gettimeofday({1289244077, 33901}, NULL) = 0 open("/root/certs/901f571b.0", O_RDONLY|O_LARGEFILE) = 5 fstat64(5, {st_mode=S_IFREG|0644, st_size=1793, ...}) = 0 read(5, "-----BEGIN CERTIFICATE-----\nMIIE"..., 1793) = 1793 close(5) = 0 I've also verified that I can actually follow the whole chain using (though that uses the cert chain in /etc/pki/tls/certs/): root > openssl s_client -showcerts -connect ldap.bigcorporation.com:636 Short of getting some decent error messages (which I saw was noted during the test day), no idea how to progress this...
> I've also verified that I can actually follow the whole chain using (though > that uses the cert chain in /etc/pki/tls/certs/): > root > openssl s_client -showcerts -connect ldap.bigcorporation.com:636 The CA cert from https://bugzilla.redhat.com/show_bug.cgi?id=650288#c7 is a self signed CA cert - the issuer Issuer: O=BigCompany/emailAddress=support.com , L=RTP, C=US, ST=North Carolina, CN=BigCo SSL Certificate Authority, OU=BigCo SSL Certificate Authority is the same as the subject Subject: O=BigCompany/emailAddress=support.com , L=RTP, C=US, ST=North Carolina, CN=BigCo SSL Certificate Authority, OU=BigCo SSL Certificate Authority There should be no chaining since there is only one CA. Is this the same CA cert that was used to issue the ldap.bigcorporation.com server cert? Can you attach the "obscured" output of openssl s_client -showcerts -connect ldap.bigcorporation.com:636 ? I'd like to confirm that the client has the CA cert (and chain, if applicable) for the CA that issued the ldap.bigcorporation.com server cert.
After some private discussions, it turns out that TLS_CACERTDIR /etc/pki/tls/certs does not work - for some reason, it's not picking up one of the CA certs from the ca-bundle.crt. However, specifying TLS_CACERT /etc/pki/tls/certs/ca-bundle.crt does work. I'm going to do some more debugging of this case.
Turns out that if there's only one crt file in the /etc/pki/tls/certs/ca-bundle.crt directory, this works without having to specify the file. # ls -la /etc/pki/tls/certs total 576 drwxr-xr-x 2 root root 4096 Nov 10 08:18 . drwxr-xr-x. 7 root root 4096 Nov 10 08:11 .. -rw-r--r-- 1 root root 578465 Apr 7 2010 ca-bundle.crt and the ldapsearch works. Add the ca-bundle.trust.crt file, it fails. # ls -la /etc/pki/tls/certs total 1220 drwxr-xr-x 2 root root 4096 Nov 10 08:26 . drwxr-xr-x. 7 root root 4096 Nov 10 08:11 .. -rw-r--r-- 1 root root 578465 Apr 7 2010 ca-bundle.crt -rw-r--r-- 1 root root 658225 Nov 10 08:24 ca-bundle.trust.crt So, looks like there's some sort of issue there. Both those get provided by the ca-certificates package.
I'm quite surprised that TLS_CACERTDIR can handle ca-bundle.crt. From manpages: TLS_CACERT Specifies the file that contains certificates for all of the Certificate Authorities the client will recognize. TLS_CACERTDIR Specifies the path of a directory that contains Certificate Authority certificates in separate individual files. The TLS_CACERT is always used before TLS_CACERTDIR. The specified directory must be managed with the OpenSSL c_rehash utility.
(In reply to comment #14) > I'm quite surprised that TLS_CACERTDIR can handle ca-bundle.crt. From manpages: > > TLS_CACERT > Specifies the file that contains certificates for all of the Certificate > Authorities the client will recognize. > > TLS_CACERTDIR > Specifies the path of a directory that contains Certificate Authority > certificates in separate individual files. The TLS_CACERT is always used before > TLS_CACERTDIR. The specified directory must be managed with the OpenSSL > c_rehash utility. Perhaps it is an openssl hack? At any rate, I believe I have found the real problem - NSS does not like certs encoded with BEGIN TRUSTED CERTIFICATE that have some sort of trust provision. This is why ca-bundle.crt works but not ca-bundle.trust.crt. I have been able to confirm this with a single CA cert. If the cert is encoded with ---BEGIN CERTIFICATE and has no added trust, everything works fine. However, if I output the cert with openssl x509 -addtrust serverAuth -trustout, it encodes the cert with ---BEGIN TRUSTED CERTIFICATE and adds extra trust bits to the cert, which NSS does not like. certutil -A will refuse to import these certs. I'm investigating to see if NSS has support for these types of certificates even though certutil does not expose/use this functionality.
Anything else I can do? I assume that by picking the right config in /etc/openldap/ldap.conf this now should work even with the default certs that ca-certificates lays down. I'll try to make that work for my system.
(In reply to comment #16) > Anything else I can do? I assume that by picking the right config in > /etc/openldap/ldap.conf this now should work even with the default certs that > ca-certificates lays down. I'll try to make that work for my system. With your help, I've been able to track down a problem with our PEM cert handling. There's nothing else we need from you right now. I think as long as you don't use the ca-bundle.trust.crt file, you should be fine.
Can anyone confirm that with openldap built with openssl, if you use a cacertdir, and you have files in that directory that are in a format other than HASH.0, they will not be considered as valid certs? If this is true, then part of the fix is to make openldap with moznss ignore files not in that format.
Would a full up to date Fedora 13 box do?
(In reply to comment #19) > Would a full up to date Fedora 13 box do? Yes, but I was hoping someone knew. I will set up an F-13 vm too.
I just did some testing on F-13 with ldapsearch If I use LDAPTLS_CACERT=/path/to/file.pem ldapsearch -x -Z ... it works if I have a directory called /var/tmp/sec which has a hexhash.0 (hexhash generated with /etc/pki/tls/misc/c_hash) symlink linked to /path/to/file.pem and I run LDAPTLS_CACERTDIR=/var/tmp/sec ldapsearch -x -Z .... it works. if I remove the symlink and instead just copy /path/to/file.pem to /var/tmp/sec and I run LDAPTLS_CACERTDIR=/var/tmp/sec ldapsearch -x -Z .... it fails. So I think we can safely say that, when using LDAP TLS_CACERTDIR, only c_hash generated hashes of CA files are used. I've changed the bug summary accordingly, and this will be fixed in openldap. There is still a bug in NSS/PEMNSS that it doesn't support the TRUSTED CERTIFICATE nonstandard X509_CERT_AUX structure.
Shouldn't it trust both the c_hash files as well as the ca-bundle.crt file? Perhaps that's your intention, but clearly the summary now states something different.
(In reply to comment #22) > Shouldn't it trust both the c_hash files as well as the ca-bundle.crt file? No. At least, that's not how it works on F-13. If you have a directory like this: /path/to/cacertdir which contains cacert.pem ee2bf20.0 ca-bundle.crt And use LDAPTLS_CACERTDIR=/path/to/cacertdir ldapsearch .... The only file it will use is the ee2bf20.0 file. If however you specify LDAPTLS_CACERTDIR=/path/to/cacertdir LDAPTLS_CACERT=/path/to/cacertdir/ca-bundle.crt ldapsearch .... it will use both ee2bf20.0 and ca-bundle.crt > Perhaps that's your intention, but clearly the summary now states something > different.
Patch submitted upstream: http://www.openldap.org/its/index.cgi/Incoming?id=6705;selectid=6705
(In reply to comment #18) > Can anyone confirm that with openldap built with openssl, if you use a > cacertdir, and you have files in that directory that are in a format other than > HASH.0, they will not be considered as valid certs? If this is true, then part > of the fix is to make openldap with moznss ignore files not in that format. Rich, I'm sorry that I didn't confirm you that this is a correct behavior in F13. I haven't noticed that question. The man page (comment 14) is right. Thank you very much. I will apply that patch soon.
Fixed in openldap-2.4.23-3.fc15, openldap-2.4.23-3.fc14.
openldap-2.4.23-3.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-3.fc14
openldap-2.4.23-3.fc14 has been pushed to the Fedora 14 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update openldap'. You can provide feedback for this update here: https://admin.fedoraproject.org/updates/openldap-2.4.23-3.fc14
openldap-2.4.23-3.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.