Created attachment 463394 [details] sssd config used on both F13 and F14 Description of problem: My SSSD configuration which uses LDAP as the id_provider and auth_provider worked in Fedora 12, works in Fedora 13, but fails in Fedora 14 resulting in no user accounts (other than local accounts). It looks like a ldaps:// connection issue that only affects my F14 client, other F13 clients with the same configuration continue to work. (Note: I've sanitised the configs and logs in this report.) Version-Release number of selected component (if applicable): sssd-1.4.1-1.fc14.i686 openldap-2.4.22-7.fc14.i686 (also tried sssd-1.4.1-3.fc14.i686 from rawhide but got same results) How reproducible: Always. Steps to Reproduce: 1. Configure /etc/sssd/sssd.conf, see attached sssd.conf. 2. Start SSSD. I've included the results of the ldapsearch command as they may be related. The following /etc/openldap/ldap.conf was used on both Fedora 13 and 14. /etc/openldap/ldap.conf: URI ldaps://server.example.com.au/ BASE dc=example,dc=com,dc=au TLS_CACERT /etc/pki/tls/certs/cacert.pem Actual results: /var/log/messages from F14: Nov 24 15:42:28 localhost sssd: Starting up Nov 24 15:42:28 localhost sssd[be[example.com.au]]: Starting up Nov 24 15:42:28 localhost sssd[nss]: Starting up Nov 24 15:42:28 localhost sssd[pam]: Starting up Nov 24 15:42:30 localhost sssd[be[example.com.au]]: LDAP connection error: (null) Nov 24 15:42:30 localhost sssd[be[example.com.au]]: LDAP connection error: (null) The last line repeats. I tested changing the ldap_uri in sssd.conf from ldaps://... to ldap://... on F14. SSSD was able to get LDAP user accounts(displayed by "getent passwd"). Trying to login the users with their passwords failed (which is expected because sssd-ldap requires TLS/SSL when used for authentication). ldapsearch displays the following on F14: # ldapsearch -x -d 1 ldap_create ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP server.example.com.au:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying XX.XX.XX.XX:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 TLS: loaded CA certificate file /etc/pki/tls/certs/cacert.pem. TLS certificate verification: Error, -8156: Unknown code ___f 36 TLS: error: connect - force handshake failure -1 - error -8156:Unknown code ___f 36 TLS: can't connect: . ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) Error -8156 is SEC_ERROR_CA_CERT_INVALID, "Issuer certificate is invalid." according to http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html . My CA certificate file /etc/pki/tls/certs/cacert.pem is exactly the same(md5sum compared) on both F13 and F14 computers. The following openssl command successfully connects to the ldap server, when run on Fedora 14. # openssl s_client -connect server.example.com.au:636 -showcerts -state -CAfile /etc/pki/tls/certs/cacert.pem Could the problem be related to the Fedora 14 OpenLDAP switch from OpenSSL to NSS? On F14, the ldapsearch command succeeds if the URI in ldap.conf is changed from ldaps://... to ldap://... which is like the results from SSSD. Expected results: Using ldaps, SSSD retrives LDAP user accounts and allows those accounts to be loged-in using their passwords. I expect Fedora 14 should be able to get the same results as this Fedora 13 ldapsearch which successfully connects and displays LDAP users' info. # ldapsearch -x -d 1 ldap_create ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP server.example.com.au:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying XX.XX.XX.XX:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 TLS trace: SSL_connect:before/connect initialization TLS trace: SSL_connect:SSLv2/v3 write client hello A TLS trace: SSL_connect:SSLv3 read server hello A TLS certificate verification: depth: 1, err: 0, subject: /C=AU/ST=Victoria/O=Example Company/CN=server, issuer: /C=AU/ST=Victoria/O=Example Company/CN=server TLS certificate verification: depth: 0, err: 0, subject: /C=AU/ST=Victoria/L=Melbourne/O=Example Company/CN=server.example.com.au, issuer: /C=AU/ST=Victoria/O=Example Company/CN=server TLS trace: SSL_connect:SSLv3 read server certificate A TLS trace: SSL_connect:SSLv3 read server done A TLS trace: SSL_connect:SSLv3 write client key exchange A TLS trace: SSL_connect:SSLv3 write change cipher spec A TLS trace: SSL_connect:SSLv3 write finished A TLS trace: SSL_connect:SSLv3 flush data TLS trace: SSL_connect:SSLv3 read finished A ldap_open_defconn: successful ...SNIP... Additional info:
Jared, please, can you try openldap from updates-testing? There were some changes targeting SSL/TLS log messages issues. https://admin.fedoraproject.org/updates/openldap-2.4.23-4.fc14 (If it helps, please, increase the karma.)
Reassigning this bug to openldap. If ldapsearch is also affected, then it is an openldap issue, not an SSSD one. SSSD uses the openldap libraries internally to manage its connections.
Thanks Stephen for reassigning. I wasn't sure if or how SSSD used openldap. Jan, it's now running openldap-2.4.23-4.fc14.i686 from updates-testing. The ldapsearch error message is a bit more detailed but mostly the same. I also turned up the debug level a bit. # ldapsearch -x -d 3 ldap_create ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP server.example.com.au:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying XX.XX.XX.XX:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 TLS: loaded CA certificate file /etc/pki/tls/certs/cacert.pem. tls_write: want=57, written=57 0000: 80 37 01 03 01 00 1e 00 00 00 10 00 00 04 00 fe .7.............. ...<snip>... 0030: 7b c7 06 04 d6 1d 1f 43 18 {......C. tls_read: want=3, got=3 0000: 16 03 01 ... tls_read: want=2, got=2 0000: 00 4a .J tls_read: want=74, got=74 0000: 02 00 00 46 03 01 4c f4 33 ea db 09 40 43 4a b0 ...F..L.3...@CJ. ...<snip>... 0040: 8d 9f 33 71 08 f8 76 00 04 00 ..3q..v... tls_read: want=5, got=5 0000: 16 03 01 05 5b ....[ tls_read: want=1371, got=1371 0000: 0b 00 05 57 00 05 54 00 02 b9 30 82 02 b5 30 82 ...W..T...0...0. ...<snip>... 0550: ce 48 50 b4 00 c0 4f 03 ca 1f 28 .HP...O...( TLS certificate verification: Error, -8156: Unknown code ___f 36 tls_write: want=7, written=7 0000: 15 03 01 00 02 02 2a ......* TLS: error: connect - force handshake failure: errno 21 - moznss error -8156 TLS: can't connect: . ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
Can you use openssl verify to verify /etc/pki/tls/certs/cacert.pem ? see man verify for more info
does ldapsearch -Z -H ldap://server.example.com.au work?
Created attachment 463647 [details] result of ldapsearch -x -Z -H... (openldap-2.4.23-4)
Both the CA certificate and the LDAP server certificate seem OK. # openssl verify -verbose -x509_strict -issuer_checks -CAfile /etc/pki/tls/certs/cacert.pem /etc/pki/tls/certs/cacert.pem /etc/pki/tls/certs/cacert.pem: OK # openssl verify -purpose sslserver -verbose -x509_strict -issuer_checks -CAfile /etc/pki/tls/certs/cacert.pem servercert.pem servercert.pem: C = AU, ST = Victoria, L = Melbourne, O = Example Company, CN = server.example.com.au error 29 at 0 depth lookup:subject issuer mismatch C = AU, ST = Victoria, L = Melbourne, O = Example Company, CN = server.example.com.au error 29 at 0 depth lookup:subject issuer mismatch C = AU, ST = Victoria, L = Melbourne, O = Example Company, CN = server.example.com.au error 29 at 0 depth lookup:subject issuer mismatch OK "ldapsearch -x -Z -H ldap://server.example.com.au" did not work on F14 but did work on F13 (I needed to add -x, I hope you don't mind). The F14 results are attached: https://bugzilla.redhat.com/attachment.cgi?id=463647
Can you attach /etc/pki/tls/certs/cacert.pem to this bug? If not, can you do openssl x509 -in /etc/pki/tls/certs/cacert.pem -text and obscure the sensitive parts? I'm thinking there is something about the CA cert that NSS doesn't like.
Created attachment 463896 [details] obscured output from "openssl x509 -in cacert.pem -text" I've attached the partly obscured output of openssl x509 -in /etc/pki/tls/certs/cacert.pem -text. The "Issuer:" and "Subject:" lines were altered and some lines were removed after "BEGIN CERTIFICATE", everything else is unaltered.
Thanks. The only thing that looks unfamiliar is Netscape Comment: OpenSSL Generated Certificate I would expect the MozNSS code to handle this, but perhaps it throws it off. Can you try this: mkdir /var/tmp/junk certutil -d /var/tmp/junk -A -t CT,, -n "CA certificate" -a -i /etc/pki/tls/certs/cacert.pem then if that works, try certutil -d /var/tmp/junk -V -n "CA certificate" -e I'm going to generate a cert with that extension and see if I can reproduce.
Ok. The problem is that the cacert is not a cacert: https://bugzilla.redhat.com/attachment.cgi?id=463896 X509v3 extensions: X509v3 Basic Constraints: CA:FALSE NSS is objecting to the use of this cert as a CA cert. I'm trying to find out if there is a workaround.
Nice find, thanks for looking into that. So this change from F13 to F14 openldap is probably not a bug, just stricter checking? In my case creating a new valid CA certificate should be an OK solution. Thanks for your help!
(In reply to comment #12) > Nice find, thanks for looking into that. So this change from F13 to F14 > openldap is probably not a bug, just stricter checking? Yes, although I'm looking at the openssl code to determine why it doesn't complain about the CA basic constraints set to FALSE > > In my case creating a new valid CA certificate should be an OK solution. > > Thanks for your help!
Hi, I have the same problem. In my self-signed certificate I have: X509v3 Basic Constraints: CA:TRUE Thanks
(In reply to comment #14) > Hi, > I have the same problem. > In my self-signed certificate I have: > X509v3 Basic Constraints: > CA:TRUE > > Thanks Are you using a self signed CA certificate as your ldap server certificate? Or do you have a separate server cert and key issued by the CA? Can you provide the information such as: the output of running ldapsearch -d 3 which shows the TLS/SSL trace information, including the moznss error code the output of openssl x509 -in /path/to/your/cacert.pem -text (be sure to obscure any sensitive information first)
Created attachment 464286 [details] Output of ldapsearch -d 3 This is the output of "ldapsearch -d 3"
Created attachment 464287 [details] obscured output of "openssl x509 -text -in cert.pem" This is the obscured output of "openssl x509 -text -in cert.pem". We are using self signed certificates
This is a different error: TLS: error: connect - force handshake failure -1 - error -8179:Unknown code ___f 13 -8179 is SEC_ERROR_UNKNOWN_ISSUER Can you attach the obscured output of openssl x509 -text -in /path/to/ldap.server.cert.pem
Created attachment 464300 [details] obscured output of "openssl x509 -text -in ldap-master.pem" Sorry, I thought it was the same error :-( Would you please investigate further?
(In reply to comment #19) > Created attachment 464300 [details] > obscured output of "openssl x509 -text -in ldap-master.pem" > > Sorry, I thought it was the same error :-( No. Your problem produces a different moznss error code (you have -8179 while the original reporter had -8156) > Would you please investigate further? Yes. moznss is correctly reporting that the ca cert is incorrect. Your server cert has this as the issuer: https://bugzilla.redhat.com/attachment.cgi?id=464300 Issuer: C=IT, ST=Italy, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster But the CA cert has this as the subject: https://bugzilla.redhat.com/attachment.cgi?id=464287 Subject: C=IT, ST=Italy, L=Rome, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster Note that the Issuer in the server cert does not have L=Rome but the CA cert does.
Hi, I understand that there's a problem with this certificate. The thing that don't understand is why this cert works in F13 (and lower) with the option "TLS_REQCERT hard" in /etc/openldap/ldap.conf.
(In reply to comment #21) > Hi, > I understand that there's a problem with this certificate. > The thing that don't understand is why this cert works in F13 (and lower) > with the option "TLS_REQCERT hard" in /etc/openldap/ldap.conf. Does the CA cert for Issuer: C=IT, ST=Italy, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster have the same signature as the CA cert for Issuer: C=IT, ST=Italy, L=Rome, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster ? Although, if the Subject: DN in the CA certs are different, the signatures should also be different. Do you still have your F-13 setup available? If so, could you provide the output of running ldapsearch with the -d 1 option to show the client checking the CA cert as part of the TLS/SSL handshake? For example, from the original bug report: TLS certificate verification: depth: 1, err: 0, subject: /C=AU/ST=Victoria/O=Example Company/CN=server, issuer: /C=AU/ST=Victoria/O=Example Company/CN=server TLS certificate verification: depth: 0, err: 0, subject: /C=AU/ST=Victoria/L=Melbourne/O=Example Company/CN=server.example.com.au, issuer: /C=AU/ST=Victoria/O=Example Company/CN=server
I just tried an experiment on F-13 with the latest updates. I generated a CA cert + server cert/key with /etc/pki/tls/misc/CA - I set up slapd to use this CA cert and server cert/key. I then generated a new CA cert with a slightly different subject (cacert2.pem) I then did the following tests: * LDAPTLS_CACERT=cacert.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... works * LDAPTLS_CACERT=cacert2.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... fails with the following error: ldap_start_tls: Connect error (-11) additional info: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (self signed certificate in certificate chain) * LDAPTLS_REQCERT=never LDAPTLS_CACERT=cacert2.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... works * LDAPTLS_REQCERT=allow LDAPTLS_CACERT=cacert2.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... works * LDAPTLS_REQCERT=demand LDAPTLS_CACERT=cacert2.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... fails with the above error * LDAPTLS_REQCERT=hard LDAPTLS_CACERT=cacert2.pem ldapsearch -x -LLL -ZZ -H ldap://myhost ... fails with the above error So I would like some more information about how to reproduce the behaviour that you are seeing.
Created attachment 464614 [details] output of ldapwhoami -d 1 in F13 I know there's something wrong with my certificates, but they work in my environment. The ldap server is a FC8 The signatures are different, but I see that is a normal situation. It's not simple in my big network to change all certificates so I would like to know what's wrong and take some decisions consequently The output of F13 is made with the right CA cert (before there was another one, a sort of a renewal of the original CA cert, sorry!) and it works Furthermore, something has changed in F14 using the right CA cert and the output of ldapwhoami -d 1 -x -H ldaps://ldap.example.it is ldap_url_parse_ext(ldaps://ldap-slave.example.it) ldap_create ldap_url_parse_ext(ldaps://ldap-slave.example.it:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap-slave.example.it:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 192.168.1.2:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 TLS: loaded CA certificate file /etc/pki/tls/certs/LDAP.cacert.pem. TLS certificate verification: Error, -8156: Unknown code ___f 36 TLS: error: connect - force handshake failure: errno 21 - moznss error -8156 TLS: can't connect: . ldap_err2string ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) Now the error is -8156
> Now the error is -8156 Ok. That's the original error. Is the CA cert the same as https://bugzilla.redhat.com/attachment.cgi?id=464287 ?
Not the same: Certificate: Data: Version: 3 (0x2) Serial Number: 0 (0x0) Signature Algorithm: sha1WithRSAEncryption Issuer: C=IT, ST=Italy, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster Validity Not Before: Nov 9 16:52:57 2007 GMT Not After : Nov 6 16:52:57 2017 GMT Subject: C=IT, ST=Italy, O=example, OU=LDAP, CN=ldap-master.example.it/emailAddress=postmaster Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Comment: OpenSSL Generated Certificate X509v3 Subject Key Identifier: X509v3 Authority Key Identifier: Signature Algorithm: sha1WithRSAEncryption
(In reply to comment #26) > Not the same: > snip Ok. So this is exactly the same problem. I'm still trying to slog through the openssl code to figure out why this bogus CA cert is acceptable to openssl. Once I do that, I can figure out how to develop a workaround/patch for F-14.
Do we need to be bug per bug compatible ? If the cert is bogus what's wrong with failing ? Just asking what is the rationale, not recommending a course of action here.
(In reply to comment #28) > Do we need to be bug per bug compatible ? If there are differences in behaviour, we need to understand why, and be ready to give customers/users explanations, workarounds, and patches. > If the cert is bogus what's wrong with failing ? The problem is that openssl and moznss have a different view of "trust". I'm trying to figure out why and how. > Just asking what is the rationale, not recommending a course of action here.
New bug opened with openldap upstream: http://www.openldap.org/its/index.cgi/Incoming?id=6742 Patch submitted to openldap upstream: ftp://ftp.openldap.org/incoming/openldap-2.4.23-centralize-and-improve-cert-verification-20101209.patch
Better ITS URL: http://www.openldap.org/its/index.cgi?findid=6742
Additional patch ftp://ftp.openldap.org/incoming/openldap-2.4.23-allow-self-issued-certs-to-verify-20101210.patch
Thanks Rich! Fixed in: openldap-2.4.23-5.fc14 openldap-2.4.23-5.fc15
openldap-2.4.23-5.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-5.fc14
openldap-2.4.23-5.fc14 has been pushed to the Fedora 14 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update openldap'. You can provide feedback for this update here: https://admin.fedoraproject.org/updates/openldap-2.4.23-5.fc14
This patch causes a regression. The same issue as in bug 668899 reported: ryan.ca (Anonymous) - 2011-01-12 23:07:30 This update broke client access with TLS/SSL to our OpenLDAP server. ldapsearch issues moznss error -8101 (SEC_ERROR_INADEQUATE_CERT_TYPE). The server cert works fine with 2.4.23-4.fc14 clients but fails with 2.4.23-5.fc14, preventing LDAP access.
(In reply to comment #36) > This patch causes a regression. The same issue as in bug 668899 reported: > > ryan.ca (Anonymous) - 2011-01-12 23:07:30 > This update broke client access with TLS/SSL to our OpenLDAP server. ldapsearch > issues moznss error -8101 (SEC_ERROR_INADEQUATE_CERT_TYPE). The server cert > works fine with 2.4.23-4.fc14 clients but fails with 2.4.23-5.fc14, preventing > LDAP access. It's not quite the same error. In this case, the error code is -8101, and in the bug 668899 case, it is error -8102. Can you post the exact error message from above? The error message should list the subject name of the cert. Is the problem with the server cert or the CA cert? Can you attach the output of openssl x509 -in /path/to/problem/cert.pem -text? If the error is coming from the server cert - is the server's cert in the client's "trusted" cert store (i.e. in the CACERTDIR)? Can you post your client's TLS settings from /etc/ldap.conf or /etc/openldap/ldap.conf or wherever the app stores the settings?
I believe https://bugzilla.redhat.com/show_bug.cgi?id=657984#c36 is the same problem as bug 668899
openldap-2.4.23-6.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-6.fc14
openldap-2.4.23-7.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-7.fc14
openldap-2.4.23-8.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-8.fc14
Package openldap-2.4.21-12.fc13: * should fix your issue, * was pushed to the Fedora 13 updates-testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing openldap-2.4.21-12.fc13' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/openldap-2.4.21-12.fc13 then log in and leave karma (feedback).
Package openldap-2.4.23-9.fc14: * should fix your issue, * was pushed to the Fedora 14 updates-testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing openldap-2.4.23-9.fc14' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/openldap-2.4.23-9.fc14 then log in and leave karma (feedback).
openldap-2.4.23-10.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/openldap-2.4.23-10.fc14
openldap-2.4.23-10.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.