Bug 738456

Summary: OpenLDAP linked with Mozilla NSS intermittently fails to initialize
Product: Red Hat Enterprise Linux 6 Reporter: Rich Graves <rgraves>
Component: curlAssignee: Kamil Dudka <kdudka>
Status: CLOSED ERRATA QA Contact: Jiri Jaburek <jjaburek>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.1CC: anshockm, craig, ddumas, emaldona, jjaburek, jon.dufresne, jorton, jplans, jvcelak, jwest, kdudka, kengert, mniranja, msvoboda, nalin, ngupta, ovasik, prc, rainer.traut, rdassen, redhat, rrelyea, steve
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The OpenLDAP suite was recently modified to use NSS instead of OpenSSL as the SSL back end. This change led to collisions between libcurl and OpenLDAP on NSS initialization and shutdown. Consequently, applications that were using both, libcurl and OpenLDAP, failed to establish SSL connections. This update modifies libcurl to use the same NSS API as OpenLDAP, which prevents collisions from occurring. Applications using OpenLDAP and libcurl can now connect to the LDAP server over SSL as expected.
Story Points: ---
Clone Of:
: 1152026 (view as bug list) Environment:
Last Closed: 2013-02-21 10:09:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 772628, 772642, 812423    
Bug Blocks: 800904, 1152026    
Attachments:
Description Flags
minimal reproducer in C
none
NSS_InitContext for CURL
none
NSS_INIT_PK11RELOAD for CURL
none
NSS_InitContext for CURL V2
none
NSS_InitContext for CURL V3 rrelyea: review+, rmeggins: review+

Description Rich Graves 2011-09-14 20:40:41 UTC
Description of problem:

With varying frequency (0.1%-5.0%), outbound LDAPS binds using the client fail.

Version-Release number of selected component (if applicable):

openldap-2.4.23-15.el6_1.1.x86_64

How reproducible:

Always (given enough trials)

Steps to Reproduce:

1. Install fully patched RHEL6.1 Xen DomU guest on top of RHEL5 DomU, currently 2.6.18-238.12.1.el5xen and xen-3.0.3-120.el5_6.2. I do not know if Xen has anything to do with it, but if you have trouble reproducing on KVM or bare metal, try Xen.

2. Install PHP and Apache.

Configure httpd.conf with default-ish prefork MPM settings.

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
</IfModule>

3. Install any LDAP server on some other system

4. Point a web browser at ldaptest.php as follows:

<?php
echo "Using process id " . getmypid() . "<p>";
ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7);
$ldapconn = ldap_connect("ldaps://ldapserver1/ ldaps://ldapserver2/")
    or die("Could not connect to LDAP server.");
$ldapbind = ldap_bind($ldapconn);
    if ($ldapbind) {
        // echo "LDAP bind successful...";
        ldap_close($ldapconn);
    } else {
        echo "LDAP bind failed... $errormsg<br>";
        ldap_close($ldapconn);
        exit();
    }
}
usleep(50000);
echo "<script language=\"javascript\">location.replace(\"https://ldaptest.php?count=$count\");</script>";
?>
 
Actual results:

Individual httpd/php threads experience errors initializing Mozilla NSS.

Expected results:

Smooth LDAP binds on all httpd child processes.

Additional info:

Resolved by downgrade to openldap-2.4.19-15.el6_0.2.x86_64, the last version linked with OpenSSL. I have not yet tried recompiling the 2.4.23 SRPM agains OpenSSL.

Comment 1 Rich Graves 2011-09-14 20:44:39 UTC
See also the thread September 8th - September 11th on rhelv6-list.

What led me to pinpoint the change in linking to Mozilla NSS rather than OpenLDAP:

From time to time, a specific Apache child process loses the ability to connect to any LDAP server over SSL. It is not clear how processes get into this state (lsof, and adding %P %k %T %X to LogFormat, show no obvious differences), but once they do, all attempts to reach ldaps:// URLs via either ldap_connect/ldap_bind or curl_init/curl_exec from that specific httpd child process fail.

Reducing Apache MaxRequestsPerChild (currently at 200) appears to reduce the incidence of this problem, but it never goes away entirely. apachectl graceful appears to stop it for a while.

ldapserver1 and ldapserver2 are most certainly up, actively serving other clients with no resource constraints. Capturing network traffic, we see a completed 3-way handshake, then an immediate FIN from the client with no data pushed. If I run passthru("/usr/bin/ldapsearch -x -LLL -H ldaps://ldapserver1.com ou=People") in the same PHP script where ldap_bind and curl_init("ldaps://") fail, only the external ldapsearch binary succeeds.

In PHP, jacking up ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7) results in the PHP error log output below, suggesting that the problem is in moznss, or the OpenLDAP linkage thereto.

What else can I do here?

ldap_create
ldap_url_parse_ext(ldaps://ldapserver1.com/)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ldapserver1.com:636
ldap_new_socket: 25
ldap_prepare_socket: 25
ldap_connect_to_host: Trying 172.20.9.5:636
ldap_pvt_connect: fd: 25 tm: 20 async: 0
ldap_ndelay_on: 25
ldap_int_poll: fd: 25 tm: 20
ldap_is_sock_ready: 25
ldap_ndelay_off: 25
ldap_pvt_connect: 0
TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available
TLS: can't create ssl handle.
ldap_err2string
ldap_err2string
ldap_create
ldap_url_parse_ext(ldaps://ldapserver2.com/)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ldapserver2.com:636
ldap_new_socket: 25
ldap_prepare_socket: 25
ldap_connect_to_host: Trying 172.20.9.6:636
ldap_pvt_connect: fd: 25 tm: 20 async: 0
ldap_ndelay_on: 25
ldap_int_poll: fd: 25 tm: 20
ldap_is_sock_ready: 25
ldap_ndelay_off: 25
ldap_pvt_connect: 0
TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available
TLS: can't create ssl handle.
ldap_err2string

Comment 3 Jan Vcelak 2011-09-17 12:31:33 UTC
Thank you for your report. I can reproduce it very easily. Although ldap.conf is configured correctly and 'ldapsearch -x -ZZ -H ldap://server' works, the same connection from php script with mod_php fails. Starting the script in shell works as well.

Multiple requests share the same process. If the first SSL initialization fails (which I do not know why at the moment), all subseqent connections from the same  process fail as well.

The first request:

TLS: could not add the certificate (null) - error -8192:Unknown code ___f 0.
TLS: /etc/pki/tls/certs/slapd.pem is not a valid CA certificate file - error -8192:Unknown code ___f 0.
TLS: did not find any valid CA certificates in /etc/pki/tls/certs/slapd.pem
TLS: could perform TLS system initialization.
TLS: error: could not initialize moznss security context - error -8192:Unknown code ___f 0
TLS: can't create ssl handle.

The second request.

TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available
TLS: can't create ssl handle.

Give me some time to investigate.

Comment 4 Jan Vcelak 2011-09-17 13:12:27 UTC
(In reply to comment #3)
> I can reproduce it very easily.

Well, not really. The certificate was inaccessible because of SELinux restrictions. Sorry. I will be trying.

Comment 5 Jan Vcelak 2011-09-17 18:39:25 UTC
(In reply to comment #1)
> What else can I do here?

Please, can you take a look at the Apache error log again and try to find another error which starts with:
TLS: error: could not initialize moznss security context
And is followed by:
TLS: can't create ssl handle.

Is there any?

Comment 6 Rich Graves 2011-09-18 02:26:08 UTC
Caught one. I didn't get this before because I was only jacking up LDAP_OPT_DEBUG_LEVEL for second and subsequent attempts. Alas, this doesn't tell me a lot...

ldap_pvt_connect: fd: 17 tm: -1 async: 0
TLS: could not initialize moznss - error -8192:Unknown code ___f 0.
TLS: could perform TLS system initialization.
TLS: error: could not initialize moznss security context - error -8192:Unknown code ___f 0
TLS: can't create ssl handle.

Comment 7 Jan Vcelak 2011-09-18 12:55:25 UTC
-8192 is SEC_ERROR_IO. == An I/O error occurred during authentication; or
an error occurred during crypto operation (other than signature verification).

I didn't manage to reproduce it on bare metal and on KVM. I will try to get a Xen guest.

Does the problem appear always, even if there is almost no load on the server? Or do I need to reproduce some load?

Comment 8 Rich Graves 2011-09-18 21:53:29 UTC
-8192 is a broad error category indeed.

Could it have something to do with low entropy for session key generation? I noticed that /proc/sys/kernel/random/entropy_avail was hovering around 128, but I'd expect /dev/urandom to be in use, so I wouldn't expect that to be a problem.

Great. Now I'm having trouble reproducing. I hate that. It only took 600 cycles last night, but now I'm up to 10,000 with no error so far.

Our server was never heavily loaded, but there were other requests coming in, both static files (some PDFs up to 100MB) and PHP. Only a small minority of page hits exercise php-ldap.

The code I'm actually using is below (lots of bugs in the badly redacted version above). Also, FWIW, all connections are https.

<?php
echo "Using process id " . getmypid() . "<p>";
ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7);
$ldapconn = ldap_connect("ldaps://ldap2/ ldaps://ldap0/")
    or die("Could not connect to LDAP server.");
ldap_set_option($ldapconn, LDAP_OPT_DEBUG_LEVEL, 7);
$ldapbind = ldap_bind($ldapconn);
    if ($ldapbind) {
        // echo "LDAP bind successful...";
        ldap_close($ldapconn);
    } else {
        echo "LDAP bind failed... $errormsg<br>";
        ldap_close($ldapconn);
        exit();
    }
$count = $_GET["count"] + 1;
echo "<script language=\"javascript\">location.replace(\"/ldaptest2.php?count=$count\");</script>";
?>

<?php
header('Location: /ldaptest.php?count=' . $_GET['count']);
?>

Comment 9 Rich Graves 2011-09-18 21:58:18 UTC
6 more failures in about 20,000 requests, interleaved with production traffic (where errors were also seen; some application logic works around it, mostly). 

All essentially the same:

ldap_pvt_connect: fd: 25 tm: 20 async: 0
ldap_ndelay_on: 25
ldap_int_poll: fd: 25 tm: 20
ldap_is_sock_ready: 25
ldap_ndelay_off: 25
ldap_pvt_connect: 0
TLS: could not initialize moznss - error -8192:Unknown code ___f 0.
TLS: could perform TLS system initialization.
TLS: error: could not initialize moznss security context - error -8192:Unknown code ___f 0
TLS: can't create ssl handle.
ldap_err2string
ldap_err2string
ldap_create
ldap_url_parse_ext(ldaps://ldap/)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request

Comment 10 Jan Vcelak 2011-09-21 12:41:37 UTC
I tried 40,000 requests, also with different configuration. And I had not hit the problem.

You will have to provide more details. It would be great, if you could attach relevant parts of httpd configuration and ldap.conf. Is the LDAP server running the same machine? I will also need your server certificates (without the keys and with other sensitive data obfuscated) - both for httpd and openldap.

Comment 12 Jan Vcelak 2011-09-21 12:43:00 UTC
(In reply to comment #10)
> You will have to provide more details.

You can also include full httpd error log. Again, you can obfuscate any sensitive data.

Thanks

Comment 13 Rich Graves 2011-09-21 17:40:36 UTC
Thanks for trying. I cloned the VMs at issue and am trying to reproduce the exact clicktrail.

What I know for sure is that if I upgrade to 2.4.23 on our production Moodle 1.99 instances, we will get hundreds of moznss failures per day. Reverting to 2.4.19 eliminates the errors completely, with no other changes.

It *seems* to get worse the longer the Xen VM and/or Apache have been running, though I don't have good proof of this.

Some wild theories:

1) Time skew for Xen guests -- RHEL6 guests lose time, and I step with ntpdate to compensate.
2) Limited entropy, or something like that.
3) An Apache or PHP module instantiated by a previous request served by the same Apache child process pollutes some unknown namespace -- a library, a mutex, or something else.

If you can flag this bug as Private, I'll upload the full configs, else I'll try to redact.

Comment 14 Rich Graves 2011-09-21 17:53:39 UTC
Both web server and LDAP servers have the same cert chain.

When an individual httpd/php/openldap/moznss child process gets unhapy, it is unable to connect to *any* ldaps server, including an Active Directory server with a private root.

The web server cert is Internet-accessible:

depth=2 /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
verify return:1
depth=1 /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO High-Assurance Secure Server CA
verify return:1
depth=0 /C=US/postalCode=55057/ST=MN/L=Northfield/streetAddress=1 North College St./O=Carleton College/OU=ITS/OU=PlatinumSSL/CN=moodle.carleton.edu
verify return:1

/etc/openldap/ldap.conf
URI ldap://(ldapserver)/ ldap://(ldapserver2)/
BASE ou=People,dc=carleton,dc=edu
TLS_CACERTDIR /etc/openldap/cacerts

# c_rehash /etc/openldap/cacerts/
Doing /etc/openldap/cacerts/
incommon-root.pem => 157753a5.0              <=== ldap root
incommon-intermediate2.pem => 9ef5f911.0     <=== ldap intermediate
ads-dc01-CA.pem => 7f36f249.0
incommon-intermediate.pem => 95f6ee89.0

All of the above are also present and hashed in /etc/pki/tls/certs.

selinux is Enforcing, but problem also occurs with Permissive and Disabled.

I'll see what I can do about configs and logs.

Comment 15 Rich Graves 2011-09-21 19:41:35 UTC
OK, I can reproduce on a clone now, 100% of the time.

The problem manifests on httpd/php/openldap/moznss child processes on which the Moodle maintenance script /admin/cron.php, which we tend to disable on test clones, has previously run. This script does a lot of things, including fetching a bunch of RSS feeds with php-curl. Since "install and configure Moodle" is an absurd test case, I'll narrow it down.

Obviously this involves a lot more than just openldap/moznss, but the "fix" remains downgrading to the last package before you switched from openssl to moznss, so I think this bugzilla is still properly assigned.

Comment 16 Rich Graves 2011-09-21 20:15:17 UTC
(We accept that finding a concise test case is our job, not yours. But FYI, in case anyone is working in parallel...)

We're starting to get suspicious of php-xml, not curl.

Relevant code includes:

https://github.com/moodle/moodle/blob/v1.9.13/admin/cron.php
https://github.com/moodle/moodle/blob/v1.9.13/lib/rsslib.php

Full list of PHP packages we have installed is below. I don't believe we have any nonstandard PEAR builds. We believe that some function, provided by something in this list, is causing moznss initialization in php-ldap to fail.

php-devel-5.3.3-3.el6.x86_64
php-xmlrpc-5.3.3-3.el6.x86_64
php-odbc-5.3.3-3.el6.x86_64
php-pdo-5.3.3-3.el6.x86_64
php-mysql-5.3.3-3.el6.x86_64
php-cli-5.3.3-3.el6.x86_64
php-pear-1.9.0-2.el6.noarch
php-mbstring-5.3.3-3.el6.x86_64
php-ldap-5.3.3-3.el6.x86_64
php-common-5.3.3-3.el6.x86_64
php-5.3.3-3.el6.x86_64
php-gd-5.3.3-3.el6.x86_64
php-xml-5.3.3-3.el6.x86_64

Comment 17 Jan Vcelak 2011-09-21 20:57:33 UTC
(In reply to comment #13)
> If you can flag this bug as Private, I'll upload the full configs, else I'll
> try to redact.

I have switched the bug visibility to private in case you decide to upload your configuration files. Of course, it is not necessary if you manage to find the cause of the problem.

Comment 18 Rich Graves 2011-09-22 03:25:42 UTC
Bug can be public again.

Isolated. The ldaps init in php-ldap conflicts with the https init in php curl.so (part of php-common package). This can be reproduced from the command line, php test.php:


<?php

$url = 'https://bugzilla.redhat.com/';
$fp = curl_init();
curl_setopt($fp, CURLOPT_URL, $url);
curl_setopt($fp, CURLOPT_HEADER, 1);
curl_setopt($fp, CURLOPT_RETURNTRANSFER, 1);

$headers = curl_exec($fp);
curl_close($fp);

// You don't actually need a working LDAP server, just
// something that will accept an TCP connection.
ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7);
$ldapconn = ldap_connect("ldaps://bugzilla.redhat.com:443/");
$ldapbind = ldap_bind($ldapconn);
$errormsg = ldap_error($ldapconn);

if ($ldapbind) {
  echo "\nok\n";
} else {
  echo "\n$errormsg\n";
}

?>

Comment 19 Jan Vcelak 2011-09-22 08:02:06 UTC
(In reply to comment #18)
> Bug can be public again.

OK.

> <?php

Perfect, I can reproduce. Give me some time...

Comment 20 Jan Vcelak 2011-09-29 12:39:08 UTC
Created attachment 525552 [details]
minimal reproducer in C

Attaching minimal reproducer.

Comment 21 Jan Vcelak 2011-09-29 12:47:28 UTC
Created attachment 525555 [details]
NSS_InitContext for CURL

- use NSS_InitContext instead of NSS_Initialize
- add NSS_INIT_PK11RELOAD to init flags

Results with the reproducer:

$ ./reproducer https://bugzilla.redhat.com ldaps://ldapserver 0
curl done
ldap done
$ ./reproducer https://bugzilla.redhat.com ldaps://ldapserver 1
ldap done
curl done

Comment 22 Jan Vcelak 2011-09-29 12:49:22 UTC
Created attachment 525556 [details]
NSS_INIT_PK11RELOAD for CURL

- adds NSS_INIT_PK11RELOAD flag to current NSS_Initialize

Results with the reproducer:

$ ./reproducer https://bugzilla.redhat.com ldaps://ldapserver 0
curl done
ldap done
$ ./reproducer https://bugzilla.redhat.com ldaps://ldapserver 1
ldap done
curl_easy_perform() failed: Peer certificate cannot be authenticated with known CA certificates

Comment 23 Jan Vcelak 2011-09-29 13:14:02 UTC
The problem lays somewhere between libldap and libcurl. Actually, there are two problems.

1.) CURL uses NSS_Initialize and NSS_Shutdown, which is not context aware and affects the whole process. I wrote the patch which adds context aware initialization (NSS_InitContext, NSS_ShutdownContext) to CURL. NSS documentation is not very up to date, but OpenLDAP already uses this functionality. This should be added to CURL, otherwise the library could close other libraries NSS context.

2.) The second initialization always fails, because PKCS#11 module is reinitialized (CKR_CRYPTOKI_ALREADY_INITIALIZED). This can be resolved by adding NSS_INIT_PK11RELOAD to NSS initialization flags.

If the third option of the reproducer influences whether the ldap or curl TLS operation is run first. As you may see from the reproducer output, the bahavior of NSS_INIT_PK11RELOAD is a bit confusing (at least for me). I made no changes in libldap.

So, the question are:
- Which component(s) should we patch?
- Do we have to add NSS_INIT_PK11RELOAD to all libraries which use Mozilla NSS to be sure, that they will work simultaneously? Or can this option be set somehow by the application, which uses the libraries (in this case by mod_php or httpd)? Is there some envvar or something?


I'm adding NSS, CURL and PHP maintainer to CC.

(Btw, I was trying to reproduce/resolve on Fedora 15. The patches will apply on it.)

Comment 24 Jan Vcelak 2011-09-29 13:16:05 UTC
(In reply to comment #23)
> - Do we have to add NSS_INIT_PK11RELOAD to all libraries which use Mozilla NSS
> to be sure, that they will work simultaneously? Or can this option be set
> somehow by the application, which uses the libraries (in this case by mod_php
> or httpd)? Is there some envvar or something?

Elio?

Comment 25 Kamil Dudka 2011-09-29 13:44:27 UTC
Thanks for the patches.  I will have a look at them tomorrow hopefully.

Comment 26 Rich Graves 2011-10-18 16:05:51 UTC
OK, it's tomorrow. Any hope that this will be addressed in the coming 6.2?

We just ran into another conflict with libcurl and nss, documented here:

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPLinuxRH6

Comment 27 Kamil Dudka 2011-10-18 16:39:41 UTC
Sorry for the delay.  The patch looks good.  While testing corner cases of libcurl initialization, I ran into some hard to debug crashes of NSS initialization.  But they seemed to be unrelated to the patch, as they happened on my test-cases with/without the patch applied.  I will try to narrow it down to a minimal example and report separately.

This will need some more testing before applying the patch upstream/el6.  The upstream patch will probably also need a fallback for ancient versions of nss.

Perhaps we should start with reassigning this bug to curl?

Comment 28 Elio Maldonado Batiz 2011-10-18 17:36:08 UTC
NSS 3.13 came out upstream and the bug fixes are enumerated in
https://bugzilla.mozilla.org/buglist.cgi?resolution=FIXED&classification=Components&query_format=advanced&target_milestone=3.13&product=NSS&list_id=1526688
I do see some bug fixes relating to initialization that may be worth reading about for possible relevance to this. In Rawhide we already have updated nss to 3.13. Some retesting with Rawhide may be worthwhile.

Comment 29 Kamil Dudka 2011-11-03 10:26:08 UTC
(In reply to comment #27)
> Perhaps we should start with reassigning this bug to curl?

Nobody seems to object.  I am switching the component to curl in order to get this properly evaluated by Product Management.

Comment 30 Rich Graves 2011-12-07 16:23:25 UTC
Is this a duplicate of bug 747387? I'll try the test case.

Comment 31 Rich Graves 2011-12-07 16:42:36 UTC
No such luck; it's still broken on fully patched RHEL6.2.

./reproducer https://www ldaps://ldap 1 succeeds, ./reproducer https://www ldaps://ldap 0 fails.

Comment 32 Kamil Dudka 2012-01-09 16:31:37 UTC
(In reply to comment #26)
> OK, it's tomorrow. Any hope that this will be addressed in the coming 6.2?

If you want to increase the chance of having the fix for the issue included into RHEL-6 sooner, please contact the product support.  Bugzilla is only a bug tracking tool and RHEL updates are driven by customer requests.

Comment 33 Kamil Dudka 2012-01-11 11:53:07 UTC
Created attachment 552091 [details]
NSS_InitContext for CURL V2

The original patch provided by Jan Vcelak, but rebased against current sources (needs to be applied on top of the patch for bug 772642).

Comment 36 Suzanne Logcher 2012-02-14 23:15:50 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

Comment 46 Miroslav Svoboda 2012-03-14 13:10:18 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
The OpenLDAP suite was recently modified to use NSS instead of OpenSSL as the SSL back end. This change led to collisions between libcurl and OpenLDAP on NSS initialization and shutdown. Consequently, applications that were using both, libcurl and OpenLDAP, failed to establish SSL connections. This update modifies libcurl to use the same NSS API as OpenLDAP, which prevents collisions from occurring. Applications using OpenLDAP and libcurl can now connect to the LDAP server over SSL as expected.

Comment 48 Kamil Dudka 2012-03-19 13:10:20 UTC
Created attachment 571119 [details]
NSS_InitContext for CURL V3

Bob, could you please review the updated patch?  Thanks in advance!

Comment 50 Bob Relyea 2012-03-21 19:01:35 UTC
Comment on attachment 571119 [details]
NSS_InitContext for CURL V3

r+

yes, libcurl certianly should use initcontext.

bob

Comment 54 Kamil Dudka 2012-04-13 10:26:56 UTC
upstream commit:

https://github.com/bagder/curl/commit/20cb12d

Comment 58 errata-xmlrpc 2013-02-21 10:09:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0393.html

Comment 59 Rich Graves 2014-09-26 16:15:19 UTC
This bug seems to have returned in nss-3.16.1-7.el6_5.x86_64.

The April 29 change is suspicious.

* Thu Sep 25 2014 Kai Engert <kaie> - 3.16.1-4
- Adjust softokn patch to be compatible with legacy softokn API.
- Resolves: Bug 1145430 - CVE-2014-1568

* Wed Sep 24 2014 Elio Maldonado <emaldona> - 3.16.1-3
- Add patches published with NSS 3.16.2.1
- Resolves: Bug 1145430 - CVE-2014-1568

* Mon Jun 30 2014 Elio Maldonado <emaldona> - 3.16.1-2
- Backport nss-3.12.6 upstream fix required by Firefox 31 ESR
- Resolves: Bug 1110860

* Tue Jun 24 2014 Elio Maldonado <emaldona> - 3.16.1-1
- Rebase to nss-3.16.1 for FF31
- Resolves: Bug 1110860 - Rebase nss in RHEL 5.11 to NSS 3.16.1, required for FF 31

* Tue Apr 29 2014 Elio Maldonado <emaldona> - 3.15.3-6
- Remove unused and obsolete patches
- Related: Bug 1032468

Comment 60 Elio Maldonado Batiz 2014-09-26 17:16:48 UTC
(In reply to Rich Graves from comment #59)

Rich, I'm a bit comfused by as you start with
> This bug seems to have returned in nss-3.16.1-7.el6_5.x86_64.
> 
> The April 29 change is suspicious.
> 
..... skipped ...
> 
> * Tue Apr 29 2014 Elio Maldonado <emaldona> - 3.15.3-6
> - Remove unused and obsolete patches
> - Related: Bug 1032468

yet the lines you quote above are from the rhel-5.11 spec file.

Elio

Comment 61 Rich Graves 2014-09-26 17:43:06 UTC
Yeah, I copy-pasted from the wrong window. The suspicious entry for rhel6 is

* Tue Jul 08 2014 Elio Maldonado <emaldona> - 3.16.1-5
- Removed listed but unused patches detected by the rpmdiff test
- Resolves: Bug 1099619

According to the application owner here, 3.16.1-4 seems to be good, 3.16.1-7 bad. I will try to better describe "bad."

The test.php code in comment 18 does *not* reproduce the problem. So it's not the same bug. I will open a new bug later if needed.

Comment 62 Rich Graves 2014-09-26 19:48:02 UTC
The verbose PHP error log looks the same as it did for this bug. We will try to create a minimal test case.

ldap_create
ldap_url_parse_ext(ldaps://ldap0.its.carleton.edu/)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ldap0.its.carleton.edu:636
ldap_new_socket: 19
ldap_prepare_socket: 19
ldap_connect_to_host: Trying 137.22.94.105:636
ldap_pvt_connect: fd: 19 tm: 20 async: 0
ldap_ndelay_on: 19
ldap_int_poll: fd: 19 tm: 20
ldap_is_sock_ready: 19
ldap_ndelay_off: 19
ldap_pvt_connect: 0
TLS: could not initialize moznss - error -5925:The one-time function was previously called and failed. Its error code is no longer available.
TLS: could not perform TLS system initialization.
TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available
TLS: can't create ssl handle.
ldap_err2string
ldap_err2string

Comment 63 Craig 2014-10-01 18:29:23 UTC
I have ran into this bug as well, our php ldap thingy suddently stopped working, I thought it was due to other $issues, but running yum downgrade nss* has fixed it.

---> Package nss.x86_64 0:3.16.1-4.el6_5 will be a downgrade
---> Package nss.x86_64 0:3.16.1-7.el6_5 will be erased

The PHP ldap log is:

ldap_create
ldap_extended_operation_s
ldap_extended_operation
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP xxxxx:389
ldap_new_socket: 24
ldap_prepare_socket: 24
ldap_connect_to_host: Trying xxxxx:389
ldap_pvt_connect: fd: 24 tm: -1 async: 0
ldap_open_defconn: successful
ldap_send_server_request
ldap_result ld 0x2b26380 msgid 1
wait4msg ld 0x2b26380 msgid 1 (infinite timeout)
wait4msg continue ld 0x2b26380 msgid 1 all 1
** ld 0x2b26380 Connections:
* host: xxxxx  port: 389  (default)
  refcnt: 2  status: Connected
  last used: Wed Oct  1 19:11:15 2014


** ld 0x2b26380 Outstanding Requests:
 * msgid 1,  origid 1, status InProgress
   outstanding referrals 0, parent count 0
  ld 0x2b26380 request count 1 (abandoned 0)
** ld 0x2b26380 Response Queue:
   Empty
  ld 0x2b26380 response count 0
ldap_chkResponseList ld 0x2b26380 msgid 1 all 1
ldap_chkResponseList returns ld 0x2b26380 NULL
ldap_int_select
read1msg: ld 0x2b26380 msgid 1 all 1
read1msg: ld 0x2b26380 msgid 1 message type extended-result
read1msg: ld 0x2b26380 0 new referrals
read1msg:  mark request completed, ld 0x2b26380 msgid 1
request done: ld 0x2b26380 msgid 1
res_errno: 0, res_error: <Start TLS request accepted.Server willing to negotiate SSL.>, res_matched: <>
ldap_free_request (origid 1, msgid 1)
ldap_parse_extended_result
ldap_parse_result
ldap_msgfree
TLS: error: could not initialize moznss security context - error -5925:The one-time function was previously called and failed. Its error code is no longer available
TLS: can't create ssl handle.
ldap_err2string
ldap_free_connection 1 1
ldap_send_unbind
ldap_free_connection: actually freed
ldap_create
ldap_extended_operation_s