This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 483855 - Creating more than 66 easy curl handles causes a double free
Creating more than 66 easy curl handles causes a double free
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: nss (Show other bugs)
10
All Linux
high Severity high
: ---
: ---
Assigned To: Kai Engert (:kaie)
Fedora Extras Quality Assurance
:
: 483222 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-02-03 17:47 EST by Toshio Ernie Kuratomi
Modified: 2009-09-17 18:03 EDT (History)
9 users (show)

See Also:
Fixed In Version: 3.12.2.0-3.fc9
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-04-13 15:37:24 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
output from the crash (2.90 KB, text/plain)
2009-02-03 17:47 EST, Toshio Ernie Kuratomi
no flags Details
Simple test case (304 bytes, text/x-python)
2009-02-03 18:26 EST, Toshio Ernie Kuratomi
no flags Details
simple test in C (527 bytes, text/plain)
2009-02-03 18:40 EST, Toshio Ernie Kuratomi
no flags Details
program causing error within nss (1.19 KB, text/plain)
2009-02-06 05:30 EST, Kamil Dudka
no flags Details
proposed patch (567 bytes, patch)
2009-02-25 06:54 EST, Kamil Dudka
no flags Details | Diff
same fix but patch leads to smaller code (582 bytes, patch)
2009-02-28 09:33 EST, Kamil Dudka
no flags Details | Diff

  None (edit)
Description Toshio Ernie Kuratomi 2009-02-03 17:47:36 EST
Created attachment 330791 [details]
output from the crash

Description of problem:
When using pycurl in the PackageDB, I eventually get a backtrace from glibc because of a double free.

Version-Release number of selected component (if applicable):
nspr-4.7.3-2.fc10.i386
nss-3.12.2.0-3.fc10.i386
python-pycurl-7.18.2-1.fc10.i386
curl-7.18.2-9.fc10.i386
python-2.5.2-1.fc10.i386

How reproducible:
In my application I can cause a crash over and over.  I haven't been able to generate a small test case yet, though.

  
Actual results:
Crash that glibc tells me is from a detected Double Free

Expected results:
Continue to make html requests to retrieve data.

Additional info:
Comment 1 Toshio Ernie Kuratomi 2009-02-03 18:26:11 EST
Created attachment 330796 [details]
Simple test case

Okay, Here's a very simple test case that demonstrates how things are breaking.

In my testing, the crash occurs consistently on the 67th request.
Comment 2 Toshio Ernie Kuratomi 2009-02-03 18:40:42 EST
Created attachment 330799 [details]
simple test in C

Just to make sure that the problem is in libcurl and not in pycurl, I wrote the test again in C.  Same results, curl causes a double free on the 67th invocation.
Comment 3 Kamil Dudka 2009-02-04 09:19:47 EST
seems to be related to Bug 483222
Comment 4 Toshio Ernie Kuratomi 2009-02-04 12:06:03 EST
(In reply to comment #3)
> seems to be related to Bug 483222

Could be. Do you know if git would be opening 67 curl objects in its lifetime?
(BTW, if git doesn't have to worry about thread safety, you might be able to workaround this and increase performance by only opening a single curl connection and making all requests through that.)
Comment 5 Toshio Ernie Kuratomi 2009-02-04 12:43:28 EST
Testing with the simple test case found that: 

7.18.2 built from our SRPM without any of our patches is affected.
7.19.3 built from our SRPM without any of our patches is affected.

RHEL5's curl: curl-7.15.5-2.el5 is not affected.

Changing the configure options to --with-ssl=%{prefix} --without-nss::
    7.19.3 built from our SRPM without any of our patches is not affected
    7.18.2 with our patches is not affected

So -- this problem affects upstream as well.  It is linked to either nss or curl's use of nss, same as Bug 483222.
Comment 6 Jindrich Novy 2009-02-04 13:02:10 EST
Well, it's more likely a NSS/NSPR fault I presume:

67 *** glibc detected *** ./a.out: double free or corruption (!prev): 0x0c7222a0 ***
======= Backtrace: =========
/lib/libc.so.6[0x3ba3a4]
/lib/libc.so.6(cfree+0x96)[0x3bc356]
/lib/libnspr4.so(PR_Free+0x27)[0x32f6b37]
/lib/libnsspem.so[0x2022f8]
/lib/libnsspem.so[0x1f079b]
/lib/libnsspem.so[0x1f23ce]
/lib/libnsspem.so[0x1f67f4]
/lib/libnsspem.so[0x1fe2c9]
/lib/libnsspem.so[0x1ed4ac]
/lib/libnss3.so[0x336aede]
/lib/libnss3.so(PK11_CreateGenericObject+0x50)[0x336b160]
/usr/lib/libcurl.so.4[0x1dc9a3]
/usr/lib/libcurl.so.4(Curl_nss_connect+0x747)[0x1dd3b7]
/usr/lib/libcurl.so.4(Curl_ssl_connect+0x37)[0x1d5017]
/usr/lib/libcurl.so.4(Curl_http_connect+0xa1)[0x1b4de1]
/usr/lib/libcurl.so.4(Curl_protocol_connect+0xe9)[0x1bbda9]
/usr/lib/libcurl.so.4(Curl_connect+0x314)[0x1bedc4]
/usr/lib/libcurl.so.4(Curl_perform+0x99)[0x1c9609]
/usr/lib/libcurl.so.4(curl_easy_perform+0x43)[0x1ca263]
./a.out[0x80486d3]

I had the same failure with either curl's downstream patches applied or not. Reassigning to NSS to get more info. Feel free to assign back or to nspr if needed.
Comment 7 Kamil Dudka 2009-02-06 05:30:35 EST
Created attachment 331111 [details]
program causing error within nss

Indeed. Attaching minimal example. The output is:
...
61
62
63
==24187== Invalid write of size 4
==24187==    at 0x419E7BF: pem_CreateObject (pobject.c:1080)
==24187==    by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156)
==24187==    by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353)
==24187==    by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991)
==24187==    by 0x419B4AB: pemC_CreateObject (nssck.api:566)
==24187==    by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412)
==24187==    by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346)
==24187==    by 0x804873A: main (in /tmp/curl/a.out)
==24187==  Address 0x8a2d578 is 0 bytes after a block of size 63,496 alloc'd
==24187==    at 0x4004BA2: calloc (vg_replace_malloc.c:397)
==24187==    by 0x703E039: PR_Calloc (in /lib/libnspr4.so)
==24187==    by 0x41B0297: nss_ZRealloc (arena.c:1056)
==24187==    by 0x419E79A: pem_CreateObject (pobject.c:1080)
==24187==    by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156)
==24187==    by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353)
==24187==    by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991)
==24187==    by 0x419B4AB: pemC_CreateObject (nssck.api:566)
==24187==    by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412)
==24187==    by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346)
==24187==    by 0x804873A: main (in /tmp/curl/a.out)
Comment 8 Kamil Dudka 2009-02-25 06:54:43 EST
Created attachment 333153 [details]
proposed patch

It's bug in pem/pobject.c, kind of +/-1 problem. Fix attached.
Comment 9 Kamil Dudka 2009-02-25 06:55:56 EST
*** Bug 483222 has been marked as a duplicate of this bug. ***
Comment 10 Toshio Ernie Kuratomi 2009-02-25 11:40:42 EST
A rebuilt nss with the supplied patch no longer crashes.  However, it either causes or allows a memory leak to surface.  Running any of the three test programs runs my 1GB machine into swap.

The simple curl test cases run me into swap before we hit 200 iterations.  The nss test case before we hit 2000 iterations.
Comment 11 Kamil Dudka 2009-02-26 05:44:04 EST
I am sure there are no leaks caused by the patch. Looking at the nss test, there is no stop condition, no free, no destroy, no shutdown, ... The test was intended to be minimal, not to free all resources. If you think there is a leak within nss or curl please open separate bug for it.

Slightly raising priority as the bug is obvious and patch is simple.
Comment 12 Kamil Dudka 2009-02-28 09:33:52 EST
Created attachment 333603 [details]
same fix but patch leads to smaller code
Comment 13 Toshio Ernie Kuratomi 2009-03-05 11:32:50 EST
The new patch works here as well.  Something was not right in how I tested for the memory leak before... I no longer leak memory in the C curl test so it seems like an issue with the python bindings.  I'll report a bug there after I gather more info.
Comment 14 Kamil Dudka 2009-03-11 06:34:58 EDT
reported upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=482678
Comment 15 Kamil Dudka 2009-03-11 13:15:53 EDT
reported again, now the right place hopefully :-)
https://bugzilla.mozilla.org/show_bug.cgi?id=402712
Comment 16 Kamil Dudka 2009-03-16 05:07:07 EDT
Ready to patch Fedora?
Comment 17 Kai Engert (:kaie) 2009-03-31 19:01:42 EDT
Kamil, thanks for producing the patch and getting it reviewed upstream.
Comment 18 Kai Engert (:kaie) 2009-04-01 09:23:37 EDT
I produced a new NSS build with the patch applied. It's not in the update channel yet but can be downloaded from this page:

https://koji.fedoraproject.org/koji/taskinfo?taskID=1268901

For example click i386 which brings you to:
https://koji.fedoraproject.org/koji/taskinfo?taskID=1268907

Which has links to the .rpm files.

Could you please test this build and report whether it fixes your problem?
Thanks!

...
the above build is for fedora 10
f9 build: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268909
rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268884
Comment 19 Toshio Ernie Kuratomi 2009-04-06 17:31:47 EDT
Tested.  The build fixes the crash.  The memory leak is a separate issue that Kamil is talking to upstream about.
Comment 20 Joerg (kital) Simon 2009-04-09 13:54:53 EDT
this fixed Version NSS solved my problem with a 
error:
*** glibc detected *** python: double free or corruption (!prev): 0x0af39ee0 ***
from the script https://fedorahosted.org/fama/browser/fas-group-inactive.py
Comment 21 Fedora Update System 2009-04-09 21:21:11 EDT
nss-3.12.2.0-5.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/nss-3.12.2.0-5.fc10
Comment 22 Fedora Update System 2009-04-09 21:21:17 EDT
nss-3.12.2.0-3.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/nss-3.12.2.0-3.fc9
Comment 23 Kai Engert (:kaie) 2009-04-09 21:23:02 EDT
Thanks for the test feedback.

I've pushed the updated packages to the fedora update system.
Comment 24 Fedora Update System 2009-04-13 15:37:18 EDT
nss-3.12.2.0-5.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 25 Fedora Update System 2009-04-13 15:44:11 EDT
nss-3.12.2.0-3.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 26 Bryan Field-Elliot 2009-05-18 16:15:56 EDT
I would be very grateful if someone could build and publish this update to nss for Fedora 8. If it's already been done, then perhaps someone could tell me where to find it? Thank you.
Comment 27 Kamil Dudka 2009-05-18 16:32:08 EDT
Fedora 8 is no longer supported, so there are AFAIK no updates. Does a scratch build do the job for you?
Comment 28 Kai Engert (:kaie) 2009-05-19 17:55:50 EDT
Bryan, as Kamil said, we are no longer able to build updates for you.

Get the Fedora 8 source RPM, find documentation about building from a source RPM, add the patch we added in Fedora 9 and above, and build it.

Patch is attached in this bug, attachment 333603 [details]
Comment 30 Kamil Dudka 2009-05-19 18:55:38 EDT
Well, considering severity of the bug I've made the scratch build anyway:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1364823
Comment 31 Bryan Field-Elliot 2009-05-23 12:14:46 EDT
Kamil, thanks very much for building this update to nss for FC8. I've installed the binary RPM and it looks like it fixed the problem for me. 

Thank you again,

Bryan
Comment 32 Fedora Update System 2009-09-16 00:09:04 EDT
nss-3.12.4-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/nss-3.12.4-2.fc11
Comment 33 Fedora Update System 2009-09-17 18:03:35 EDT
nss-3.12.4-3.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/nss-3.12.4-3.fc11?_csrf_token=d6eee54945d598c83d89d6481b2a20cf972bc76b

Note You need to log in before you can comment on or make changes to this bug.