Created attachment 330791 [details] output from the crash Description of problem: When using pycurl in the PackageDB, I eventually get a backtrace from glibc because of a double free. Version-Release number of selected component (if applicable): nspr-4.7.3-2.fc10.i386 nss-3.12.2.0-3.fc10.i386 python-pycurl-7.18.2-1.fc10.i386 curl-7.18.2-9.fc10.i386 python-2.5.2-1.fc10.i386 How reproducible: In my application I can cause a crash over and over. I haven't been able to generate a small test case yet, though. Actual results: Crash that glibc tells me is from a detected Double Free Expected results: Continue to make html requests to retrieve data. Additional info:
Created attachment 330796 [details] Simple test case Okay, Here's a very simple test case that demonstrates how things are breaking. In my testing, the crash occurs consistently on the 67th request.
Created attachment 330799 [details] simple test in C Just to make sure that the problem is in libcurl and not in pycurl, I wrote the test again in C. Same results, curl causes a double free on the 67th invocation.
seems to be related to Bug 483222
(In reply to comment #3) > seems to be related to Bug 483222 Could be. Do you know if git would be opening 67 curl objects in its lifetime? (BTW, if git doesn't have to worry about thread safety, you might be able to workaround this and increase performance by only opening a single curl connection and making all requests through that.)
Testing with the simple test case found that: 7.18.2 built from our SRPM without any of our patches is affected. 7.19.3 built from our SRPM without any of our patches is affected. RHEL5's curl: curl-7.15.5-2.el5 is not affected. Changing the configure options to --with-ssl=%{prefix} --without-nss:: 7.19.3 built from our SRPM without any of our patches is not affected 7.18.2 with our patches is not affected So -- this problem affects upstream as well. It is linked to either nss or curl's use of nss, same as Bug 483222.
Well, it's more likely a NSS/NSPR fault I presume: 67 *** glibc detected *** ./a.out: double free or corruption (!prev): 0x0c7222a0 *** ======= Backtrace: ========= /lib/libc.so.6[0x3ba3a4] /lib/libc.so.6(cfree+0x96)[0x3bc356] /lib/libnspr4.so(PR_Free+0x27)[0x32f6b37] /lib/libnsspem.so[0x2022f8] /lib/libnsspem.so[0x1f079b] /lib/libnsspem.so[0x1f23ce] /lib/libnsspem.so[0x1f67f4] /lib/libnsspem.so[0x1fe2c9] /lib/libnsspem.so[0x1ed4ac] /lib/libnss3.so[0x336aede] /lib/libnss3.so(PK11_CreateGenericObject+0x50)[0x336b160] /usr/lib/libcurl.so.4[0x1dc9a3] /usr/lib/libcurl.so.4(Curl_nss_connect+0x747)[0x1dd3b7] /usr/lib/libcurl.so.4(Curl_ssl_connect+0x37)[0x1d5017] /usr/lib/libcurl.so.4(Curl_http_connect+0xa1)[0x1b4de1] /usr/lib/libcurl.so.4(Curl_protocol_connect+0xe9)[0x1bbda9] /usr/lib/libcurl.so.4(Curl_connect+0x314)[0x1bedc4] /usr/lib/libcurl.so.4(Curl_perform+0x99)[0x1c9609] /usr/lib/libcurl.so.4(curl_easy_perform+0x43)[0x1ca263] ./a.out[0x80486d3] I had the same failure with either curl's downstream patches applied or not. Reassigning to NSS to get more info. Feel free to assign back or to nspr if needed.
Created attachment 331111 [details] program causing error within nss Indeed. Attaching minimal example. The output is: ... 61 62 63 ==24187== Invalid write of size 4 ==24187== at 0x419E7BF: pem_CreateObject (pobject.c:1080) ==24187== by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156) ==24187== by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353) ==24187== by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991) ==24187== by 0x419B4AB: pemC_CreateObject (nssck.api:566) ==24187== by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412) ==24187== by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346) ==24187== by 0x804873A: main (in /tmp/curl/a.out) ==24187== Address 0x8a2d578 is 0 bytes after a block of size 63,496 alloc'd ==24187== at 0x4004BA2: calloc (vg_replace_malloc.c:397) ==24187== by 0x703E039: PR_Calloc (in /lib/libnspr4.so) ==24187== by 0x41B0297: nss_ZRealloc (arena.c:1056) ==24187== by 0x419E79A: pem_CreateObject (pobject.c:1080) ==24187== by 0x41A03CD: pem_mdSession_CreateObject (psession.c:156) ==24187== by 0x41A47F3: nssCKFWSession_CreateObject (session.c:1353) ==24187== by 0x41AC2C8: NSSCKFWC_CreateObject (wrap.c:1991) ==24187== by 0x419B4AB: pemC_CreateObject (nssck.api:566) ==24187== by 0x4095ECD: PK11_CreateNewObject (pk11obj.c:412) ==24187== by 0x409614F: PK11_CreateGenericObject (pk11obj.c:1346) ==24187== by 0x804873A: main (in /tmp/curl/a.out)
Created attachment 333153 [details] proposed patch It's bug in pem/pobject.c, kind of +/-1 problem. Fix attached.
*** Bug 483222 has been marked as a duplicate of this bug. ***
A rebuilt nss with the supplied patch no longer crashes. However, it either causes or allows a memory leak to surface. Running any of the three test programs runs my 1GB machine into swap. The simple curl test cases run me into swap before we hit 200 iterations. The nss test case before we hit 2000 iterations.
I am sure there are no leaks caused by the patch. Looking at the nss test, there is no stop condition, no free, no destroy, no shutdown, ... The test was intended to be minimal, not to free all resources. If you think there is a leak within nss or curl please open separate bug for it. Slightly raising priority as the bug is obvious and patch is simple.
Created attachment 333603 [details] same fix but patch leads to smaller code
The new patch works here as well. Something was not right in how I tested for the memory leak before... I no longer leak memory in the C curl test so it seems like an issue with the python bindings. I'll report a bug there after I gather more info.
reported upstream: https://bugzilla.mozilla.org/show_bug.cgi?id=482678
reported again, now the right place hopefully :-) https://bugzilla.mozilla.org/show_bug.cgi?id=402712
Ready to patch Fedora?
Kamil, thanks for producing the patch and getting it reviewed upstream.
I produced a new NSS build with the patch applied. It's not in the update channel yet but can be downloaded from this page: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268901 For example click i386 which brings you to: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268907 Which has links to the .rpm files. Could you please test this build and report whether it fixes your problem? Thanks! ... the above build is for fedora 10 f9 build: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268909 rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=1268884
Tested. The build fixes the crash. The memory leak is a separate issue that Kamil is talking to upstream about.
this fixed Version NSS solved my problem with a error: *** glibc detected *** python: double free or corruption (!prev): 0x0af39ee0 *** from the script https://fedorahosted.org/fama/browser/fas-group-inactive.py
nss-3.12.2.0-5.fc10 has been submitted as an update for Fedora 10. http://admin.fedoraproject.org/updates/nss-3.12.2.0-5.fc10
nss-3.12.2.0-3.fc9 has been submitted as an update for Fedora 9. http://admin.fedoraproject.org/updates/nss-3.12.2.0-3.fc9
Thanks for the test feedback. I've pushed the updated packages to the fedora update system.
nss-3.12.2.0-5.fc10 has been pushed to the Fedora 10 stable repository. If problems still persist, please make note of it in this bug report.
nss-3.12.2.0-3.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.
I would be very grateful if someone could build and publish this update to nss for Fedora 8. If it's already been done, then perhaps someone could tell me where to find it? Thank you.
Fedora 8 is no longer supported, so there are AFAIK no updates. Does a scratch build do the job for you?
Bryan, as Kamil said, we are no longer able to build updates for you. Get the Fedora 8 source RPM, find documentation about building from a source RPM, add the patch we added in Fedora 9 and above, and build it. Patch is attached in this bug, attachment 333603 [details]
http://fedoraproject.org/wiki/LifeCycle http://lwn.net/Articles/308913/
Well, considering severity of the bug I've made the scratch build anyway: http://koji.fedoraproject.org/koji/taskinfo?taskID=1364823
Kamil, thanks very much for building this update to nss for FC8. I've installed the binary RPM and it looks like it fixed the problem for me. Thank you again, Bryan
nss-3.12.4-2.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/nss-3.12.4-2.fc11
nss-3.12.4-3.fc11 has been submitted as an update for Fedora 11. http://admin.fedoraproject.org/updates/nss-3.12.4-3.fc11?_csrf_token=d6eee54945d598c83d89d6481b2a20cf972bc76b