Created attachment 343771 [details] Backtrace 7.18 (sorry for my english) From about 5-6 month from version 7.18 or maybe earlier to 7.19.4-5 there is a bug into curl linked with libnss of memory leak I has a my application that use libcurl with TLS/SSL features. With the curl 7.19.4-5 linked to libnsss, the memory of my program grow untill all memory is filled. A workaround of the bug is recompile the original curl and libcurl linked with OpenSSL. The bug is still present because, after yum update to 7.19.4-5 (that has overwritten my compiled curl) my application has restarted to has memory leak until i has recompiled curl. I has extrapolated this code (5-6 month ago ???) from my application to reproduce the bug into libcurl (7.18 ???), i has not tested if reproduce the bug also into 7.19, but there are an high probability that this code reproduce the bug of memory leak also into 7.19.4-5 like my application does. I has also attached the backtrace of the 7.18 and reported the little example used to reproduce the problem into 7.18 (and probably 7.19, not tested) #include <iostream> #include <curl/curl.h> #include <gtk/gtk.h> size_t NoProcess(void *buffer, size_t size, size_t nmemb, void *userp) { return size * nmemb; } gpointer Test(gpointer data) { CURL * Fin; Fin = curl_easy_init(); // if (curl_easy_setopt(Fin, CURLOPT_NOSIGNAL, 1) != CURLE_OK) return NULL; while (1) { if (curl_easy_setopt(Fin, CURLOPT_WRITEDATA, NULL) != CURLE_OK) return NULL; if (curl_easy_setopt(Fin, CURLOPT_WRITEFUNCTION, NoProcess) != CURLE_OK) return NULL; if (curl_easy_setopt(Fin, CURLOPT_URL, "https://www.fineco.it/fineco/jsp/login/content.jsp") != CURLE_OK) return NULL; if (curl_easy_perform(Fin) != CURLE_OK) return NULL; sleep(1); } return NULL; } int main() { curl_global_init(CURL_GLOBAL_ALL); g_thread_init(NULL); g_thread_create(Test,NULL,false,NULL); while (1) { sleep(1); } curl_global_cleanup(); return 0; } This code continue to read an https page, after a lot of iteration the memory should grow after some day or may be some hours the code use all memory of the PC and probably crash ( Is an extimation based on the behavior based on my application (NOT TESTED with reported code), into 7.18 the code crash after 7 - 8 hour of running with backtrace attached (TESTED with reported code) ) At this moment to use curl with my application i need to recompile libcurl from source and link libcurl to OpenSSL
I has founded my report to Curl mail list Sat, 29 Nov 2008 16:52:34 +0100 http://curl.haxx.se/mail/lib-2008-11/0276.html Daniel Stenberg has answered in the next message The code reported was used to reproduce the bug with Fedore Core 8 curl and libcurl 7.18.2-6.fc8 so a little old but i has also founded this 8 April 2009 http://www.mail-archive.com/curl-library@cool.haxx.se/msg00470.html use a code very similar to my example code to reproduce the bug and the problem is present into libcurl-7.19.4-3.fc10.i386 I has not tested the reported code with libcurl-7.19.4-5.fc10.i386, but there is an high probably that this release don' t solve the problem, because my application (from which I discover the bug ) continue to has memory leak with 7.19.4-5, untill i don' t recompile libcurl from source downloaded from the original project (linked with OpenSSL)
Created attachment 343790 [details] simplified test case Are the two threads necessary to reproduce the leak? Could you please test the attached code on your system?
Are the two threads necessary to reproduce the leak? Probably no Could you please test the attached code on your system? OK, but on a different machine because i need a working libcurl and i has already recompiled and overwritten the not working fc10 libcurl
(In reply to comment #3) > OK, but on a different machine because i need a working libcurl > and i has already recompiled and overwritten the not working fc10 libcurl Then please do and make sure you have up to date versions of libcurl and nss: libcurl-7.19.4-5.fc10 nss-3.12.2.0-5.fc10
I realized we have fixed some memory leaks on rawhide and the patches have not been applied on Fedora 10 yet. You can test the following scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=1353469
I confirm the memory leak i am using test.c Fedora core 10 64 bit and libcurl 64 bit version test is linked to /usr/lib64/libcurl.so.4 I am not doing deep test, but monitoring with top after few minutes test was using 0.2% memory after 1 hour has grown to 0.6% 3x times after 2 hour has grown to 1.2% 6x times I continue to monitor
Ok i stop the program and test the new build
updated to 7.19.4-11 results: 0.1% on start 0.6% after 1 hour don' t seem to solve I continue to monitor
1.1% after 2 hour 1.6% after 3 hour nss is 3.12.2.0-5 Stopped because i don't find the reason to continue the memory leak is clear in my opinion
The amount of used memory is constant for hours with the same test case and OpenSSL-enabled libcurl? Please also report the version of libcurl you've compiled from sources.
curl 7.19.4 (i686-pc-linux-gnu) libcurl/7.19.4 OpenSSL/0.9.8g zlib/1.2.3 libidn/0.6.14 libssh2/0.18 Protocols: tftp ftp telnet dict http file https ftps scp sftp Features: IDN IPv6 Largefile NTLM SSL libz The amount of used memory is constant for hours with the same test case and OpenSSL-enabled libcurl? I will test but, Should be constant because i use a more complex my appliucation that use TLS/SSL and curl in a similar way ( and frequency )of the test. today are 4 day running and is using 0.7% of memory ( and the PC where is running my application has 1GB memory , PC where test is running has 4GB of memory )
Compiled and installed curl --version is the same as reported up with the difference (x86_64-unknown-linux-gnu) compiled test with gcc -o test -Wl,-R/usr/local/lib test.c -lcurl linked to /usr/local/lib/libcurl.so.4 (compiled source) Results of test 0.1% on start 0.1% after 1 hour 0.1% after 2 hour .......
Thanks for testing it! It'll take some time to catch the leaks as there are AFAIK not caused by the (lib)curl code itself.
The problem is more complex than we initially expected to. We are now working on review of a few patches for the nss package and it will take some time before the update will be ready. If you are interested in testing not yet approved packages of fixed curl/nss, here they are: https://bugzilla.redhat.com/show_bug.cgi?id=501138#c6 Please do not test them on a production machine.
(In reply to comment #14) > https://bugzilla.redhat.com/show_bug.cgi?id=501138#c6 Now we have have all the patches at least once reviewed -- the scratch builds should be stable enough. Please, have a go at that. Note one patch has not been applied yet, but it doesn't affect this bug (you have no private key in the test case).
Ok thanks, i will download and test in this weekend, if work i will return to libcurl + nss
Have you tried it? Any new experiences?
Ops.... at the moment i am using curl with OpenSSL, i will test as soon as possible.
I has updated the test machine to FC11 now i has nss 3.12.3.99.3-2.11.3.fc11 and curl 7.19.4-9.fc11 and libcurl 7.19.4-9.fc11 using the simple test case, the memory continue to grow. On my test machine has memory leak On the others machines i continue to use OpenSSL Waiting for other tests if needed
Thanks for the feedback! Could you please try it again with the patches for NSS from bug #509705? Let me know if a scratch build of NSS can help you with testing.
Where i can find the fedora NSS git sources to apply the paches ?
You can download SRPM of the latest build from Koji: $ curl -O http://kojipkgs.fedoraproject.org/packages/nss/3.12.3.99.3/2.11.3.fc11/src/nss-3.12.3.99.3-2.11.3.fc11.src.rpm Then extract the sources: $ rpm2cpio nss-3.12.3.99.3-2.11.3.fc11.src.rpm | cpio -idv Do whatever you want ... and finally build by: $ rpmbuild -bb nss.spec
Created attachment 357448 [details] bundle of fixes for nss/pem which are not included in stable release Elio, could you please prepare the F-11 scratch build for testing. I've tried it but it died with: error: File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/lib64/pkgconfig/nss.pc error: File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/bin/nss-config RPM build errors: File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/lib64/pkgconfig/nss.pc File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/bin/nss-config The patch against F-11 stable is attached. Thanks in advance!
What i has done, probably something wrong rpm2cpio nss-3.12.3.99.3-2.11.3.fc11.src.rpm | cpio -idv tar -xjf nss-pem-20090622.tar.bz2 created a dir nss-3.12.3.99.3 inserted folder mozilla applied patch patch -p0 < nss-pem-bz500636.patch recompress tar cjfv nss-pem-20090622.tar.bz2 mozilla/ rpmbuild -bb nss.spec Tests summary: -------------- Passed: 5474 Failed: 3 Failed with core: 0 Unknown status: 0 + cd ../../../../ + killall selfserv_9829 selfserv_9829: no process killed + : ++ grep -c FAILED ./mozilla/tests_results/security/localhost.1/output.log + TEST_FAILURES=3 + '[' 3 -ne 0 ']' + echo 'error: test suite returned failure(s)' error: test suite returned failure(s) + exit 1 error: Bad exit status from /var/tmp/rpm-tmp.SXtOHT (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.SXtOHT (%build) No Build package into RPMS folder Probably for the 3 test failed ? Failed test #1910: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot #3403: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot #5467: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot
(In reply to comment #24) > No Build package into RPMS folder > > Probably for the 3 test failed ? Yes, that might be the reason. You can disable the test suite by inserting 'exit 0' line before the line '# Run test suite.'. But then please do not install the package on your system. Just build it locally using 'rpmbuild -bc' and use the just-built libnsspem.so. You can force its loading by exporting LD_PRELOAD=absolute_path/libnsspem.so while running your tests. And of course you don't need to patch the sources manually, rpm will do the job for you. Add the two following lines to .spec to the corresponding sections: Patch8: nss-pem-bz500636.patch %patch8 -p1 Thanks in advance for testing!
Like your comment 23 Processing files: nss-devel-3.12.3.99.3-2.11.3.fc11.x86_64 error: File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/lib64/pkgconfig/nss.pc error: File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/bin/nss-config RPM build errors: File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/lib64/pkgconfig/nss.pc File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/bin/nss-config
The latest rpm build tools exposed latent bugs in nss.spec. They are now fixed See http://koji.fedoraproject.org/koji/buildinfo?buildID=127403
Elio, thanks for the speedy fix! Pietro, you can download the scratch build with the patches applied here: http://koji.fedoraproject.org/koji/taskinfo?taskID=1609106 Please test along with latest F-11 (lib)curl from updates-testing: http://koji.fedoraproject.org/koji/buildinfo?buildID=127267
Memory leak seem to be solved testing now is on start 0.1% after 10 minutes it grow to 0.2% after 1h:30 is fixed to 0.2%
nss-3.12.4-3.fc11 has been pushed to the Fedora 11 stable repository. If problems still persist, please make note of it in this bug report.