Bug 500636 - libcurl 7.19.4-5 linked with libnss has memory leak
libcurl 7.19.4-5 linked with libnss has memory leak
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: nss (Show other bugs)
11
All Linux
low Severity high
: ---
: ---
Assigned To: Elio Maldonado Batiz
Fedora Extras Quality Assurance
:
Depends On: 509705
Blocks: 517000
  Show dependency treegraph
 
Reported: 2009-05-13 10:07 EDT by Pietro Incardona
Modified: 2010-01-14 03:41 EST (History)
3 users (show)

See Also:
Fixed In Version: 3.12.4-3.fc11
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-24 02:53:03 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Backtrace 7.18 (23.10 KB, application/octet-stream)
2009-05-13 10:07 EDT, Pietro Incardona
no flags Details
simplified test case (925 bytes, application/octet-stream)
2009-05-13 11:28 EDT, Kamil Dudka
no flags Details
bundle of fixes for nss/pem which are not included in stable release (28.01 KB, patch)
2009-08-14 08:16 EDT, Kamil Dudka
no flags Details | Diff

  None (edit)
Description Pietro Incardona 2009-05-13 10:07:38 EDT
Created attachment 343771 [details]
Backtrace 7.18

(sorry for my english)
From about 5-6 month from version 7.18 or maybe earlier to 7.19.4-5 there is 
a bug into curl linked with libnss of memory leak


I has a my application that use libcurl with TLS/SSL features. With the curl
7.19.4-5 linked to libnsss, the memory of my program grow untill all 
memory is filled.
A workaround of the bug is recompile the original curl and libcurl linked with
OpenSSL. The bug is still present because, after yum update to 7.19.4-5 (that has overwritten my compiled curl) my application has restarted to has memory leak until i has recompiled curl.

I has extrapolated this code (5-6 month ago ???) from my application to
reproduce the bug into libcurl (7.18 ???), i has not tested if reproduce the bug
also into 7.19, but there are an high probability that this code reproduce the
bug of memory leak also into 7.19.4-5 like my application does.

I has also attached the backtrace of the 7.18

and reported the little example used to reproduce the problem into 7.18 (and
probably 7.19, not tested)

#include <iostream>
#include <curl/curl.h>
#include <gtk/gtk.h>



size_t NoProcess(void *buffer, size_t size, size_t nmemb, void *userp)
{
 return size * nmemb;
}

gpointer Test(gpointer data)
{
 CURL * Fin;

 Fin = curl_easy_init();

// if (curl_easy_setopt(Fin, CURLOPT_NOSIGNAL, 1) != CURLE_OK) return NULL;

 while (1)
 {
  if (curl_easy_setopt(Fin, CURLOPT_WRITEDATA, NULL) != CURLE_OK) return NULL;
  if (curl_easy_setopt(Fin, CURLOPT_WRITEFUNCTION, NoProcess) != CURLE_OK)
return NULL;
  if (curl_easy_setopt(Fin, CURLOPT_URL,
"https://www.fineco.it/fineco/jsp/login/content.jsp") != CURLE_OK) return NULL;


  if (curl_easy_perform(Fin) != CURLE_OK) return NULL;

  sleep(1);
 }

 return NULL;
}

int main()
{
 curl_global_init(CURL_GLOBAL_ALL);
 g_thread_init(NULL);

 g_thread_create(Test,NULL,false,NULL);

 while (1) 
 {
  sleep(1);
 }

 curl_global_cleanup();
 return 0;
} 

This code continue to read an https page, after a lot of iteration
the memory should grow after some day or may be some hours the code use all memory of the PC and probably crash
( Is an extimation based on the behavior based on my application (NOT TESTED
with reported code), into 7.18 the code crash after 7 - 8 hour of running with
backtrace attached (TESTED with reported code) )

At this moment to use curl with my application i need to recompile libcurl from
source and link libcurl to OpenSSL
Comment 1 Pietro Incardona 2009-05-13 11:27:52 EDT
I has founded my report to Curl mail list

Sat, 29 Nov 2008 16:52:34 +0100
http://curl.haxx.se/mail/lib-2008-11/0276.html

Daniel Stenberg has answered in the next message

The code reported was used to reproduce the bug with

Fedore Core 8
curl and libcurl 7.18.2-6.fc8


so a little old but
i has also founded this

8 April 2009
http://www.mail-archive.com/curl-library@cool.haxx.se/msg00470.html

use a code very similar to my example code to reproduce the bug and 
the problem is present into 

libcurl-7.19.4-3.fc10.i386

I has not tested the reported code with
libcurl-7.19.4-5.fc10.i386, but there is an high probably that this release
don' t solve the problem, because my application (from which I discover the bug ) continue to has memory leak with 7.19.4-5, untill i don' t recompile libcurl from source downloaded from the original project (linked with OpenSSL)
Comment 2 Kamil Dudka 2009-05-13 11:28:37 EDT
Created attachment 343790 [details]
simplified test case

Are the two threads necessary to reproduce the leak? Could you please test the attached code on your system?
Comment 3 Pietro Incardona 2009-05-13 13:18:56 EDT
Are the two threads necessary to reproduce the leak?

Probably no

Could you please test the attached code on your system?

OK, but on a different machine because i need a working libcurl
and i has already recompiled and overwritten the not working fc10 libcurl
Comment 4 Kamil Dudka 2009-05-13 13:41:16 EDT
(In reply to comment #3)
> OK, but on a different machine because i need a working libcurl
> and i has already recompiled and overwritten the not working fc10 libcurl  

Then please do and make sure you have up to date versions of libcurl and nss:

libcurl-7.19.4-5.fc10
nss-3.12.2.0-5.fc10
Comment 5 Kamil Dudka 2009-05-13 15:12:35 EDT
I realized we have fixed some memory leaks on rawhide and the patches have not been applied on Fedora 10 yet. You can test the following scratch build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1353469
Comment 6 Pietro Incardona 2009-05-13 15:53:04 EDT
I confirm the memory leak

i am using test.c

Fedora core 10 64 bit and libcurl 64 bit version 

test is linked to /usr/lib64/libcurl.so.4

I am not doing deep test, but monitoring with top

after few minutes test was using 0.2% memory

after 1 hour has grown to 0.6% 3x times
after 2 hour has grown to 1.2% 6x times

I continue to monitor
Comment 7 Pietro Incardona 2009-05-13 15:55:54 EDT
Ok i stop the program and test the new build
Comment 8 Pietro Incardona 2009-05-13 17:39:50 EDT
updated to 7.19.4-11

results:

0.1% on start
0.6% after 1 hour

don' t seem to solve

I continue to monitor
Comment 9 Pietro Incardona 2009-05-13 19:48:04 EDT
1.1% after 2 hour
1.6% after 3 hour

nss is 3.12.2.0-5

Stopped because i don't find the reason to continue

the memory leak is clear in my opinion
Comment 10 Kamil Dudka 2009-05-14 02:18:12 EDT
The amount of used memory is constant for hours with the same test case and OpenSSL-enabled libcurl? Please also report the version of libcurl you've compiled from sources.
Comment 11 Pietro Incardona 2009-05-14 04:37:07 EDT
curl 7.19.4 (i686-pc-linux-gnu) libcurl/7.19.4 OpenSSL/0.9.8g zlib/1.2.3 libidn/0.6.14 libssh2/0.18
Protocols: tftp ftp telnet dict http file https ftps scp sftp 
Features: IDN IPv6 Largefile NTLM SSL libz 

The amount of used memory is constant for hours with the same test case and
OpenSSL-enabled libcurl?

I will test but,
Should be constant because i use a more complex my appliucation that use 
TLS/SSL and curl in a similar way ( and frequency )of the test. today are 4 day running and is using 0.7% of memory ( and the PC where is running my application has 1GB memory , PC where test is running has 4GB of memory )
Comment 12 Pietro Incardona 2009-05-14 07:55:28 EDT
Compiled and installed

curl --version is the same as reported up with the difference
(x86_64-unknown-linux-gnu)

compiled test with

gcc -o test -Wl,-R/usr/local/lib test.c -lcurl

linked to 

/usr/local/lib/libcurl.so.4 (compiled source)

Results of test

0.1% on start
0.1% after 1 hour
0.1% after 2 hour
.......
Comment 13 Kamil Dudka 2009-05-14 08:15:08 EDT
Thanks for testing it! It'll take some time to catch the leaks as there are AFAIK not caused by the (lib)curl code itself.
Comment 14 Kamil Dudka 2009-05-19 05:28:14 EDT
The problem is more complex than we initially expected to. We are now working on review of a few patches for the nss package and it will take some time before the update will be ready. If you are interested in testing not yet approved packages of fixed curl/nss, here they are:

https://bugzilla.redhat.com/show_bug.cgi?id=501138#c6

Please do not test them on a production machine.
Comment 15 Kamil Dudka 2009-05-22 14:16:15 EDT
(In reply to comment #14)
> https://bugzilla.redhat.com/show_bug.cgi?id=501138#c6

Now we have have all the patches at least once reviewed -- the scratch builds should be stable enough. Please, have a go at that.

Note one patch has not been applied yet, but it doesn't affect this bug (you have no private key in the test case).
Comment 16 Pietro Incardona 2009-05-22 15:46:26 EDT
Ok thanks, i will download and test in this weekend, if work i will return
to libcurl + nss
Comment 17 Kamil Dudka 2009-08-11 10:26:16 EDT
Have you tried it? Any new experiences?
Comment 18 Pietro Incardona 2009-08-11 16:17:25 EDT
Ops.... at the moment i am using curl with OpenSSL, i will test as soon as possible.
Comment 19 Pietro Incardona 2009-08-11 20:29:24 EDT
I has updated the test machine to FC11

now i has nss 3.12.3.99.3-2.11.3.fc11
and curl 7.19.4-9.fc11
and libcurl 7.19.4-9.fc11

using the simple test case, the memory continue to grow.
On my test machine has memory leak

On the others machines i continue to use OpenSSL

Waiting for other tests if needed
Comment 20 Kamil Dudka 2009-08-12 02:38:14 EDT
Thanks for the feedback! Could you please try it again with the patches for NSS from bug #509705? Let me know if a scratch build of NSS can help you with testing.
Comment 21 Pietro Incardona 2009-08-12 09:30:14 EDT
Where i can find the fedora NSS git sources to apply the paches ?
Comment 22 Kamil Dudka 2009-08-12 09:45:40 EDT
You can download SRPM of the latest build from Koji:
$ curl -O http://kojipkgs.fedoraproject.org/packages/nss/3.12.3.99.3/2.11.3.fc11/src/nss-3.12.3.99.3-2.11.3.fc11.src.rpm

Then extract the sources:
$ rpm2cpio nss-3.12.3.99.3-2.11.3.fc11.src.rpm | cpio -idv

Do whatever you want ... and finally build by:
$ rpmbuild -bb nss.spec
Comment 23 Kamil Dudka 2009-08-14 08:16:27 EDT
Created attachment 357448 [details]
bundle of fixes for nss/pem which are not included in stable release

Elio, could you please prepare the F-11 scratch build for testing. I've tried it but it died with:
error: File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/lib64/pkgconfig/nss.pc
error: File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/bin/nss-config
RPM build errors:
    File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/lib64/pkgconfig/nss.pc
    File not found: /builddir/build/BUILDROOT/nss-3.12.3.99.3-2.11.3.1.fc11.x86_64/usr/bin/nss-config

The patch against F-11 stable is attached. Thanks in advance!
Comment 24 Pietro Incardona 2009-08-14 11:02:52 EDT
What i has done, probably something wrong

rpm2cpio nss-3.12.3.99.3-2.11.3.fc11.src.rpm | cpio -idv

tar -xjf nss-pem-20090622.tar.bz2

created a dir nss-3.12.3.99.3 inserted folder mozilla
applied patch

patch -p0 < nss-pem-bz500636.patch

recompress

tar cjfv nss-pem-20090622.tar.bz2 mozilla/

rpmbuild -bb nss.spec


Tests summary:
--------------
Passed:             5474
Failed:             3
Failed with core:   0
Unknown status:     0

+ cd ../../../../
+ killall selfserv_9829
selfserv_9829: no process killed
+ :
++ grep -c FAILED ./mozilla/tests_results/security/localhost.1/output.log
+ TEST_FAILURES=3
+ '[' 3 -ne 0 ']'
+ echo 'error: test suite returned failure(s)'
error: test suite returned failure(s)
+ exit 1
error: Bad exit status from /var/tmp/rpm-tmp.SXtOHT (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.SXtOHT (%build)


No Build package into RPMS folder

Probably for the 3 test failed ?

Failed test

#1910: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot

#3403: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot

#5467: OCSP: Verifying certificate(s) OCSPEE12.cert OCSPCA1.cert with flags -g leaf -m ocsp -d OCSPRootDB -t OCSPRoot
Comment 25 Kamil Dudka 2009-08-14 11:56:26 EDT
(In reply to comment #24)
> No Build package into RPMS folder
> 
> Probably for the 3 test failed ?

Yes, that might be the reason.

You can disable the test suite by inserting 'exit 0' line before the line '# Run test suite.'. But then please do not install the package on your system. Just build it locally using 'rpmbuild -bc' and use the just-built libnsspem.so. You can force its loading by exporting LD_PRELOAD=absolute_path/libnsspem.so while running your tests.

And of course you don't need to patch the sources manually, rpm will do the job for you. Add the two following lines to .spec to the corresponding sections:

Patch8: nss-pem-bz500636.patch

%patch8 -p1

Thanks in advance for testing!
Comment 26 Pietro Incardona 2009-08-15 20:32:18 EDT
Like your comment 23

Processing files: nss-devel-3.12.3.99.3-2.11.3.fc11.x86_64
error: File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/lib64/pkgconfig/nss.pc
error: File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/bin/nss-config


RPM build errors:
    File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/lib64/pkgconfig/nss.pc
    File not found: /home/I-BIRD/rpmbuild/BUILDROOT/nss-3.12.3.99.3-2.11.3.fc11.x86_64/usr/bin/nss-config
Comment 27 Elio Maldonado Batiz 2009-08-16 12:12:50 EDT
The latest rpm build tools exposed latent bugs in nss.spec. They are now fixed
See http://koji.fedoraproject.org/koji/buildinfo?buildID=127403
Comment 28 Kamil Dudka 2009-08-17 05:48:46 EDT
Elio, thanks for the speedy fix!

Pietro, you can download the scratch build with the patches applied here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1609106

Please test along with latest F-11 (lib)curl from updates-testing:
http://koji.fedoraproject.org/koji/buildinfo?buildID=127267
Comment 29 Pietro Incardona 2009-08-19 12:56:42 EDT
Memory leak seem to be solved

testing now is

on start
0.1%
after 10 minutes it grow to
0.2%
after 1h:30 is fixed to
0.2%
Comment 30 Kamil Dudka 2009-09-24 02:53:03 EDT
nss-3.12.4-3.fc11 has been pushed to the Fedora 11 stable repository.  If
problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.