Bug 453612

I'm getting this with the latest curl-7.18.2-8 with the improved NSS patch:

Array
(
    [url] => https://XXXXX/cgi-bin/messages.cgi
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => 0
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 0
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => 0
    [upload_content_length] => 0
    [starttransfer_time] => 0
    [redirect_time] => 0
)
* getaddrinfo(3) failed for XXXXX:443
* Couldn't resolve host 'XXXXX'
* Closing connection #0
Couldn't resolve host 'XXXXX'

It doesn't seem to hang in curl_exec() any more for me.
Could you please give it a try?

I have also seen a memory leak issue on FC10 with libcurl-7.18.2-7.fc10.i386.

I'm doing https calls using the curl_easy interface.
I've got the following stripped down code that causes the error.
====================================================
#include <curl/curl.h>
#include <string>
#include <stdlib.h>

int main (int argc, char ** argv)
{
  curl_global_init(CURL_GLOBAL_ALL);
  CURL * pcurl;
  CURLcode res = CURLE_OK;
  std::string url=argv[1];
  std::string certfile=argv[2];
  int iattempts=atoi(argv[3]);

  printf("Create curl connection\n");
  pcurl = curl_easy_init();

  if ((res = curl_easy_setopt(pcurl,CURLOPT_CAINFO,certfile.c_str())) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;
  }
  if ((res = curl_easy_setopt(pcurl,CURLOPT_SSL_VERIFYPEER,1)) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;

  }

  if ((res = curl_easy_setopt(pcurl, CURLOPT_URL, url.c_str())) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;
  }

  for (int i=0;i<iattempts;i++)
  {
     printf("Doing call number: %i\n",i);
     // Perform the SSL call
     if ((res = curl_easy_perform(pcurl)) != CURLE_OK)
     {
        printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
        curl_easy_cleanup(pcurl);
        curl_global_cleanup();
        return res;
     }
         usleep(250000);
  }

  curl_easy_cleanup(pcurl);
  pcurl=NULL;

  curl_global_cleanup();
  printf("PID:%d\n",getpid());
  abort();
  return 0;
}
====================================================
compile:  g++ -o curlleak -lcurl curlleak.cpp
Usage: ./curlleak <https_url> <cacertfile> <iterations>

If I open the core file generated as a result of the abort() on the second last line, I can see the CA information repeating many times over.  The more iterations, the more the cacert info repeats.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

I see the same behaviour seen in comment #2.  I'll attach an even simpler test program with instructions for seeing it.

Created attachment 339057 [details]
Test program to see duplicated CA Information

Compile like this:
gcc -g -o break break-curl.c -lcurl

break takes one argument, the number of times to retrieve a https URL.  Use it something like this:
$ ./break 1
$ strings core.22138|grep AOL |wc -l
32
$ ./break 2
$ strings core.22140|grep AOL |wc -l
64
$ ./break 3
$ strings core.22141|grep AOL |wc -l
96
$ ./break 4
$ strings core.22142|grep AOL |wc -l
128

Created attachment 339060 [details]
valgrind log for 10 iterations

Here's the output of  valgrind --leak-check=full ./break 10

The "possibly lost" blocks seem to be where this uncontrolled growth is occuring.

Created attachment 339061 [details]
Patch from upstream for generic objects

Upstream (Daniel Stenberg) has given me this patch which I have applied to my curl.  It reduces the number of bytes that are "definitely lost".  It makes "possibly lost" larger probably because it increases the size of the data structures being leaked there.

(In reply to comment #7)
> Upstream (Daniel Stenberg) has given me this patch which I have applied to my
> curl.  It reduces the number of bytes that are "definitely lost".  It makes
> "possibly lost" larger probably because it increases the size of the data
> structures being leaked there.  

Toshio, thanks for negotiating it with upstream. I am going to apply the patch on rawhide next week. Note it still leaks heavily while using a client certificate.

Built as curl-7.19.4-6.fc11, but leaving this bug open to track the remaining libcurl/nss bugs.

Yet another patch sent upstream:
http://curl.haxx.se/mail/lib-2009-04/0381.html

Also opened a new bug against nspr: Bug 497535

new build for rawhide: curl-7.19.4-10.fc12

I am also seeing this issue with the libcurl library with a C program and certificate/private key combinations on Fedora 10 (libcurl-7.19.4-4.fc10.i386). A backtrace reveals the following:

#0  0x00dca416 in __kernel_vsyscall ()
#1  0x007e3986 in gettimeofday () from /lib/libc.so.6
#2  0x04bc8e8d in PR_Now () from /lib/libnspr4.so
#3  0x033fa9bf in ?? () from /lib/libnss3.so
#4  0x033fac05 in ?? () from /lib/libnss3.so
#5  0x033fad65 in PK11_Authenticate () from /lib/libnss3.so
#6  0x0045cbbd in ?? () from /usr/lib/libcurl.so.4
#7  0x0045d8e5 in Curl_nss_connect () from /usr/lib/libcurl.so.4
#8  0x0045433f in Curl_ssl_connect () from /usr/lib/libcurl.so.4
#9  0x0043386a in Curl_http_connect () from /usr/lib/libcurl.so.4
#10 0x0043a9c1 in Curl_protocol_connect () from /usr/lib/libcurl.so.4
#11 0x0043fdaa in Curl_connect () from /usr/lib/libcurl.so.4
#12 0x00448a79 in Curl_perform () from /usr/lib/libcurl.so.4
#13 0x004497a3 in curl_easy_perform () from /usr/lib/libcurl.so.4

This backtrace is not constant as it seems to be running in a tight loop. CPU usage is at 100% and memory usage climbs about 10 megs per second. The one consistent function in the bt is PK11_Authenticate() so it never leaves there. I have not investigated any further.

> This backtrace is not constant as it seems to be running in a tight loop. CPU
> usage is at 100% and memory usage climbs about 10 megs per second. The one
> consistent function in the bt is PK11_Authenticate() so it never leaves there.
> I have not investigated any further.  

Is it reproducible? Could you please attach something like minimal example?

Created attachment 343159 [details]
Complete libcurl test case

Yes, it is always reproducible. I am attaching a complete test case. It includes a test program (source code w/ Makefile) and appropriate SSL certificates. You will need to add a "test.local" address to your /etc/hosts file and point it to an SSL enabled HTTP server. Then just execute: ./test-libcurl

You will need to kill the app quickly or attach gdb before-hand to prevent your system from entering an OOM state.

Created attachment 343184 [details]
proposed patch

Indeed. Thanks for the test case!

A scratch build is ready:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1344514

(In reply to comment #15)
> A scratch build is ready:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=1344514  

Thanks. This stops the tight loop/memory eater, but now I get:

* Unable to load client key: Incorrect password
* Closing connection #0
* Problem with the local SSL certificate

This error is incorrect. I verified the private key passphrase with openssl rsa. Were you seeing this using my test case?

Created attachment 343198 [details]
patch leading to shorter code

"Incorrect password" may also mean the password is not strong enough. Could you please try it again with at least 7 characters long password with a mix of upper-case, lower-case, digits, punctuation and non-ASCII characters?

Created attachment 343199 [details]
patch leading to shorter code

I am able to load your key after the change of password to "asDF89*(" by following command:
$ openssl rsa -in test-key.pem -des3 -out test-key-stronger.pem

I first attempted the following passwords:
A!mj8z$mc
A!m(8z$mC
A!m(89$mC

None of those would work with libcurl. I then tried your password, and that worked fine. I'm puzzled how my passwords were not strong enough. The real program I am using libcurl with is using a password that is 20 characters long with upper, lower, numeric, and non-ascii and that was rejected by libcurl as well.

Is there a special CURLOPT define that turns off this check for password strength? It's rather annoying.

This behavior is behind the scope of (lib)curl. It's rather an nss/pem issue and I am afraid the behavior is intended. The libcurl can only explicitly detect the weak passwords and print more verbose debug message. I've seen this approach in nss_compat_ossl package. You can open a separate low priority bug for it. I'd like to solve the infinite loop bug first as curl-7.19.5 is being released in few days. Thanks for your help with tracking it down!

curl-7.19.4-5.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/curl-7.19.4-5.fc10

curl-7.19.4-8.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/curl-7.19.4-8.fc11

curl-7.19.4-5.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/curl-7.19.4-5.fc9

(In reply to comment #20)
> You can open a separate low priority bug
> for it. I'd like to solve the infinite loop bug first as curl-7.19.5 is being
> released in few days. Thanks for your help with tracking it down!  

Done[1], thanks!

[1] https://bugzilla.redhat.com/show_bug.cgi?id=500180

curl-7.19.4-5.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.

curl-7.19.4-8.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.

curl-7.19.4-5.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

Created attachment 343741 [details]
backtrace from 7.18

(sorry for my english)
The problem is still present into 7.19.4-5

memory leak

I has an my application that use curl with TLS/SSL, i has recompiled from source
to link to OpenSSL to workaround the bug. After yum update to 7.19.4-5 my application has restarted to has memory leak until i has recompiled. The only difference i see from the old version is that the memory leak and frequency of crash is slower and probabbly different than before.

I has extrapolated this code (5-6 month ago ???) from my application to reproduce the bug into curl (7.18 ???), i has not tested if reproduce the bug also into 7.19, but there are an high probability that this code reproduce the bug of memory leak also into 7.19.4-5 like my application does.

I has also attached the backtrace of the 7.18

and reported the little example used to reproduce the problem into 7.18 (and probably 7.19)

#include <iostream>
#include <curl/curl.h>
#include <gtk/gtk.h>



size_t NoProcess(void *buffer, size_t size, size_t nmemb, void *userp)
{
	return size * nmemb;
}

gpointer Test(gpointer data)
{
	CURL * Fin;

	Fin = curl_easy_init();

//	if (curl_easy_setopt(Fin, CURLOPT_NOSIGNAL, 1) != CURLE_OK)	return NULL;

	while (1)
	{
		if (curl_easy_setopt(Fin, CURLOPT_WRITEDATA, NULL) != CURLE_OK)	return NULL;
		if (curl_easy_setopt(Fin, CURLOPT_WRITEFUNCTION, NoProcess) != CURLE_OK)	return NULL;
		if (curl_easy_setopt(Fin, CURLOPT_URL, "https://www.fineco.it/fineco/jsp/login/content.jsp") != CURLE_OK)	return NULL;


		if (curl_easy_perform(Fin) != CURLE_OK)	return NULL;

		sleep(1);
	}

	return NULL;
}

int main()
{
	curl_global_init(CURL_GLOBAL_ALL);
	g_thread_init(NULL);

	g_thread_create(Test,NULL,false,NULL);

	while (1)	
	{
		sleep(1);
	}

	curl_global_cleanup();
	return 0;
} 

This code continue to read an https page, after a lot of iteration
the memory should grow after some day the code use all memory of the PC
( Is an extimation based on the behavior based on my application (NOT TESTED with reported code), into 7.18 the code crash after 7 - 8 hour of running with backtrace attached (TESTED with reported code) )

At this moment to use curl with my application i need to recompile curl from source and link to OpenSSL

"recompile" mean recompile curl NOT my application
and link OpenSSL mean link curl to OpenSSL

I tried libcurl-7.19.4-5.fc9.i386.rpm. Now I have a different problem from the initial post:

[..snip..]
$result = curl_exec($curl);
[..snip..]

When I print the result of curl_exec(), I get an empty string! (the program is not stuck forever anymore)

I'm also sticking to OpenSSL with a locally compiled version of libcurl since it's still the only way to have my system functional.

(In reply to comment #31)
> When I print the result of curl_exec(), I get an empty string! (the program is
> not stuck forever anymore)

Bernard, thanks for your feedback. Could you please try it again with an exactly 8 characters length password? (bug 500180)

(In reply to comment #30)
> "recompile" mean recompile curl NOT my application
> and link OpenSSL mean link curl to OpenSSL  

This issue seems to not match the original bug report. Could you open a separate bug for it? Please also attach the testcase as an attachment. This ticket starts to be a bit overloaded. Thanks!

(In reply to comment #28)
> Created an attachment (id=343741) [details]
> backtrace from 7.18  

Looking at the backtrace it seems like already fixed bug 483855. Please consider update to nss-3.12.2.0-5.fc10. If the problems still persist, open a separate bug for it.

I'm using nss-3.12.2.0-3.fc9.i386 (I did a yum update to get libcurl-7.19.4-5.fc9.i386.rpm and my nss is the version currently available from yum for F9).

I have several problems to perform more extensive testings, the biggest being lack of time. As to test with a particular password, unfortunately the password is not under my control: the remote service we use with curl is a third party commercial service that manages its own access info. I may request a new password but I won't be able to get one with a specific format.

(In reply to comment #35)
> I have several problems to perform more extensive testings, the biggest being
> lack of time. As to test with a particular password, unfortunately the password
> is not under my control: the remote service we use with curl is a third party
> commercial service that manages its own access info. I may request a new
> password but I won't be able to get one with a specific format.  

I understand. You can subscribe yourself to the bug 500180 and watch its progress. I am pretty sure you're affected by the PEM reader bug (nss package).

I've just checked, the password we must use is 10 characters long, so you may be right! I've subscribed to bug #500180 and will perform new tests when a fixed version of nss is made available.

(In reply to comment #37)
> I've just checked, the password we must use is 10 characters long, so you may
> be right! I've subscribed to bug #500180 and will perform new tests when a
> fixed version of nss is made available.  

The infinite loop bug (from the original report) should be resolved by the updates listed above, therefore closing CURRENTRELEASE. For other NSS related (lib)curl bugs is now opened the tracker bug 501138. Thanks for you assistance with tracking down the 8 characters long passwords issue!