Bug 453612 - infinite loop consuming memory while loading a private key
infinite loop consuming memory while loading a private key
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: curl (Show other bugs)
10
All Linux
low Severity high
: ---
: ---
Assigned To: Kamil Dudka
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 501138
  Show dependency treegraph
 
Reported: 2008-07-01 11:14 EDT by Bernard Fouché
Modified: 2015-10-19 08:25 EDT (History)
6 users (show)

See Also:
Fixed In Version: curl-7.19.4-5.fc10
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-17 04:03:11 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Test program to see duplicated CA Information (1.20 KB, text/plain)
2009-04-10 01:16 EDT, Toshio Ernie Kuratomi
no flags Details
valgrind log for 10 iterations (3.82 KB, application/octet-stream)
2009-04-10 01:22 EDT, Toshio Ernie Kuratomi
no flags Details
Patch from upstream for generic objects (5.31 KB, patch)
2009-04-10 01:26 EDT, Toshio Ernie Kuratomi
no flags Details | Diff
Complete libcurl test case (5.47 KB, application/x-gzip)
2009-05-08 15:56 EDT, Michael Cronenworth
no flags Details
proposed patch (562 bytes, patch)
2009-05-08 18:03 EDT, Kamil Dudka
no flags Details | Diff
patch leading to shorter code (2.93 KB, patch)
2009-05-09 02:57 EDT, Kamil Dudka
no flags Details | Diff
patch leading to shorter code (3.22 KB, patch)
2009-05-09 03:59 EDT, Kamil Dudka
no flags Details | Diff
backtrace from 7.18 (23.10 KB, application/octet-stream)
2009-05-13 06:17 EDT, Pietro Incardona
no flags Details

  None (edit)
Description Bernard Fouché 2008-07-01 11:14:28 EDT
Description of problem:

When using libcurl, curl(1) or curl queries from PHP, the program never returns
and grows until it fails because of a too big memory usage.

Version-Release number of selected component (if applicable):

curl-7.18.1.1

How reproducible:

Systematic. 

Steps to Reproduce:

Difficult to completely show it here because certificates are needed and I can't
post them here.

However here is a sample of PHP code that shows the problem:

#!/usr/bin/php
<?php

error_reporting(E_ALL);
ini_set('display_errors', 'On');

define("SMS_URL_MESSAGE", 	"https://XXXXX/cgi-bin/messages.cgi");
define("SMS_SSL_PATH_CERTIF", "XXXX.ssl.pem");
define("SMS_SSL_PATH_KEY",	"XXXX.key.pem");
define("SMS_SSL_PASSWORD", 	"XXXXXX");
		
$curl = curl_init(SMS_URL_MESSAGE);

curl_setopt($curl, CURLOPT_VERBOSE, 1);
curl_setopt($curl, CURLOPT_TIMEOUT,3);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT,3);
curl_setopt($curl, CURLOPT_RETURNTRANSFER,1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSLCERT, SMS_SSL_PATH_CERTIF);
curl_setopt($curl, CURLOPT_SSLKEY, SMS_SSL_PATH_KEY);
curl_setopt($curl, CURLOPT_SSLKEYPASSWD, SMS_SSL_PASSWORD);

print_r(curl_getinfo($curl));

$result = curl_exec($curl);

print_r(curl_error($curl));
curl_close($curl);

print $result;
?>

The program will never returns from curl_exec(). Using the command line curl(1)
and providing similar parameters give the same result.
  
It seems that current curl code does not handle correctly libnss:
http://curl.haxx.se/mail/tracker-2008-06/0000.html

I recompiled curl with './configure --without-nss', forcing openSSL usage,
replaced /usr/lib/libcurl.so.4.1.0 with the new library, and the previous PHP
code sample now works correctly.

I would suggest to have the curl rpm compiled with '--without-nss' until curl
correctly handles libnss.

Thanks.

  Bernard
Comment 1 Jindrich Novy 2008-12-15 03:55:58 EST
I'm getting this with the latest curl-7.18.2-8 with the improved NSS patch:

Array
(
    [url] => https://XXXXX/cgi-bin/messages.cgi
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => 0
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 0
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => 0
    [upload_content_length] => 0
    [starttransfer_time] => 0
    [redirect_time] => 0
)
* getaddrinfo(3) failed for XXXXX:443
* Couldn't resolve host 'XXXXX'
* Closing connection #0
Couldn't resolve host 'XXXXX'

It doesn't seem to hang in curl_exec() any more for me.
Could you please give it a try?
Comment 2 Jacques Verryn 2009-01-13 07:07:52 EST
I have also seen a memory leak issue on FC10 with libcurl-7.18.2-7.fc10.i386.

I'm doing https calls using the curl_easy interface.
I've got the following stripped down code that causes the error.
====================================================
#include <curl/curl.h>
#include <string>
#include <stdlib.h>

int main (int argc, char ** argv)
{
  curl_global_init(CURL_GLOBAL_ALL);
  CURL * pcurl;
  CURLcode res = CURLE_OK;
  std::string url=argv[1];
  std::string certfile=argv[2];
  int iattempts=atoi(argv[3]);

  printf("Create curl connection\n");
  pcurl = curl_easy_init();

  if ((res = curl_easy_setopt(pcurl,CURLOPT_CAINFO,certfile.c_str())) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;
  }
  if ((res = curl_easy_setopt(pcurl,CURLOPT_SSL_VERIFYPEER,1)) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;

  }

  if ((res = curl_easy_setopt(pcurl, CURLOPT_URL, url.c_str())) != CURLE_OK)
  {
     printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
     curl_easy_cleanup(pcurl);
     curl_global_cleanup();
     return res;
  }

  for (int i=0;i<iattempts;i++)
  {
     printf("Doing call number: %i\n",i);
     // Perform the SSL call
     if ((res = curl_easy_perform(pcurl)) != CURLE_OK)
     {
        printf("CURL ERROR: Line %i Error: %i\n",__LINE__,res);
        curl_easy_cleanup(pcurl);
        curl_global_cleanup();
        return res;
     }
         usleep(250000);
  }

  curl_easy_cleanup(pcurl);
  pcurl=NULL;

  curl_global_cleanup();
  printf("PID:%d\n",getpid());
  abort();
  return 0;
}
====================================================
compile:  g++ -o curlleak -lcurl curlleak.cpp
Usage: ./curlleak <https_url> <cacertfile> <iterations>

If I open the core file generated as a result of the abort() on the second last line, I can see the CA information repeating many times over.  The more iterations, the more the cacert info repeats.
Comment 3 Fedora Admin XMLRPC Client 2009-04-03 05:22:22 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 4 Toshio Ernie Kuratomi 2009-04-10 01:13:15 EDT
I see the same behaviour seen in comment #2.  I'll attach an even simpler test program with instructions for seeing it.
Comment 5 Toshio Ernie Kuratomi 2009-04-10 01:16:49 EDT
Created attachment 339057 [details]
Test program to see duplicated CA Information

Compile like this:
gcc -g -o break break-curl.c -lcurl

break takes one argument, the number of times to retrieve a https URL.  Use it something like this:
$ ./break 1
$ strings core.22138|grep AOL |wc -l
32
$ ./break 2
$ strings core.22140|grep AOL |wc -l
64
$ ./break 3
$ strings core.22141|grep AOL |wc -l
96
$ ./break 4
$ strings core.22142|grep AOL |wc -l
128
Comment 6 Toshio Ernie Kuratomi 2009-04-10 01:22:20 EDT
Created attachment 339060 [details]
valgrind log for 10 iterations

Here's the output of  valgrind --leak-check=full ./break 10

The "possibly lost" blocks seem to be where this uncontrolled growth is occuring.
Comment 7 Toshio Ernie Kuratomi 2009-04-10 01:26:13 EDT
Created attachment 339061 [details]
Patch from upstream for generic objects

Upstream (Daniel Stenberg) has given me this patch which I have applied to my curl.  It reduces the number of bytes that are "definitely lost".  It makes "possibly lost" larger probably because it increases the size of the data structures being leaked there.
Comment 8 Kamil Dudka 2009-04-10 07:19:24 EDT
(In reply to comment #7)
> Upstream (Daniel Stenberg) has given me this patch which I have applied to my
> curl.  It reduces the number of bytes that are "definitely lost".  It makes
> "possibly lost" larger probably because it increases the size of the data
> structures being leaked there.  

Toshio, thanks for negotiating it with upstream. I am going to apply the patch on rawhide next week. Note it still leaks heavily while using a client certificate.
Comment 9 Kamil Dudka 2009-04-14 06:52:28 EDT
Built as curl-7.19.4-6.fc11, but leaving this bug open to track the remaining libcurl/nss bugs.
Comment 10 Kamil Dudka 2009-04-24 10:27:48 EDT
Yet another patch sent upstream:
http://curl.haxx.se/mail/lib-2009-04/0381.html

Also opened a new bug against nspr: Bug 497535
Comment 11 Kamil Dudka 2009-04-27 04:47:36 EDT
new build for rawhide: curl-7.19.4-10.fc12
Comment 12 Michael Cronenworth 2009-05-08 12:16:49 EDT
I am also seeing this issue with the libcurl library with a C program and certificate/private key combinations on Fedora 10 (libcurl-7.19.4-4.fc10.i386). A backtrace reveals the following:

#0  0x00dca416 in __kernel_vsyscall ()
#1  0x007e3986 in gettimeofday () from /lib/libc.so.6
#2  0x04bc8e8d in PR_Now () from /lib/libnspr4.so
#3  0x033fa9bf in ?? () from /lib/libnss3.so
#4  0x033fac05 in ?? () from /lib/libnss3.so
#5  0x033fad65 in PK11_Authenticate () from /lib/libnss3.so
#6  0x0045cbbd in ?? () from /usr/lib/libcurl.so.4
#7  0x0045d8e5 in Curl_nss_connect () from /usr/lib/libcurl.so.4
#8  0x0045433f in Curl_ssl_connect () from /usr/lib/libcurl.so.4
#9  0x0043386a in Curl_http_connect () from /usr/lib/libcurl.so.4
#10 0x0043a9c1 in Curl_protocol_connect () from /usr/lib/libcurl.so.4
#11 0x0043fdaa in Curl_connect () from /usr/lib/libcurl.so.4
#12 0x00448a79 in Curl_perform () from /usr/lib/libcurl.so.4
#13 0x004497a3 in curl_easy_perform () from /usr/lib/libcurl.so.4

This backtrace is not constant as it seems to be running in a tight loop. CPU usage is at 100% and memory usage climbs about 10 megs per second. The one consistent function in the bt is PK11_Authenticate() so it never leaves there. I have not investigated any further.
Comment 13 Kamil Dudka 2009-05-08 13:43:03 EDT
> This backtrace is not constant as it seems to be running in a tight loop. CPU
> usage is at 100% and memory usage climbs about 10 megs per second. The one
> consistent function in the bt is PK11_Authenticate() so it never leaves there.
> I have not investigated any further.  

Is it reproducible? Could you please attach something like minimal example?
Comment 14 Michael Cronenworth 2009-05-08 15:56:31 EDT
Created attachment 343159 [details]
Complete libcurl test case

Yes, it is always reproducible. I am attaching a complete test case. It includes a test program (source code w/ Makefile) and appropriate SSL certificates. You will need to add a "test.local" address to your /etc/hosts file and point it to an SSL enabled HTTP server. Then just execute: ./test-libcurl

You will need to kill the app quickly or attach gdb before-hand to prevent your system from entering an OOM state.
Comment 15 Kamil Dudka 2009-05-08 18:03:48 EDT
Created attachment 343184 [details]
proposed patch

Indeed. Thanks for the test case!

A scratch build is ready:
http://koji.fedoraproject.org/koji/taskinfo?taskID=1344514
Comment 16 Michael Cronenworth 2009-05-08 18:16:39 EDT
(In reply to comment #15)
> A scratch build is ready:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=1344514  

Thanks. This stops the tight loop/memory eater, but now I get:

* Unable to load client key: Incorrect password
* Closing connection #0
* Problem with the local SSL certificate

This error is incorrect. I verified the private key passphrase with openssl rsa. Were you seeing this using my test case?
Comment 17 Kamil Dudka 2009-05-09 02:57:26 EDT
Created attachment 343198 [details]
patch leading to shorter code

"Incorrect password" may also mean the password is not strong enough. Could you please try it again with at least 7 characters long password with a mix of upper-case, lower-case, digits, punctuation and non-ASCII characters?
Comment 18 Kamil Dudka 2009-05-09 03:59:22 EDT
Created attachment 343199 [details]
patch leading to shorter code

I am able to load your key after the change of password to "asDF89*(" by following command:
$ openssl rsa -in test-key.pem -des3 -out test-key-stronger.pem
Comment 19 Michael Cronenworth 2009-05-09 14:01:04 EDT
I first attempted the following passwords:
A!mj8z$mc
A!m(8z$mC
A!m(89$mC

None of those would work with libcurl. I then tried your password, and that worked fine. I'm puzzled how my passwords were not strong enough. The real program I am using libcurl with is using a password that is 20 characters long with upper, lower, numeric, and non-ascii and that was rejected by libcurl as well.

Is there a special CURLOPT define that turns off this check for password strength? It's rather annoying.
Comment 20 Kamil Dudka 2009-05-09 14:55:35 EDT
This behavior is behind the scope of (lib)curl. It's rather an nss/pem issue and I am afraid the behavior is intended. The libcurl can only explicitly detect the weak passwords and print more verbose debug message. I've seen this approach in nss_compat_ossl package. You can open a separate low priority bug for it. I'd like to solve the infinite loop bug first as curl-7.19.5 is being released in few days. Thanks for your help with tracking it down!
Comment 21 Fedora Update System 2009-05-11 04:54:08 EDT
curl-7.19.4-5.fc10 has been submitted as an update for Fedora 10.
http://admin.fedoraproject.org/updates/curl-7.19.4-5.fc10
Comment 22 Fedora Update System 2009-05-11 04:54:14 EDT
curl-7.19.4-8.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/curl-7.19.4-8.fc11
Comment 23 Fedora Update System 2009-05-11 04:54:19 EDT
curl-7.19.4-5.fc9 has been submitted as an update for Fedora 9.
http://admin.fedoraproject.org/updates/curl-7.19.4-5.fc9
Comment 24 Michael Cronenworth 2009-05-11 10:37:57 EDT
(In reply to comment #20)
> You can open a separate low priority bug
> for it. I'd like to solve the infinite loop bug first as curl-7.19.5 is being
> released in few days. Thanks for your help with tracking it down!  

Done[1], thanks!

[1] https://bugzilla.redhat.com/show_bug.cgi?id=500180
Comment 25 Fedora Update System 2009-05-11 23:54:23 EDT
curl-7.19.4-5.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 26 Fedora Update System 2009-05-11 23:56:47 EDT
curl-7.19.4-8.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 27 Fedora Update System 2009-05-12 00:12:19 EDT
curl-7.19.4-5.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 28 Pietro Incardona 2009-05-13 06:17:08 EDT
Created attachment 343741 [details]
backtrace from 7.18
Comment 29 Pietro Incardona 2009-05-13 06:28:56 EDT
(sorry for my english)
The problem is still present into 7.19.4-5

memory leak

I has an my application that use curl with TLS/SSL, i has recompiled from source
to link to OpenSSL to workaround the bug. After yum update to 7.19.4-5 my application has restarted to has memory leak until i has recompiled. The only difference i see from the old version is that the memory leak and frequency of crash is slower and probabbly different than before.

I has extrapolated this code (5-6 month ago ???) from my application to reproduce the bug into curl (7.18 ???), i has not tested if reproduce the bug also into 7.19, but there are an high probability that this code reproduce the bug of memory leak also into 7.19.4-5 like my application does.

I has also attached the backtrace of the 7.18

and reported the little example used to reproduce the problem into 7.18 (and probably 7.19)

#include <iostream>
#include <curl/curl.h>
#include <gtk/gtk.h>



size_t NoProcess(void *buffer, size_t size, size_t nmemb, void *userp)
{
	return size * nmemb;
}

gpointer Test(gpointer data)
{
	CURL * Fin;

	Fin = curl_easy_init();

//	if (curl_easy_setopt(Fin, CURLOPT_NOSIGNAL, 1) != CURLE_OK)	return NULL;

	while (1)
	{
		if (curl_easy_setopt(Fin, CURLOPT_WRITEDATA, NULL) != CURLE_OK)	return NULL;
		if (curl_easy_setopt(Fin, CURLOPT_WRITEFUNCTION, NoProcess) != CURLE_OK)	return NULL;
		if (curl_easy_setopt(Fin, CURLOPT_URL, "https://www.fineco.it/fineco/jsp/login/content.jsp") != CURLE_OK)	return NULL;


		if (curl_easy_perform(Fin) != CURLE_OK)	return NULL;

		sleep(1);
	}

	return NULL;
}

int main()
{
	curl_global_init(CURL_GLOBAL_ALL);
	g_thread_init(NULL);

	g_thread_create(Test,NULL,false,NULL);

	while (1)	
	{
		sleep(1);
	}

	curl_global_cleanup();
	return 0;
} 

This code continue to read an https page, after a lot of iteration
the memory should grow after some day the code use all memory of the PC
( Is an extimation based on the behavior based on my application (NOT TESTED with reported code), into 7.18 the code crash after 7 - 8 hour of running with backtrace attached (TESTED with reported code) )

At this moment to use curl with my application i need to recompile curl from source and link to OpenSSL
Comment 30 Pietro Incardona 2009-05-13 07:04:20 EDT
"recompile" mean recompile curl NOT my application
and link OpenSSL mean link curl to OpenSSL
Comment 31 Bernard Fouché 2009-05-13 08:23:07 EDT
I tried libcurl-7.19.4-5.fc9.i386.rpm. Now I have a different problem from the initial post:

[..snip..]
$result = curl_exec($curl);
[..snip..]

When I print the result of curl_exec(), I get an empty string! (the program is not stuck forever anymore)

I'm also sticking to OpenSSL with a locally compiled version of libcurl since it's still the only way to have my system functional.
Comment 32 Kamil Dudka 2009-05-13 09:17:15 EDT
(In reply to comment #31)
> When I print the result of curl_exec(), I get an empty string! (the program is
> not stuck forever anymore)

Bernard, thanks for your feedback. Could you please try it again with an exactly 8 characters length password? (bug 500180)
Comment 33 Kamil Dudka 2009-05-13 09:21:14 EDT
(In reply to comment #30)
> "recompile" mean recompile curl NOT my application
> and link OpenSSL mean link curl to OpenSSL  

This issue seems to not match the original bug report. Could you open a separate bug for it? Please also attach the testcase as an attachment. This ticket starts to be a bit overloaded. Thanks!
Comment 34 Kamil Dudka 2009-05-13 09:28:01 EDT
(In reply to comment #28)
> Created an attachment (id=343741) [details]
> backtrace from 7.18  

Looking at the backtrace it seems like already fixed bug 483855. Please consider update to nss-3.12.2.0-5.fc10. If the problems still persist, open a separate bug for it.
Comment 35 Bernard Fouché 2009-05-13 09:58:45 EDT
I'm using nss-3.12.2.0-3.fc9.i386 (I did a yum update to get libcurl-7.19.4-5.fc9.i386.rpm and my nss is the version currently available from yum for F9).

I have several problems to perform more extensive testings, the biggest being lack of time. As to test with a particular password, unfortunately the password is not under my control: the remote service we use with curl is a third party commercial service that manages its own access info. I may request a new password but I won't be able to get one with a specific format.
Comment 36 Kamil Dudka 2009-05-13 11:10:43 EDT
(In reply to comment #35)
> I have several problems to perform more extensive testings, the biggest being
> lack of time. As to test with a particular password, unfortunately the password
> is not under my control: the remote service we use with curl is a third party
> commercial service that manages its own access info. I may request a new
> password but I won't be able to get one with a specific format.  

I understand. You can subscribe yourself to the bug 500180 and watch its progress. I am pretty sure you're affected by the PEM reader bug (nss package).
Comment 37 Bernard Fouché 2009-05-13 11:49:15 EDT
I've just checked, the password we must use is 10 characters long, so you may be right! I've subscribed to bug #500180 and will perform new tests when a fixed version of nss is made available.
Comment 38 Kamil Dudka 2009-05-17 04:03:11 EDT
(In reply to comment #37)
> I've just checked, the password we must use is 10 characters long, so you may
> be right! I've subscribed to bug #500180 and will perform new tests when a
> fixed version of nss is made available.  

The infinite loop bug (from the original report) should be resolved by the updates listed above, therefore closing CURRENTRELEASE. For other NSS related (lib)curl bugs is now opened the tracker bug 501138. Thanks for you assistance with tracking down the 8 characters long passwords issue!

Note You need to log in before you can comment on or make changes to this bug.