Bug 1572438 - libcurl w/ ssl (nss) consumes memory
Summary: libcurl w/ ssl (nss) consumes memory
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss-pem
Version: 7.7-Alt
Hardware: x86_64
OS: Linux
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: BaseOS QE Security Team
Depends On:
Blocks: 1643104
TreeView+ depends on / blocked
Reported: 2018-04-27 02:22 UTC by Marcus Watts
Modified: 2019-06-11 20:22 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-11-19 13:38:56 UTC
Target Upstream Version:

Attachments (Terms of Use)
source to test program (10.42 KB, text/x-csrc)
2018-04-27 02:22 UTC, Marcus Watts
no flags Details
massif output - rhel7 w/ libcurl-7.29.0-47.el7.x86_64 + nss-3.34.0-4.el7.x86_64 (510.61 KB, text/plain)
2018-04-27 02:32 UTC, Marcus Watts
no flags Details
massif output: xenial w/ curl+openssl. (192.57 KB, text/plain)
2018-04-27 02:43 UTC, Marcus Watts
no flags Details
massif output: xenial w/ curl+gnutls. (488.16 KB, text/plain)
2018-04-27 02:56 UTC, Marcus Watts
no flags Details

Description Marcus Watts 2018-04-27 02:22:38 UTC
Created attachment 1427442 [details]
source to test program

Description of problem:

Version-Release number of selected component (if applicable):
 tested with:

How reproducible:

Steps to Reproduce:
1. setup a local web server w/ ssl.
2. setup rhel instance w/ c++ compiler, libcurl-devel, valgrind
3. c++ -g -o curl5 curl5.cc -lcurl -lpthread -std=c++11
4. valgrind  --massif-out-file=/tmp/t.%p curl5 -t 1 -m 1 1024 https:<<your-web-servers-fqdn>>
5. ms_print /tmp/t.* >/tmp/out
6. sed 30q /tmp/out

Actual results:

 chart showing ramp up of memory usage - from about 2.5m to about 5m.

Expected results:

 chart showing bounded flat memory usage - all peaks < 1.5m.

Additional info:

Comment 2 Marcus Watts 2018-04-27 02:32:34 UTC
Created attachment 1427444 [details]
massif output - rhel7 w/ libcurl-7.29.0-47.el7.x86_64 + nss-3.34.0-4.el7.x86_64

This is the full output I got using massif with rhel7, libcurl+nss.  This output includes the memory ramp described, plus much more detailed accountings of memory usage at various points during execution.  You have to install the appropriate debug information to get all this detail.

Comment 3 Marcus Watts 2018-04-27 02:43:11 UTC
Created attachment 1427446 [details]
massif output: xenial w/ curl+openssl.

xenial ships with 3 different versions of libcurl.  Each
is linked with a different soname so runtimes for all 3
can be loaded at the same time.  Which version is used is selected at at compile time for the application by which version of the dev library is present.

This run:
libcurl3:amd64: 7.47.0-1ubuntu2.7
libssl1.0.0:amd64: 1.0.2g-1ubuntu4.12

This shows the flat memory usage I expect to see.

Comment 4 Marcus Watts 2018-04-27 02:56:48 UTC
Created attachment 1427462 [details]
massif output: xenial w/ curl+gnutls.

This is output from xenial with curl+gnutls.  This is only included to document that this combination is less desirable.

libcurl3-gnutls:amd64   7.47.0-1ubuntu2.7
libgnutls30:amd64       3.4.10-4ubuntu1.4

This has flat bounded output.  However, the peaks are far larger than for openssl.  I haven't bothered to document execution time for this bug report, but execution time is very noticeably slower with gnutls.

I'm not going to bother with the xenial libcurl+nss combination, because that doesn't look for ca certs the same way as gnutls or openssl and doesn't have a good workflow for adding new ca certs.

Comment 6 Kamil Dudka 2018-04-27 13:34:52 UTC
Thank you for providing the reproducer!  I can confirm that libcurl-7.29.0-47.el7 consumes less memory than libcurl-7.29.0-46.el7.  The remaining memory waste is not in libcurl but in nss-pem.  It is a known issue.

nss-pem is used to load CA certificates from PEM files.  The memory consumption grows per CA cert (re)load (and they are reloaded per each connection).  Hence you can significantly reduce the memory consumption by giving libcurl fewer CA certificates from PEM file, or even make the memory consumption constant by not loading CA certificates from PEM file at all.

Could something like this be used as a workaround?

--- a/curl5.cc
+++ b/curl5.cc
@@ -273,7 +273,7 @@ doit(int id, struct doit_stats *ds)
  if (Vflag) {
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
- if (capath)
+ //if (capath)
    curl_easy_setopt(curl, CURLOPT_CAINFO, capath);
  if (clock_gettime(CLOCK_MONOTONIC_RAW, ts) < 0) {
    std::cerr << "gettime failed " << errno << std::endl;

$ c++ -g -o curl5 curl5.cc -lcurl -lpthread -std=c++11
$ mkdir $HOME/nssdb
$ certutil -d sql:$HOME/nssdb -N
$ certutil -d sql:$HOME/nssdb -A -n cacert -t C,, <ca-cert.pem
$ export SSL_DIR=$HOME/nssdb
$ valgrind --tool=massif --massif-out-file=/tmp/t.%p curl5 -t 1 -m 1 1024 https://...

Comment 13 Kamil Dudka 2018-11-19 13:38:51 UTC
There is no known fix for this bug.  nss-pem is already in maintenance mode because it is going to be substituted by OpenSSL in RHEL-8.  Closing as WONTFIX.

Note that the memory consumption of (lib)curl running on top of NSS was already significantly improved in RHEL-7.6 per bug #1510247.

Comment 14 Red Hat Bugzilla Rules Engine 2018-11-19 13:38:56 UTC
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.

Note You need to log in before you can comment on or make changes to this bug.