RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1572438 - libcurl w/ ssl (nss) consumes memory
Summary: libcurl w/ ssl (nss) consumes memory
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss-pem
Version: 7.7-Alt
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks: 1643104
TreeView+ depends on / blocked
 
Reported: 2018-04-27 02:22 UTC by Marcus Watts
Modified: 2022-03-13 14:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-19 13:38:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
source to test program (10.42 KB, text/x-csrc)
2018-04-27 02:22 UTC, Marcus Watts
no flags Details
massif output - rhel7 w/ libcurl-7.29.0-47.el7.x86_64 + nss-3.34.0-4.el7.x86_64 (510.61 KB, text/plain)
2018-04-27 02:32 UTC, Marcus Watts
no flags Details
massif output: xenial w/ curl+openssl. (192.57 KB, text/plain)
2018-04-27 02:43 UTC, Marcus Watts
no flags Details
massif output: xenial w/ curl+gnutls. (488.16 KB, text/plain)
2018-04-27 02:56 UTC, Marcus Watts
no flags Details

Description Marcus Watts 2018-04-27 02:22:38 UTC
Created attachment 1427442 [details]
source to test program

Description of problem:


Version-Release number of selected component (if applicable):
 tested with:
  libcurl-7.29.0-47.el7.x86_64
  nss-3.34.0-4.el7.x86_64

How reproducible:


Steps to Reproduce:
1. setup a local web server w/ ssl.
2. setup rhel instance w/ c++ compiler, libcurl-devel, valgrind
3. c++ -g -o curl5 curl5.cc -lcurl -lpthread -std=c++11
4. valgrind  --massif-out-file=/tmp/t.%p curl5 -t 1 -m 1 1024 https:<<your-web-servers-fqdn>>
5. ms_print /tmp/t.* >/tmp/out
6. sed 30q /tmp/out

Actual results:

 chart showing ramp up of memory usage - from about 2.5m to about 5m.

Expected results:

 chart showing bounded flat memory usage - all peaks < 1.5m.

Additional info:

Comment 2 Marcus Watts 2018-04-27 02:32:34 UTC
Created attachment 1427444 [details]
massif output - rhel7 w/ libcurl-7.29.0-47.el7.x86_64 + nss-3.34.0-4.el7.x86_64

This is the full output I got using massif with rhel7, libcurl+nss.  This output includes the memory ramp described, plus much more detailed accountings of memory usage at various points during execution.  You have to install the appropriate debug information to get all this detail.

Comment 3 Marcus Watts 2018-04-27 02:43:11 UTC
Created attachment 1427446 [details]
massif output: xenial w/ curl+openssl.

xenial ships with 3 different versions of libcurl.  Each
is linked with a different soname so runtimes for all 3
can be loaded at the same time.  Which version is used is selected at at compile time for the application by which version of the dev library is present.

This run:
libcurl3:amd64: 7.47.0-1ubuntu2.7
libssl1.0.0:amd64: 1.0.2g-1ubuntu4.12

This shows the flat memory usage I expect to see.

Comment 4 Marcus Watts 2018-04-27 02:56:48 UTC
Created attachment 1427462 [details]
massif output: xenial w/ curl+gnutls.

This is output from xenial with curl+gnutls.  This is only included to document that this combination is less desirable.

libcurl3-gnutls:amd64   7.47.0-1ubuntu2.7
libgnutls30:amd64       3.4.10-4ubuntu1.4

This has flat bounded output.  However, the peaks are far larger than for openssl.  I haven't bothered to document execution time for this bug report, but execution time is very noticeably slower with gnutls.

I'm not going to bother with the xenial libcurl+nss combination, because that doesn't look for ca certs the same way as gnutls or openssl and doesn't have a good workflow for adding new ca certs.

Comment 6 Kamil Dudka 2018-04-27 13:34:52 UTC
Thank you for providing the reproducer!  I can confirm that libcurl-7.29.0-47.el7 consumes less memory than libcurl-7.29.0-46.el7.  The remaining memory waste is not in libcurl but in nss-pem.  It is a known issue.

nss-pem is used to load CA certificates from PEM files.  The memory consumption grows per CA cert (re)load (and they are reloaded per each connection).  Hence you can significantly reduce the memory consumption by giving libcurl fewer CA certificates from PEM file, or even make the memory consumption constant by not loading CA certificates from PEM file at all.

Could something like this be used as a workaround?

--- a/curl5.cc
+++ b/curl5.cc
@@ -273,7 +273,7 @@ doit(int id, struct doit_stats *ds)
  if (Vflag) {
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
  }
- if (capath)
+ //if (capath)
    curl_easy_setopt(curl, CURLOPT_CAINFO, capath);
  if (clock_gettime(CLOCK_MONOTONIC_RAW, ts) < 0) {
    std::cerr << "gettime failed " << errno << std::endl;

$ c++ -g -o curl5 curl5.cc -lcurl -lpthread -std=c++11
$ mkdir $HOME/nssdb
$ certutil -d sql:$HOME/nssdb -N
$ certutil -d sql:$HOME/nssdb -A -n cacert -t C,, <ca-cert.pem
$ export SSL_DIR=$HOME/nssdb
$ valgrind --tool=massif --massif-out-file=/tmp/t.%p curl5 -t 1 -m 1 1024 https://...

Comment 13 Kamil Dudka 2018-11-19 13:38:51 UTC
There is no known fix for this bug.  nss-pem is already in maintenance mode because it is going to be substituted by OpenSSL in RHEL-8.  Closing as WONTFIX.

Note that the memory consumption of (lib)curl running on top of NSS was already significantly improved in RHEL-7.6 per bug #1510247.

Comment 14 Red Hat Bugzilla Rules Engine 2018-11-19 13:38:56 UTC
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.


Note You need to log in before you can comment on or make changes to this bug.