RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1659108 - performance regression in libcurl caused by the use of PK11_CreateManagedGenericObject()
Summary: performance regression in libcurl caused by the use of PK11_CreateManagedGene...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss-pem
Version: 7.6
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: Daniel Rusek
URL:
Whiteboard:
Depends On:
Blocks: 1667121
TreeView+ depends on / blocked
 
Reported: 2018-12-13 15:26 UTC by fbh
Modified: 2019-08-06 13:01 UTC (History)
9 users (show)

Fixed In Version: nss-pem-1.0.3-6.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1667121 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:01:20 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2175 0 None None None 2019-08-06 13:01:32 UTC

Description fbh 2018-12-13 15:26:39 UTC 
Description of problem:
When setting up a PHP job that runs and loops calls to the internet over ssl, it starts to use more and more time and memory for memory allocation the more we often we open the pem file and connect to the internet. It's clear from traces (strace) that between reading the PEM cert and connecting to the internet the process starts after a while to use more and more time for memory allocation and the process starts slowing down over time. Often to become over 500% slower to handle each connection + request going to the internet.

Downgrading curl and libcurl to version 7.29.0-46 from version 7.29.0-51 resolves the issue as its no longer using the PL11_CreateManagedGenericObject() function in nss

Version-Release number of selected component (if applicable):
Curl and libcurl version 7.29.0-51

How reproducible:


Steps to Reproduce:
1. Create a php script that posts to google.com (could be any API call over ssl)

<?php
error_reporting(E_ALL^E_NOTICE);
echo "starting...".PHP_EOL;
echo PHP_EOL;

$handle = null;
$handlestop = null;


while(true) {
    $a = webcall();
    usleep(1000000);
}

function webcall() {
    $url = 'https://www.google.com';
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $result = curl_exec($ch);
    $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);
    if ($httpcode == 200) // (no content) == event success
        return true; 
    return false;
}

2. run script eg: php test.php

3. Strace script: strace -p <PidOfScript>

Actual results:
Initial start:

stat("/etc/pki/tls/certs/ca-bundle.crt", {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
read(7, "# ACCVRAIZ1\n-----BEGIN CERTIFICA"..., 211658) = 211658
close(7) = 0
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("216.58.211.132")}, [16]) = 0
close(6) = 0

Second phase:
stat("/etc/pki/tls/certs/ca-bundle.crt", {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
read(7, "# ACCVRAIZ1\n-----BEGIN CERTIFICA"..., 211658) = 211658
close(7) = 0
brk(NULL) = 0x5614d5a8d000
brk(NULL) = 0x5614d5a8d000
brk(0x5614d5a18000) = 0x5614d5a18000
brk(NULL) = 0x5614d5a18000
brk(NULL) = 0x5614d5a18000
brk(0x5614d5a4f000) = 0x5614d5a4f000
brk(NULL) = 0x5614d5a4f000
brk(0x5614d5a88000) = 0x5614d5a88000
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("216.58.211.132")}, [16]) = 0
close(6) 

Totally broken phase:
stat("/etc/pki/tls/certs/ca-bundle.crt", {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0444, st_size=211658, ...}) = 0
read(7, "# ACCVRAIZ1\n-----BEGIN CERTIFICA"..., 211658) = 211658
close(7) = 0
brk(NULL) = 0x5614d5ae4000
brk(NULL) = 0x5614d5ae4000
brk(0x5614d5a1a000) = 0x5614d5a1a000
brk(NULL) = 0x5614d5a1a000
brk(NULL) = 0x5614d5a1a000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab0000) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(NULL) = 0x5614d5ab0000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7e000) = 0x5614d5a7e000
brk(NULL) = 0x5614d5a7e000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4c000) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(NULL) = 0x5614d5a4c000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab1000) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(NULL) = 0x5614d5ab1000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a7f000) = 0x5614d5a7f000
brk(NULL) = 0x5614d5a7f000
brk(0x5614d5ab2000) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(NULL) = 0x5614d5ab2000
brk(0x5614d5a4d000) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(NULL) = 0x5614d5a4d000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
brk(0x5614d5ab3000) = 0x5614d5ab3000
brk(NULL) = 0x5614d5ab3000
brk(0x5614d5ae5000) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(NULL) = 0x5614d5ae5000
brk(0x5614d5a80000) = 0x5614d5a80000
brk(NULL) = 0x5614d5a80000
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("216.58.211.132")}, [16]) = 0
close(6) 

Expected results:
We don't excpect brk(NULL) and brk(0x111111111111) memory allocation calls between reading the PEM/CRT file and opening the connection over https.

Additional info:
This coincides with the change to use PK11_CreateManagedGenericObject() in NSS in bug 1510247: https://www.redhat.com/archives/rhsa-announce/2018-October/msg00052.html
Comment 2 Kamil Dudka 2018-12-13 16:36:23 UTC 
The only reason why libcurl started to use PK11_CreateManagedGenericObject() in RHEL-7.6 was to reduce the memory consumption.  It was also successfully verified that the memory footprint was reduced in the most common scenario.  Of course, some unexpected side effects might have happened.  I will have a look at your reproducer.

As a workaround, you can use CA bundle with fewer certificates inside, or avoid loading certificates from files at all and use the native NSS database instead.
Comment 3 fbh 2018-12-13 20:50:13 UTC 
Thanks for the update.

As the issue looks to be an over time issue I'm not sure loading a smaller crt/pem file will solve the issue. This was found in long running jobs that after only running for minuttets would start showing this behaviour. I posted the small php script as an example to reproduce. Our main job does more than this as well but the smaller script was easier for reproducibility. I'm not sure a smaller file do anything more than delay the issue.

For now our workaround has been to roll back libcurl and curl to the previous version and excluded them from being updated by yum and yum-cron.
Comment 4 Kamil Dudka 2018-12-18 16:54:05 UTC 
I tried your PHP reproducer against a local TLS server and the system CA bundle.  I cannot confirm the increase of memory usage -- the memory usage is actually lower after the update of libcurl.  However, I was able to observe increase of CPU usage after the update.

I suspect that it is caused by the array in nss-pem that holds pointers to all allocated PKCS #11 objects.  The array is searched sequentially and grows over the time.  The array has always been there but it probably grows faster now when PK11_CreateManagedGenericObject() is used.  I will need to analyze the root cause and optimize it somehow in a backward-compatible way.

> I'm not sure a smaller file do anything more than delay the issue.

Yes, it was suggested as a workaround.

> For now our workaround has been to roll back libcurl and curl to the previous version and excluded them from being updated by yum and yum-cron.

Good until there is a security update of (lib)curl, which I would not recommend to exclude.
Comment 6 fbh 2018-12-19 08:56:41 UTC 
We can confirm that the memory issue is possibly not the elephant in the room in terms of what caused issues and made us raise the issue. It was sparked by an extremely slow running job that used libcurl that we were debugging on and noticed the long time used post reading the PEM cert end pre opening the connection to the web service.

We have made a new script that much faster reproduces the issue and also confirms the heavily increase CPU usage.

Before running the script run:

yum install httpd mod_ssl -y
systemctl start httpd

This starts a local http server that we can use to test against.

Script:

<?php
error_reporting(E_ALL^E_NOTICE);
echo "starting...".PHP_EOL;
echo PHP_EOL;

$handle = null;
$handlestop = null;
$sec = time();
$total = 0;

while(true) {
  if($sec < time()) {
    $sec = time();
    echo "$counter HTTPS calls per second. Total $total calls.  My CPU usage: " . my_cpu_usage() . "% \n";
    $counter = 0;
  } else {
    $total++;
    webcall();
    $counter++;
  }
}


function webcall() {
    $url = 'https://127.0.0.1/test';
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $result = curl_exec($ch);
    $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);
    if ($httpcode == 200) // (no content) == event success
        return true;
    return false;
}

function my_cpu_usage() {
  exec('ps aux  |awk \'$2 == "' . getmypid() . '" { print $3 }\'', $out);
  return current($out);
}

After a short while, approximately 70 calls we see the issue occur and and CPU usage starts to rise until it maxes out the entire CPU.

We appreciate the further investigation into the issue and will stand by if there is anything else we need to provide.
As for the job runner that uses libcurl and the CPU consumption, we won't be able to upgrade to the newer libcurl even if further security updates happen as the job runners won't be able to keep up with realtime data handling if the CPU issue is still a factor in an upgraded libcurl.
Comment 7 George Machitidze 2018-12-20 16:28:40 UTC 
I confirm, this caused issues with 3rd party applications, like in my case it made Zabbix completely useless with web monitoring:
https://support.zabbix.com/browse/ZBX-15312
Comment 8 Kamil Dudka 2018-12-20 16:46:23 UTC 
I have confirmed my hypothesis.  With the old version of libcurl, the global array in nss-pem stays of a fixed size (when loading CA certificates only) whereas, with the new version, the array grows unboundedly.  Luckily, all the unnecessarily allocated array elements are NULL pointers.  So I will try to replace the array by a linked list to eliminate this waste.
Comment 9 Kamil Dudka 2018-12-20 17:50:10 UTC 
I have a work-in-progress version of the fix for nss-pem, will be able to share it tomorrow hopefully...

Would anyone be able to test an experimental build of nss-pem with the fix included (together with libcurl-7.29.0-51.el7)?
Comment 10 Kamil Dudka 2018-12-21 14:08:54 UTC 
upstream pull-request: https://github.com/kdudka/nss-pem/pull/2

copr with experimental build of nss-pem that contains the fix for this bug: https://copr.fedorainfracloud.org/coprs/kdudka/nss-pem/

Looking forward for your feedback!
Comment 11 fbh 2018-12-21 15:54:42 UTC 
Updates nss-pem with the package: 

 nss-pem      x86_64     1.0.4.20181221.153303.gbc177c0.gl_list-1.el7       kdudka-nss-pem                   72 k


I can confirm that it appears that the bug has been resolved. 
The test script now runs continuously with the same amount of requests per second (way more than before) and does not consume more CPU than it should. 

Test performed over 5 minutes. Using the old script everything would be stalled after that amount of time.
Comment 12 Kamil Dudka 2018-12-21 16:08:06 UTC 
Perfect.  Thank you for testing the package!
Comment 13 Kamil Dudka 2019-01-02 11:05:56 UTC 
upstream commits:

https://github.com/kdudka/nss-pem/commit/8fc25a6b
https://github.com/kdudka/nss-pem/commit/5a14b625
https://github.com/kdudka/nss-pem/commit/188c63f7
Comment 15 David Chiluk 2019-01-07 16:19:58 UTC 
What does the timeline look like for getting this released to Rhel/Centos 7 repos?

Also thanks Kamil for the hard work.
Comment 16 Kamil Dudka 2019-01-07 16:36:51 UTC 
I am doing my best to get the fix out ASAP.  The timeline does not depend on me though.  If you are a customer of Red Hat, please open a customer case for this.  It would help to prioritize the bug.  Sorry for the troubles!
Comment 17 David Chiluk 2019-01-09 21:17:32 UTC 
We have also now tested the proposed package and confirm that it resolves our issues as well.
Comment 25 David Chiluk 2019-01-31 19:25:14 UTC 
What is the timeline for getting this released?
Comment 26 Kamil Dudka 2019-01-31 21:44:51 UTC 
(In reply to David Chiluk from comment #25)
> What is the timeline for getting this released?

We strive hard to get the fix out as soon as possible.  No timeline guarantees via Bugzilla though...
Comment 29 Kamil Dudka 2019-03-14 08:15:20 UTC 
This bug has been fixed in nss-pem-1.0.3-5.el7_6.1:

    https://access.redhat.com/errata/RHBA-2019:0500
Comment 31 errata-xmlrpc 2019-08-06 13:01:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2175
Note You need to log in before you can comment on or make changes to this bug.