RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1610998 - libcurl/curl >= 7.29.0-47 breaks yum update functionality on EC2 RHEL instances (401 unauthorized)
Summary: libcurl/curl >= 7.29.0-47 breaks yum update functionality on EC2 RHEL instanc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss-pem
Version: 7.6
Hardware: x86_64
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: Pavlina Bartikova
URL:
Whiteboard:
Depends On:
Blocks: 1510247
TreeView+ depends on / blocked
 
Reported: 2018-08-01 20:40 UTC by Derek Whatley
Modified: 2018-10-30 12:44 UTC (History)
11 users (show)

Fixed In Version: nss-pem-1.0.3-5.el7
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-30 10:40:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
proposed fix (7.70 KB, patch)
2018-08-03 15:22 UTC, Kamil Dudka 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3157 0 None None None 2018-10-30 10:42:09 UTC

Description Derek Whatley 2018-08-01 20:40:27 UTC 
Description of problem:
------------------------
Installing libcurl >= 7.29.0-47 on a RHEL 7 EC2 instance causes failure during the yum repo metadata retrieval process (e.g. yum clean all && yum repolist all) under certain conditions.

This doesn't happen on stock RHEL 7 EC2 instances, but after adding certain additional repos (RH-internal repos containing packages for OpenShift, RHEL, Ceph, etc) the `yum repolist all` process will fail while attempting to pull repo metadata from RHUI AWS repos with `401 unauthorized`. This only happens with libcurl > 7.29.0-47 also installed.

I've isolated the issue to libcurl-7.29.0-47 by updating one package at a time until the issue starts occurring. The issue does _not_ occur in 7.29.0-46, but starts in 47.

All of the RH-internal repos that cause this problem share the same `sslclientcert` and `sslclientkey` in their yum configuration files.

Pulling yum repo metadata for _only_ AWS RHUI repos or _only_ RH-internal repos in separate transactions eliminates the problem.


I'm very willing to gather any additional debugging info needed, please just let me know how to get it and I'll do so.


Version-Release number of selected component (if applicable):
--------------------------------------------------------------
libcurl >= 7.29.0-47

How reproducible:
------------------
Every time.

Steps to Reproduce:
--------------------
1. Launch a RHEL 7.5 EC2 instance 
2. Run `yum clean all && yum repolist all` to verify there is no issue

3. Install RH-internal repos (e.g. rhel-7-ceph)
4. Run `yum clean all && yum repolist all` to verify there is no issue

5. Install libcurl >= 7.29.0-47
6. Run `yum clean all && yum repolist all`. Should see 401 unauthorized when trying to pull RHUI AWS repo metadata.

Actual results:
----------------
401 unauthorized


Expected results:
------------------
pulls repo metadata successfully


Additional info (Logs):
-----------------
=================================================================
================== libcurl 7.29.0-46 (WORKS) ==================== 
=================================================================

[root@ip-172-32-0-247 yum.repos.d]# rpm -qa | grep libcurl
libcurl-7.29.0-46.el7.x86_64


[root@ip-172-32-0-247 yum.repos.d]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
              : rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph                                                                       | 3.8 kB  00:00:00     
rhui-REGION-client-config-server-7                                                | 2.9 kB  00:00:00     
rhui-REGION-rhel-server-releases                                                  | 3.5 kB  00:00:00     
rhui-REGION-rhel-server-rh-common                                                 | 3.8 kB  00:00:00     
(1/9): rhui-REGION-client-config-server-7/x86_64/primary_db                       | 2.5 kB  00:00:00     
(2/9): rhui-REGION-rhel-server-releases/7Server/x86_64/group                      | 855 kB  00:00:00     
(3/9): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo                 | 2.9 MB  00:00:00     
(4/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo                |  33 kB  00:00:00     
(5/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group                     |  104 B  00:00:00     
(6/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db                | 121 kB  00:00:00     
(7/9): rhel-7-ceph/group_gz                                                       |  464 B  00:00:00   
(8/9): rhel-7-ceph/primary_db                                                     |  29 kB  00:00:00     
(9/9): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db                 |  55 MB  00:00:00     
repo id                                          repo name                                         status
rhel-7-ceph                                      rhel-7-ceph-2 Latest Enterprise RPMs                  31
rhui-REGION-client-config-server-7/x86_64        Red Hat Update Infrastructure 2.0 Client Configur      1
rhui-REGION-rhel-server-releases/7Server/x86_64  Red Hat Enterprise Linux Server 7 (RPMs)          20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs    233
repolist: 20,969
[root@ip-172-32-0-247 yum.repos.d]# 


=====================================================================
================== libcurl >= 7.29.0-47 (BROKEN) ==================== 
=====================================================================

[root@ip-172-32-0-247 ec2-user]# rpm -qa | grep libcurl
libcurl-7.29.0-47.el7.x86_64


[root@ip-172-32-0-247 ec2-user]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
              : rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph                                                                       | 3.8 kB  00:00:00     
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhel-7-ceph/primary_db                                                            |  29 kB  00:00:00     
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhui-REGION-rhel-server-releases/7Server/x86_64                                   | 3.5 kB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/group                             | 855 kB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo                        | 2.9 MB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db                        |  55 MB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64                                  | 3.8 kB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/group                            |  104 B  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo                       |  33 kB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db                       | 121 kB  00:00:00     
repo id                                          repo name                                         status
rhel-7-ceph                                      rhel-7-ceph-2 Latest Enterprise RPMs                  31
rhui-REGION-client-config-server-7/x86_64        Red Hat Update Infrastructure 2.0 Client Configur      0
rhui-REGION-rhel-server-releases/7Server/x86_64  Red Hat Enterprise Linux Server 7 (RPMs)          20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs    233
repolist: 20,968
[root@ip-172-32-0-247 ec2-user]#
Comment 2 Kamil Dudka 2018-08-02 11:06:16 UTC 
Thank you for analyzing the problem and finding the cause!  I am pretty sure, this is caused by the fix for bug #1510247.  My guess is that it prematurely removes a key object shared by multiple connections.
Comment 7 Kamil Dudka 2018-08-03 15:19:58 UTC 
The fix for bug #1510247 revealed multiple hidden bugs in nss-pem.  The scenario where nss-pem breaks is following:

1. load client cert #1 and the corresponding private key

2. unload private key only (because client cert is referred by session cache)

3. load client cert #2 and the corresponding private key

4. attempt to reuse cert #1 and reload the corresponding private key

5. PK11_FindPrivateKeyFromCert() fails in the SelectClientCert() callback

The step 2. did not exist until bug #1510247 was fixed.  The actual breakage happens in step 4. because the reused certificate still refers to the originally loaded private key, which has been removed in step 2.

In order to fix this, the reference needs to be updated while reusing the certificate object such that it refers to the private key that is going to be loaded by the subsequent call to PK11_CreateManagedGenericObject().
Comment 8 Kamil Dudka 2018-08-03 15:22:55 UTC 
Created attachment 1473033 [details]
proposed fix
Comment 15 Kamil Dudka 2018-08-08 12:50:49 UTC 
upstream commits:

https://github.com/kdudka/nss-pem/commit/1d51c233
https://github.com/kdudka/nss-pem/commit/e85b6f90
https://github.com/kdudka/nss-pem/commit/5e6d9ce0
https://github.com/kdudka/nss-pem/commit/0eafa24f
https://github.com/kdudka/nss-pem/commit/e14465a1
Comment 21 Pavlina Bartikova 2018-08-28 10:58:30 UTC 
Tested with RHEL-7.6_HVM_BETA-20180814-x86_64-0-Access2-GP2 (ami-011349ad9596eb082) AMI, eu-west-1 region.

I had an instance with nss-pem-1.0.3-4.el7.x86_64. When I attached RH repositories from subscription manager, yum repolist ended with the [Errno 14] HTTPS Error 401 - Unauthorized error. After updating to nss-pem-1.0.3-5.el7.x86_64, yum repolist ended without any errors.

Moving bug to VERIFIED.
Comment 23 errata-xmlrpc 2018-10-30 10:40:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3157
Note You need to log in before you can comment on or make changes to this bug.