Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
------------------------
Installing libcurl >= 7.29.0-47 on a RHEL 7 EC2 instance causes failure during the yum repo metadata retrieval process (e.g. yum clean all && yum repolist all) under certain conditions.
This doesn't happen on stock RHEL 7 EC2 instances, but after adding certain additional repos (RH-internal repos containing packages for OpenShift, RHEL, Ceph, etc) the `yum repolist all` process will fail while attempting to pull repo metadata from RHUI AWS repos with `401 unauthorized`. This only happens with libcurl > 7.29.0-47 also installed.
I've isolated the issue to libcurl-7.29.0-47 by updating one package at a time until the issue starts occurring. The issue does _not_ occur in 7.29.0-46, but starts in 47.
All of the RH-internal repos that cause this problem share the same `sslclientcert` and `sslclientkey` in their yum configuration files.
Pulling yum repo metadata for _only_ AWS RHUI repos or _only_ RH-internal repos in separate transactions eliminates the problem.
I'm very willing to gather any additional debugging info needed, please just let me know how to get it and I'll do so.
Version-Release number of selected component (if applicable):
--------------------------------------------------------------
libcurl >= 7.29.0-47
How reproducible:
------------------
Every time.
Steps to Reproduce:
--------------------
1. Launch a RHEL 7.5 EC2 instance
2. Run `yum clean all && yum repolist all` to verify there is no issue
3. Install RH-internal repos (e.g. rhel-7-ceph)
4. Run `yum clean all && yum repolist all` to verify there is no issue
5. Install libcurl >= 7.29.0-47
6. Run `yum clean all && yum repolist all`. Should see 401 unauthorized when trying to pull RHUI AWS repo metadata.
Actual results:
----------------
401 unauthorized
Expected results:
------------------
pulls repo metadata successfully
Additional info (Logs):
-----------------
=================================================================
================== libcurl 7.29.0-46 (WORKS) ====================
=================================================================
[root@ip-172-32-0-247 yum.repos.d]# rpm -qa | grep libcurl
libcurl-7.29.0-46.el7.x86_64
[root@ip-172-32-0-247 yum.repos.d]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
: rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph | 3.8 kB 00:00:00
rhui-REGION-client-config-server-7 | 2.9 kB 00:00:00
rhui-REGION-rhel-server-releases | 3.5 kB 00:00:00
rhui-REGION-rhel-server-rh-common | 3.8 kB 00:00:00
(1/9): rhui-REGION-client-config-server-7/x86_64/primary_db | 2.5 kB 00:00:00
(2/9): rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00
(3/9): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00
(4/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00
(5/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00
(6/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00
(7/9): rhel-7-ceph/group_gz | 464 B 00:00:00
(8/9): rhel-7-ceph/primary_db | 29 kB 00:00:00
(9/9): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00
repo id repo name status
rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31
rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 1
rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233
repolist: 20,969
[root@ip-172-32-0-247 yum.repos.d]#
=====================================================================
================== libcurl >= 7.29.0-47 (BROKEN) ====================
=====================================================================
[root@ip-172-32-0-247 ec2-user]# rpm -qa | grep libcurl
libcurl-7.29.0-47.el7.x86_64
[root@ip-172-32-0-247 ec2-user]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
: rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph | 3.8 kB 00:00:00
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhel-7-ceph/primary_db | 29 kB 00:00:00
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhui-REGION-rhel-server-releases/7Server/x86_64 | 3.5 kB 00:00:00
rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00
rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00
rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00
rhui-REGION-rhel-server-rh-common/7Server/x86_64 | 3.8 kB 00:00:00
rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00
rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00
rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00
repo id repo name status
rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31
rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 0
rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233
repolist: 20,968
[root@ip-172-32-0-247 ec2-user]#
Thank you for analyzing the problem and finding the cause! I am pretty sure, this is caused by the fix for bug #1510247. My guess is that it prematurely removes a key object shared by multiple connections.
The fix for bug #1510247 revealed multiple hidden bugs in nss-pem. The scenario where nss-pem breaks is following:
1. load client cert #1 and the corresponding private key
2. unload private key only (because client cert is referred by session cache)
3. load client cert #2 and the corresponding private key
4. attempt to reuse cert #1 and reload the corresponding private key
5. PK11_FindPrivateKeyFromCert() fails in the SelectClientCert() callback
The step 2. did not exist until bug #1510247 was fixed. The actual breakage happens in step 4. because the reused certificate still refers to the originally loaded private key, which has been removed in step 2.
In order to fix this, the reference needs to be updated while reusing the certificate object such that it refers to the private key that is going to be loaded by the subsequent call to PK11_CreateManagedGenericObject().
Comment 21Pavlina Bartikova
2018-08-28 10:58:30 UTC
Tested with RHEL-7.6_HVM_BETA-20180814-x86_64-0-Access2-GP2 (ami-011349ad9596eb082) AMI, eu-west-1 region.
I had an instance with nss-pem-1.0.3-4.el7.x86_64. When I attached RH repositories from subscription manager, yum repolist ended with the [Errno 14] HTTPS Error 401 - Unauthorized error. After updating to nss-pem-1.0.3-5.el7.x86_64, yum repolist ended without any errors.
Moving bug to VERIFIED.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2018:3157
Description of problem: ------------------------ Installing libcurl >= 7.29.0-47 on a RHEL 7 EC2 instance causes failure during the yum repo metadata retrieval process (e.g. yum clean all && yum repolist all) under certain conditions. This doesn't happen on stock RHEL 7 EC2 instances, but after adding certain additional repos (RH-internal repos containing packages for OpenShift, RHEL, Ceph, etc) the `yum repolist all` process will fail while attempting to pull repo metadata from RHUI AWS repos with `401 unauthorized`. This only happens with libcurl > 7.29.0-47 also installed. I've isolated the issue to libcurl-7.29.0-47 by updating one package at a time until the issue starts occurring. The issue does _not_ occur in 7.29.0-46, but starts in 47. All of the RH-internal repos that cause this problem share the same `sslclientcert` and `sslclientkey` in their yum configuration files. Pulling yum repo metadata for _only_ AWS RHUI repos or _only_ RH-internal repos in separate transactions eliminates the problem. I'm very willing to gather any additional debugging info needed, please just let me know how to get it and I'll do so. Version-Release number of selected component (if applicable): -------------------------------------------------------------- libcurl >= 7.29.0-47 How reproducible: ------------------ Every time. Steps to Reproduce: -------------------- 1. Launch a RHEL 7.5 EC2 instance 2. Run `yum clean all && yum repolist all` to verify there is no issue 3. Install RH-internal repos (e.g. rhel-7-ceph) 4. Run `yum clean all && yum repolist all` to verify there is no issue 5. Install libcurl >= 7.29.0-47 6. Run `yum clean all && yum repolist all`. Should see 401 unauthorized when trying to pull RHUI AWS repo metadata. Actual results: ---------------- 401 unauthorized Expected results: ------------------ pulls repo metadata successfully Additional info (Logs): ----------------- ================================================================= ================== libcurl 7.29.0-46 (WORKS) ==================== ================================================================= [root@ip-172-32-0-247 yum.repos.d]# rpm -qa | grep libcurl libcurl-7.29.0-46.el7.x86_64 [root@ip-172-32-0-247 yum.repos.d]# yum clean all && yum repolist Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases : rhui-REGION-rhel-server-rh-common Cleaning up everything Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos Loaded plugins: amazon-id, rhui-lb, search-disabled-repos rhel-7-ceph | 3.8 kB 00:00:00 rhui-REGION-client-config-server-7 | 2.9 kB 00:00:00 rhui-REGION-rhel-server-releases | 3.5 kB 00:00:00 rhui-REGION-rhel-server-rh-common | 3.8 kB 00:00:00 (1/9): rhui-REGION-client-config-server-7/x86_64/primary_db | 2.5 kB 00:00:00 (2/9): rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00 (3/9): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00 (4/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00 (5/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00 (6/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00 (7/9): rhel-7-ceph/group_gz | 464 B 00:00:00 (8/9): rhel-7-ceph/primary_db | 29 kB 00:00:00 (9/9): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00 repo id repo name status rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 1 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233 repolist: 20,969 [root@ip-172-32-0-247 yum.repos.d]# ===================================================================== ================== libcurl >= 7.29.0-47 (BROKEN) ==================== ===================================================================== [root@ip-172-32-0-247 ec2-user]# rpm -qa | grep libcurl libcurl-7.29.0-47.el7.x86_64 [root@ip-172-32-0-247 ec2-user]# yum clean all && yum repolist Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases : rhui-REGION-rhel-server-rh-common Cleaning up everything Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos Loaded plugins: amazon-id, rhui-lb, search-disabled-repos rhel-7-ceph | 3.8 kB 00:00:00 https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. rhel-7-ceph/primary_db | 29 kB 00:00:00 https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. rhui-REGION-rhel-server-releases/7Server/x86_64 | 3.5 kB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64 | 3.8 kB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00 repo id repo name status rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 0 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233 repolist: 20,968 [root@ip-172-32-0-247 ec2-user]#