Bug 1610998 - libcurl/curl >= 7.29.0-47 breaks yum update functionality on EC2 RHEL instances (401 unauthorized)
Summary: libcurl/curl >= 7.29.0-47 breaks yum update functionality on EC2 RHEL instanc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nss-pem
Version: 7.6
Hardware: x86_64
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Kamil Dudka
QA Contact: Pavlina Bartikova
URL:
Whiteboard:
Depends On:
Blocks: 1510247
TreeView+ depends on / blocked
 
Reported: 2018-08-01 20:40 UTC by Derek Whatley
Modified: 2018-10-30 12:44 UTC (History)
11 users (show)

Fixed In Version: nss-pem-1.0.3-5.el7
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-30 10:40:37 UTC


Attachments (Terms of Use)
proposed fix (7.70 KB, patch)
2018-08-03 15:22 UTC, Kamil Dudka
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3157 None None None 2018-10-30 10:42:09 UTC

Description Derek Whatley 2018-08-01 20:40:27 UTC
Description of problem:
------------------------
Installing libcurl >= 7.29.0-47 on a RHEL 7 EC2 instance causes failure during the yum repo metadata retrieval process (e.g. yum clean all && yum repolist all) under certain conditions.

This doesn't happen on stock RHEL 7 EC2 instances, but after adding certain additional repos (RH-internal repos containing packages for OpenShift, RHEL, Ceph, etc) the `yum repolist all` process will fail while attempting to pull repo metadata from RHUI AWS repos with `401 unauthorized`. This only happens with libcurl > 7.29.0-47 also installed.

I've isolated the issue to libcurl-7.29.0-47 by updating one package at a time until the issue starts occurring. The issue does _not_ occur in 7.29.0-46, but starts in 47.

All of the RH-internal repos that cause this problem share the same `sslclientcert` and `sslclientkey` in their yum configuration files.

Pulling yum repo metadata for _only_ AWS RHUI repos or _only_ RH-internal repos in separate transactions eliminates the problem.


I'm very willing to gather any additional debugging info needed, please just let me know how to get it and I'll do so.


Version-Release number of selected component (if applicable):
--------------------------------------------------------------
libcurl >= 7.29.0-47

How reproducible:
------------------
Every time.

Steps to Reproduce:
--------------------
1. Launch a RHEL 7.5 EC2 instance 
2. Run `yum clean all && yum repolist all` to verify there is no issue

3. Install RH-internal repos (e.g. rhel-7-ceph)
4. Run `yum clean all && yum repolist all` to verify there is no issue

5. Install libcurl >= 7.29.0-47
6. Run `yum clean all && yum repolist all`. Should see 401 unauthorized when trying to pull RHUI AWS repo metadata.

Actual results:
----------------
401 unauthorized


Expected results:
------------------
pulls repo metadata successfully


Additional info (Logs):
-----------------
=================================================================
================== libcurl 7.29.0-46 (WORKS) ==================== 
=================================================================

[root@ip-172-32-0-247 yum.repos.d]# rpm -qa | grep libcurl
libcurl-7.29.0-46.el7.x86_64


[root@ip-172-32-0-247 yum.repos.d]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
              : rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph                                                                       | 3.8 kB  00:00:00     
rhui-REGION-client-config-server-7                                                | 2.9 kB  00:00:00     
rhui-REGION-rhel-server-releases                                                  | 3.5 kB  00:00:00     
rhui-REGION-rhel-server-rh-common                                                 | 3.8 kB  00:00:00     
(1/9): rhui-REGION-client-config-server-7/x86_64/primary_db                       | 2.5 kB  00:00:00     
(2/9): rhui-REGION-rhel-server-releases/7Server/x86_64/group                      | 855 kB  00:00:00     
(3/9): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo                 | 2.9 MB  00:00:00     
(4/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo                |  33 kB  00:00:00     
(5/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group                     |  104 B  00:00:00     
(6/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db                | 121 kB  00:00:00     
(7/9): rhel-7-ceph/group_gz                                                       |  464 B  00:00:00   
(8/9): rhel-7-ceph/primary_db                                                     |  29 kB  00:00:00     
(9/9): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db                 |  55 MB  00:00:00     
repo id                                          repo name                                         status
rhel-7-ceph                                      rhel-7-ceph-2 Latest Enterprise RPMs                  31
rhui-REGION-client-config-server-7/x86_64        Red Hat Update Infrastructure 2.0 Client Configur      1
rhui-REGION-rhel-server-releases/7Server/x86_64  Red Hat Enterprise Linux Server 7 (RPMs)          20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs    233
repolist: 20,969
[root@ip-172-32-0-247 yum.repos.d]# 


=====================================================================
================== libcurl >= 7.29.0-47 (BROKEN) ==================== 
=====================================================================

[root@ip-172-32-0-247 ec2-user]# rpm -qa | grep libcurl
libcurl-7.29.0-47.el7.x86_64


[root@ip-172-32-0-247 ec2-user]# yum clean all && yum repolist
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases
              : rhui-REGION-rhel-server-rh-common
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
rhel-7-ceph                                                                       | 3.8 kB  00:00:00     
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhel-7-ceph/primary_db                                                            |  29 kB  00:00:00     
https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized
Trying other mirror.
rhui-REGION-rhel-server-releases/7Server/x86_64                                   | 3.5 kB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/group                             | 855 kB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo                        | 2.9 MB  00:00:00     
rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db                        |  55 MB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64                                  | 3.8 kB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/group                            |  104 B  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo                       |  33 kB  00:00:00     
rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db                       | 121 kB  00:00:00     
repo id                                          repo name                                         status
rhel-7-ceph                                      rhel-7-ceph-2 Latest Enterprise RPMs                  31
rhui-REGION-client-config-server-7/x86_64        Red Hat Update Infrastructure 2.0 Client Configur      0
rhui-REGION-rhel-server-releases/7Server/x86_64  Red Hat Enterprise Linux Server 7 (RPMs)          20,704
rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs    233
repolist: 20,968
[root@ip-172-32-0-247 ec2-user]#

Comment 2 Kamil Dudka 2018-08-02 11:06:16 UTC
Thank you for analyzing the problem and finding the cause!  I am pretty sure, this is caused by the fix for bug #1510247.  My guess is that it prematurely removes a key object shared by multiple connections.

Comment 7 Kamil Dudka 2018-08-03 15:19:58 UTC
The fix for bug #1510247 revealed multiple hidden bugs in nss-pem.  The scenario where nss-pem breaks is following:

1. load client cert #1 and the corresponding private key

2. unload private key only (because client cert is referred by session cache)

3. load client cert #2 and the corresponding private key

4. attempt to reuse cert #1 and reload the corresponding private key

5. PK11_FindPrivateKeyFromCert() fails in the SelectClientCert() callback

The step 2. did not exist until bug #1510247 was fixed.  The actual breakage happens in step 4. because the reused certificate still refers to the originally loaded private key, which has been removed in step 2.

In order to fix this, the reference needs to be updated while reusing the certificate object such that it refers to the private key that is going to be loaded by the subsequent call to PK11_CreateManagedGenericObject().

Comment 8 Kamil Dudka 2018-08-03 15:22:55 UTC
Created attachment 1473033 [details]
proposed fix

Comment 21 Pavlina Bartikova 2018-08-28 10:58:30 UTC
Tested with RHEL-7.6_HVM_BETA-20180814-x86_64-0-Access2-GP2 (ami-011349ad9596eb082) AMI, eu-west-1 region.

I had an instance with nss-pem-1.0.3-4.el7.x86_64. When I attached RH repositories from subscription manager, yum repolist ended with the [Errno 14] HTTPS Error 401 - Unauthorized error. After updating to nss-pem-1.0.3-5.el7.x86_64, yum repolist ended without any errors.

Moving bug to VERIFIED.

Comment 23 errata-xmlrpc 2018-10-30 10:40:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3157


Note You need to log in before you can comment on or make changes to this bug.