Bug 1610998
| Summary: | libcurl/curl >= 7.29.0-47 breaks yum update functionality on EC2 RHEL instances (401 unauthorized) | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Derek Whatley <dwhatley> | ||||
| Component: | nss-pem | Assignee: | Kamil Dudka <kdudka> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Pavlina Bartikova <pbartiko> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 7.6 | CC: | andrey.zykov, dwhatley, fsumsal, jboutaud, jmatthew, jmontleo, kdudka, pbartiko, rhack, szidek, wabouham | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | nss-pem-1.0.3-5.el7 | Doc Type: | No Doc Update | ||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-10-30 10:40:37 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1510247 | ||||||
| Attachments: |
|
||||||
Thank you for analyzing the problem and finding the cause! I am pretty sure, this is caused by the fix for bug #1510247. My guess is that it prematurely removes a key object shared by multiple connections. The fix for bug #1510247 revealed multiple hidden bugs in nss-pem. The scenario where nss-pem breaks is following: 1. load client cert #1 and the corresponding private key 2. unload private key only (because client cert is referred by session cache) 3. load client cert #2 and the corresponding private key 4. attempt to reuse cert #1 and reload the corresponding private key 5. PK11_FindPrivateKeyFromCert() fails in the SelectClientCert() callback The step 2. did not exist until bug #1510247 was fixed. The actual breakage happens in step 4. because the reused certificate still refers to the originally loaded private key, which has been removed in step 2. In order to fix this, the reference needs to be updated while reusing the certificate object such that it refers to the private key that is going to be loaded by the subsequent call to PK11_CreateManagedGenericObject(). Created attachment 1473033 [details]
proposed fix
upstream commits: https://github.com/kdudka/nss-pem/commit/1d51c233 https://github.com/kdudka/nss-pem/commit/e85b6f90 https://github.com/kdudka/nss-pem/commit/5e6d9ce0 https://github.com/kdudka/nss-pem/commit/0eafa24f https://github.com/kdudka/nss-pem/commit/e14465a1 Tested with RHEL-7.6_HVM_BETA-20180814-x86_64-0-Access2-GP2 (ami-011349ad9596eb082) AMI, eu-west-1 region. I had an instance with nss-pem-1.0.3-4.el7.x86_64. When I attached RH repositories from subscription manager, yum repolist ended with the [Errno 14] HTTPS Error 401 - Unauthorized error. After updating to nss-pem-1.0.3-5.el7.x86_64, yum repolist ended without any errors. Moving bug to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3157 |
Description of problem: ------------------------ Installing libcurl >= 7.29.0-47 on a RHEL 7 EC2 instance causes failure during the yum repo metadata retrieval process (e.g. yum clean all && yum repolist all) under certain conditions. This doesn't happen on stock RHEL 7 EC2 instances, but after adding certain additional repos (RH-internal repos containing packages for OpenShift, RHEL, Ceph, etc) the `yum repolist all` process will fail while attempting to pull repo metadata from RHUI AWS repos with `401 unauthorized`. This only happens with libcurl > 7.29.0-47 also installed. I've isolated the issue to libcurl-7.29.0-47 by updating one package at a time until the issue starts occurring. The issue does _not_ occur in 7.29.0-46, but starts in 47. All of the RH-internal repos that cause this problem share the same `sslclientcert` and `sslclientkey` in their yum configuration files. Pulling yum repo metadata for _only_ AWS RHUI repos or _only_ RH-internal repos in separate transactions eliminates the problem. I'm very willing to gather any additional debugging info needed, please just let me know how to get it and I'll do so. Version-Release number of selected component (if applicable): -------------------------------------------------------------- libcurl >= 7.29.0-47 How reproducible: ------------------ Every time. Steps to Reproduce: -------------------- 1. Launch a RHEL 7.5 EC2 instance 2. Run `yum clean all && yum repolist all` to verify there is no issue 3. Install RH-internal repos (e.g. rhel-7-ceph) 4. Run `yum clean all && yum repolist all` to verify there is no issue 5. Install libcurl >= 7.29.0-47 6. Run `yum clean all && yum repolist all`. Should see 401 unauthorized when trying to pull RHUI AWS repo metadata. Actual results: ---------------- 401 unauthorized Expected results: ------------------ pulls repo metadata successfully Additional info (Logs): ----------------- ================================================================= ================== libcurl 7.29.0-46 (WORKS) ==================== ================================================================= [root@ip-172-32-0-247 yum.repos.d]# rpm -qa | grep libcurl libcurl-7.29.0-46.el7.x86_64 [root@ip-172-32-0-247 yum.repos.d]# yum clean all && yum repolist Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases : rhui-REGION-rhel-server-rh-common Cleaning up everything Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos Loaded plugins: amazon-id, rhui-lb, search-disabled-repos rhel-7-ceph | 3.8 kB 00:00:00 rhui-REGION-client-config-server-7 | 2.9 kB 00:00:00 rhui-REGION-rhel-server-releases | 3.5 kB 00:00:00 rhui-REGION-rhel-server-rh-common | 3.8 kB 00:00:00 (1/9): rhui-REGION-client-config-server-7/x86_64/primary_db | 2.5 kB 00:00:00 (2/9): rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00 (3/9): rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00 (4/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00 (5/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00 (6/9): rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00 (7/9): rhel-7-ceph/group_gz | 464 B 00:00:00 (8/9): rhel-7-ceph/primary_db | 29 kB 00:00:00 (9/9): rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00 repo id repo name status rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 1 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233 repolist: 20,969 [root@ip-172-32-0-247 yum.repos.d]# ===================================================================== ================== libcurl >= 7.29.0-47 (BROKEN) ==================== ===================================================================== [root@ip-172-32-0-247 ec2-user]# rpm -qa | grep libcurl libcurl-7.29.0-47.el7.x86_64 [root@ip-172-32-0-247 ec2-user]# yum clean all && yum repolist Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Cleaning repos: rhel-7-ceph rhui-REGION-client-config-server-7 rhui-REGION-rhel-server-releases : rhui-REGION-rhel-server-rh-common Cleaning up everything Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos Loaded plugins: amazon-id, rhui-lb, search-disabled-repos rhel-7-ceph | 3.8 kB 00:00:00 https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. rhel-7-ceph/primary_db | 29 kB 00:00:00 https://rhui2-cds02.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. https://rhui2-cds01.us-west-2.aws.ce.redhat.com/pulp/repos//rhui-client-config/rhel/server/7/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 401 - Unauthorized Trying other mirror. rhui-REGION-rhel-server-releases/7Server/x86_64 | 3.5 kB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/group | 855 kB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/updateinfo | 2.9 MB 00:00:00 rhui-REGION-rhel-server-releases/7Server/x86_64/primary_db | 55 MB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64 | 3.8 kB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/group | 104 B 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/updateinfo | 33 kB 00:00:00 rhui-REGION-rhel-server-rh-common/7Server/x86_64/primary_db | 121 kB 00:00:00 repo id repo name status rhel-7-ceph rhel-7-ceph-2 Latest Enterprise RPMs 31 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configur 0 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 20,704 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs 233 repolist: 20,968 [root@ip-172-32-0-247 ec2-user]#