Bug 1035983

Summary:	Errors while fetching CA CERT on RHS AWS instances
Product:	Red Hat Update Infrastructure for Cloud Providers	Reporter:	Anush Shetty <ashetty>
Component:	RHUA	Assignee:	dgao
Status:	CLOSED CURRENTRELEASE	QA Contact:	mkovacik
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	2.1.2	CC:	tsanders
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-10 15:11:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Anush Shetty 2013-11-29 06:15:02 UTC

Description of problem: When I try doing "yum update", I see errors while fetching CA CERT even though the update succeeds from another mirror. This is seen in all RHS 2.1 AWS instances - http://amis.app.eng.bos.redhat.com/images?products=RHS-2.1.0&status=Any&regions=Any&arches=Any

Version-Release number of selected component (if applicable):
RHEL-6.4-RHS-2.1-x86_64-6-Access2 (ami-fec289ac)

How reproducible: Consistent


Steps to Reproduce:
1. yum update -y
2.
3.

Actual results:

Error fetching CA CERT, but update succeeds


Additional info:
Cleanup : tzdata-2013c-2.el6.noarch 319/319
https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Open URL] [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds02.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Open URL] [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Open URL] [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds02.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Open URL] [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
Verifying : e2fsprogs-libs-1.41.12-14.el6_4.4.x86_64

Show quoted text
Fri Nov 29 05:51:46 UTC 2013
[root@ip-10-138-139-155 ~]#
[root@ip-10-138-139-155 ~]# wget https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz [Open URL]
--2013-11-29 05:51:49-- https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz [Open URL]
Resolving rhui2-cds01.us-east-1.aws.ce.redhat.com... 174.129.20.92
Connecting to rhui2-cds01.us-east-1.aws.ce.redhat.com|174.129.20.92|:443... connected.
ERROR: cannot verify rhui2-cds01.us-east-1.aws.ce.redhat.com’s certificate, issued by “/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support”:
Unable to locally verify the issuer’s authority.
To connect to rhui2-cds01.us-east-1.aws.ce.redhat.com insecurely, use ‘--no-check-certificate’.

Comment 1 John Matthews 2013-11-30 16:19:55 UTC

Confirmed I am seeing same issue with RHS 2.1 AMI in useast1 and apsoutheast-1


Below attempt from useast-1

# sudo yum update -y
Loaded plugins: aliases, amazon-id, changelog, downloadonly, fastestmirror, filter-data, keys, list-data, merge-conf, priorities, product-id,
              : protectbase, rhui-lb, security, subscription-manager, tmprepo, tsflags, upgrade-helper, verify, versionlock
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Determining fastest mirrors
 * rhui-REGION-client-config-server-6-rhs: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-for-rhui-server-debug: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-for-rhui-server-rpms: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-rhel-6.4-for-rhui-server-debug: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-rhel-6.4-for-rhui-server-rpms: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-rhel-6.4-sfs-for-rhui-debug: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-rhel-6.4-sfs-for-rhui-rpms: rhui2-cds01.us-east-1.aws.ce.redhat.com
 * rhui-REGION-rhs-2.1-rhel-6.4-sfs-for-rhui-srpms: rhui2-cds01.us-east-1.aws.ce.redhat.com
Skipping filters plugin, no data

...
...

  Updating   : iputils-20071127-17.el6_4.2.x86_64                                                                                         248/319 
  Updating   : rh-amazon-rhui-client-rhs21-2.2.96-1.el6_5.noarch                                                                          249/319 
warning: /etc/yum.repos.d/redhat-rhui-client-config-rhs-2.1.repo saved as /etc/yum.repos.d/redhat-rhui-client-config-rhs-2.1.repo.rpmsave
warning: /etc/yum.repos.d/redhat-rhui-rhs-2.1.repo saved as /etc/yum.repos.d/redhat-rhui-rhs-2.1.repo.rpmsave
[INFO:choose_repo] choose_repo:36 2013-11-29 16:11:01,094: Zone [us-east-1d]
[INFO:choose_repo] choose_repo:57 2013-11-29 16:11:01,094: Enabling binary repos in redhat-rhui-rhs-2.1.repo
[INFO:choose_repo] choose_repo:76 2013-11-29 16:11:01,095: Enabling load balancer plugin
[INFO:choose_repo] choose_repo:78 2013-11-29 16:11:01,095: Executing [sed -i 's/enabled=0/enabled=1/' /etc/yum/pluginconf.d/rhui-lb.conf]
[INFO:choose_repo] choose_repo:82 2013-11-29 16:11:01,103: Setting region in load balancer config
[INFO:choose_repo] choose_repo:84 2013-11-29 16:11:01,104: Executing [sed -i 's/REGION/us-east-1/' /etc/yum.repos.d/rhui-load-balancers.conf]
[INFO:choose_repo] choose_repo:88 2013-11-29 16:11:01,111: Enabling client config repo
[INFO:choose_repo] choose_repo:91 2013-11-29 16:11:01,112: Executing [sed -i 's/enabled=0/enabled=1/' /etc/yum.repos.d/redhat-rhui-client-config-rhs-2.1.repo]
  Cleanup    : redhat-storage-server-2.1.0.3-1.el6rhs.noarch                                                                              250/319 
  Cleanup    : samba-client-3.6.9-160.3.el6rhs.x86_64             

...
...

  Cleanup    : 12:dhcp-common-4.1.1-34.P1.el6.x86_64                                                                                      316/319 
  Cleanup    : glibc-common-2.12-1.107.el6_4.4.x86_64                                                                                     317/319 
  Cleanup    : glibc-2.12-1.107.el6_4.4.x86_64                                                                                            318/319 
  Cleanup    : tzdata-2013c-2.el6.noarch                                                                                                  319/319 
https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds02.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds01.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds02.us-east-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
  Verifying  : e2fsprogs-libs-1.41.12-14.el6_4.4.x86_64                                                                                     1/319 
  Verifying  : libtar-1.2.11-17.el6_4.1.x86_64                                                                                              2/319 
  Verifying  : yum-rhn-plugin-0.9.1-49.el6.noarch                                                                                           3/319 
  Verifying  : libref_array-0.1.1-9.el6.x86_64         



/etc/yum.repos.d/redhat-rhui-rhs-2.1.repo

[rhui-REGION-rhs-2.1-rhel-6.4-sfs-for-rhui-rpms]
name=Red Hat Enterprise Linux 6.4 Scalable File System for RHS 2.1 (RPMs) from RHUI
mirrorlist=https://rhui2-cds01.REGION.aws.ce.redhat.com/pulp/mirror/content/dist/rhs/rhui/server/2.1/$basearch/rhel/6.4/scalablefilesystem/os
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify=1
sslcacert=/etc/pki/rhui/cdn.redhat.com-chain.crt
sslclientcert=/etc/pki/rhui/product/content-rhel6-rhs-2.1.crt
sslclientkey=/etc/pki/rhui/content-rhel6-rhs-2.1.key




Attempt from apsoutheast-1:
https://rhui2-cds02.ap-southeast-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds01.ap-southeast-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds02.ap-southeast-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.
https://rhui2-cds01.ap-southeast-1.aws.ce.redhat.com/pulp/repos//content/dist/rhs/rhui/server/2.1/x86_64/rhel/6.4/scalablefilesystem/os/repodata/productid.gz: [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"
Trying other mirror.

Comment 2 dgao 2013-12-09 17:15:54 UTC

The issue is caused by a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1011082. It changes the path of the ssl ca cert location from /etc/pki/entitlement to /etc/pki/rhui. As a result, the location of the CA cert changes in the middle of the update, and leads to pycurl calls failing. 

Short-term workaround is to update rh-amazon-rhui-client-rhs21 package and ignore the pycurl errors at the end of the transactions. Subsequent transactions should proceed without this error.

Long-term solution is to respin the AMI with the latest rh-amazon-rhui-client-rhs21. Any version after 2.2.94 would include the cert path change.