Bug 725515

Summary: Cannot sync from RHUA to CDS
Product: Red Hat Update Infrastructure for Cloud Providers Reporter: Jay Dobies <jason.dobies>
Component: CDSAssignee: Jay Dobies <jason.dobies>
Status: CLOSED NOTABUG QA Contact: wes hayutin <whayutin>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.0CC: kbidarka, sghai, xdmoon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1128559 (view as bug list) Environment:
Last Closed: 2011-07-25 18:58:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1128559    

Description Jay Dobies 2011-07-25 18:22:53 UTC
It looks to be similar to the nss/curl issues we saw a few weeks ago. This was done using RHEL 6.1 images in Amazon:

-----------
AMI ID:     ami-5ef90237
AMI Label:  309956199498/RHEL-6.1-Starter-EBS-x86_64-4.1-Access

[root@cds-1 rhui-1.2-5Server-i386]# cat /etc/issue
Red Hat Enterprise Linux Server release 6.1 (Santiago)

RPM Versions:
nss-3.12.9-9.el6.x86_64
curl-7.19.7-26.el6_1.1.x86_64
libcurl-7.19.7-26.el6_1.1.x86_64
-----------

I used a base RHUI install with the following versions:
Pulp:       0.208
RHUI Tools: 2.0.38
Grinder:    0.105

RHUI Manager generated an "identity" certificate the CDS uses to connect to the RHUA. This certificate is munged by RHUI Manager to remove the PKCS8 issues we saw previously. Snippet from the cert:

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA2DYRVNYjgHvMX0/+usy64usHQpIug1qazPPFWPFSuwacgU4m


I am able to use wget to download the repomd.xml file from the Pulp server to the CDS, which means a number of things aren't an issue: the content is successfully in place, SSL is set up correctly, authentication isn't a problem, the CDS can resolve the server at "rhua", firewalls aren't in the way, etc.

The issue is that grinder and curl cannot sync from the Pulp server (RHUA) to the CDS. It's still likely NSS/curl related, but I'm not sure of why this is once again an issue.

I attached the output below of running with grinder, wget, and curl.


Using Grinder:


[root@cds-1 grinder]# grinder yum --label jdob -U https://rhua//pulp/repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/os --cacert /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.ca --cert /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.cert 
grinder.RepoFetch: INFO     fetchYumRepo() basepath = ./
grinder.RepoFetch: INFO     Fetching repo metadata...
grinder.RepoFetch: ERROR    Caught exception when trying to fetch content from [https://rhua//pulp/repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/os]: Cannot retrieve repository metadata (repomd.xml) for repository: jdob. Please verify its path and try again
Traceback (most recent call last):
  File "/usr/bin/grinder", line 23, in <module>
    GrinderCLI.CLI().main()
  File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 415, in main
    cmd.main()
  File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 59, in main
    self._do_command()
  File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 263, in _do_command
    self.yfetch.fetchYumRepo()
  File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 460, in fetchYumRepo
    self.yumFetch.getRepoData()
  File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 139, in getRepoData
    for ftype in self.getRepoXmlFileTypes():
  File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 124, in getRepoXmlFileTypes
    return self.repo.repoXML.fileTypes()
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1454, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1450, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: jdob. Please verify its path and try again



Using wget:

[root@cds-1 grinder]# wget --ca-certificate /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.ca --certificate /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.cert https://rhua//pulp/repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/os/repodata/repomd.xml
--2011-07-25 13:54:21--  https://rhua//pulp/repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/os/repodata/repomd.xml
Resolving rhua... 10.118.22.17
Connecting to rhua|10.118.22.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1610 (1.6K) [text/xml]
Saving to: “repomd.xml.1”

100%[================================================>] 1,610       --.-K/s   in 0s      

2011-07-25 13:54:21 (40.1 MB/s) - “repomd.xml.1” saved [1610/1610]

[root@cds-1 grinder]# 



Using curl:


[root@cds-1 grinder]# curl -v https://rhua//pulp/repos/content/dist/rhel/rhui/server/5Server/i386/rhui/1.2/os/repodata/repomd.xml --cacert /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.ca --cert /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.cert 
* About to connect() to rhua port 443 (#0)
*   Trying 10.118.22.17... connected
* Connected to rhua (10.118.22.17) port 443 (#0)
* Initializing NSS with certpath: /etc/pki/nssdb
*   CAfile: /etc/pki/content/rhui-1.2-5Server-i386/consumer-rhui-1.2-5Server-i386.ca
  CApath: none
* NSS error -8054
* Closing connection #0
* SSL connect error
curl: (35) SSL connect error
[root@cds-1 grinder]#

Comment 1 Jay Dobies 2011-07-25 18:58:02 UTC
False alarm (mostly).

The "NSS error -8054" corresponds to the serial number of the certificate being blacklisted for some reason. In this case, that number was 2. I'm not sure why it was blacklisted since it was a fresh install, but I'm also not too worried unless we start to see this regularly.

The workaround was to:
- Delete the identity certificates on RHUI Manager (/etc/pki/rhui/identity.*).
- Restart RHUI Manager, which will generate new identity certificates.
- Use pulp-admin to add those certs to repos. This isn't something that's supported by RHUI Manager but may be added in the future.