I swear we've seen this before, but I forget the resolution. QE is testing RHUI with the following builds: Pulp 0.186 Grinder 0.100 I fell back to the grinder CLI to test and it's unable to download a repo that's protected in Pulp. Using wget, I can download the repomd.xml file, so the certificates themselves are working fine. sghai has an environment where this is reproducible, contact him for information on logins. Below is the output showing that the grinder CLI fails but wget succeeds [root@dhcp193-65 rhel-server-6-optional-releases-6Server-x86_64]# grinder yum --cacert ../pulp-server-ca.crt --cert ./consumer-rhel-server-6-optional-releases-6Server-x86_64.cert --key ./consumer-rhel-server-6-optional-releases-6Server-x86_64.key --url https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os --label jdob-test grinder.RepoFetch: INFO fetchYumRepo() basepath = ./ grinder.RepoFetch: INFO Fetching repo metadata... grinder.RepoFetch: ERROR Caught exception when trying to fetch content from [https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os]: Cannot retrieve repository metadata (repomd.xml) for repository: jdob-test. Please verify its path and try again Traceback (most recent call last): File "/usr/bin/grinder", line 23, in <module> GrinderCLI.CLI().main() File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 415, in main cmd.main() File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 59, in main self._do_command() File "/usr/lib/python2.6/site-packages/grinder/GrinderCLI.py", line 263, in _do_command self.yfetch.fetchYumRepo() File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 393, in fetchYumRepo self.yumFetch.getRepoData() File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 138, in getRepoData for ftype in self.getRepoXmlFileTypes(): File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 123, in getRepoXmlFileTypes return self.repo.repoXML.fileTypes() File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1417, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1413, in _getRepoXML raise Errors.RepoError, msg yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: jdob-test. Please verify its path and try again [root@dhcp193-65 rhel-server-6-optional-releases-6Server-x86_64]# wget --certificate consumer-rhel-server-6-optional-releases-6Server-x86_64.cert --private-key consumer-rhel-server-6-optional-releases-6Server-x86_64.key --ca-certificate /etc/pki/content/pulp-server-ca.crt https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os/repodata/repomd.xml --2011-06-03 18:09:10-- https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os/repodata/repomd.xml Resolving dhcp193-79.pnq.redhat.com... 10.65.193.79 Connecting to dhcp193-79.pnq.redhat.com|10.65.193.79|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 3452 (3.4K) [text/plain] Saving to: “repomd.xml” 100%[================================================>] 3,452 --.-K/s in 0s 2011-06-03 18:09:10 (56.2 MB/s) - “repomd.xml” saved [3452/3452]
Grinder uses libcurl, a test with just curl shows the below: # curl --cert consumer-rhel-server-6-optional-releases-6Server-x86_64.cert --key consumer-rhel-server-6-optional-releases-6Server-x86_64.key --cacert /etc/pki/content/pulp-server-ca.crt https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os/repodata/repomd.xml curl: (27) Out of memory # vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 1460 135556 8948 162040 0 0 1 1 17 17 0 0 100 0 0 # cat /proc/meminfo MemTotal: 503392 kB MemFree: 135564 kB Buffers: 8948 kB Cached: 162000 kB SwapCached: 568 kB Active: 133856 kB Inactive: 94340 kB Active(anon): 39616 kB Inactive(anon): 18064 kB Active(file): 94240 kB Inactive(file): 76276 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1015800 kB SwapFree: 1014340 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 56916 kB Mapped: 20076 kB Shmem: 428 kB Slab: 121056 kB SReclaimable: 61492 kB SUnreclaim: 59564 kB KernelStack: 1464 kB PageTables: 9624 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1267496 kB Committed_AS: 438860 kB VmallocTotal: 34359738367 kB VmallocUsed: 3796 kB VmallocChunk: 34359722812 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 8172 kB DirectMap2M: 516096 kB
I don't think it's a memory issue. I used certs from candlepin against a different demo server and curl was able to fetch protected content on the same box, 'dhcp193-65'. Looking into why wget is fine with these certs but curl is not. For background, I read that the 'Out of Memory' error from curl could be thrown if curl is unable to perform a 'new' on a SSLContext, maybe parse error with strings.
Main issue is Curl/NSS need to be upgraded. Two problems popped up. 1) Curl is treating a filename without a '/' in name to be treated as a nickname, so a reference like "--cert blah.cert" wouldn't work, we would need to do "--cert ./blah.cert" 2) Second issue is that the combination of curl/nss we are using does not like a client key in PKCS#8 format, yet it does work OK with PKCS#1 format. (upgrading fixes this) For background, the machine we saw this on was running: redhat-release-server-6Server-6.0.0.37.el6.x86_64 It had these versions installed for curl/nss: curl-7.19.7-16.el6.x86_64 nss-3.12.7-2.el6.x86_64 I registered to RHN and upgraded to: curl-7.19.7-26.el6.x86_64 nss-3.12.9-9.el6.x86_64 curl --cert ./consumer-rhel-server-6-optional-releases-6Server-x86_64.cert --key consumer-rhel-server-6-optional-releases-6Server-x86_64.key --cacert /etc/pki/content/pulp-server-ca.crt https://dhcp193-79.pnq.redhat.com//pulp/repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/optional/os/repodata/repomd.xml <?xml version="1.0" encoding="UTF-8"?> <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"> <revision>1307111331</revision> <data type="other_db"> <location href="repodata/other.sqlite.bz2"/> <checksum type="sha256">60c533f2d4d76c55d19ec89022df2fb9ee4fa817e35286573c9b2d994c70e795</checksum> <timestamp>1307111709.25</timestamp> <size>986039</size> <open-size>4503552</open-size> <open-checksum type="sha256">4d9c2e47fd3b1b81cb94f6c884c21b73452d7aba214993d77c0e18efa9d96468</open-checksum> <database_version>10</database_version> .... .... .... It was successful
Recommendation we specify pulp has a dep on RHEL-6 of => nss 3.12.9-9 => curl-7.19.7-26
Note the Out Of Memory we saw is related to bz623663 - curl returns CURLE_OUT_OF_MEMORY when client certificate is given
Fix: diff --git a/pulp.spec b/pulp.spec index 2a596a4..4dee341 100644 --- a/pulp.spec +++ b/pulp.spec @@ -59,6 +59,8 @@ Requires: python-hashlib Requires: python-uuid Requires: python-ctypes Requires: python-hashlib +Requires: nss >= 3.12.9 +Requires: curl => 7.19.7 %endif We need to get these onto RHUI ISO, in meantime for testing we can register system to RHN and update the packages.
The sync now works by updating the nss and curl rpms. [root@dhcp201-196 os]# ls grinder-0.0.57-1.el5.noarch.rpm httpd-2.2.3-43.el5_5.3.x86_64.rpm libyaml-0.1.2-3.el5.x86_64.rpm m2crypto-0.16-6.1.el5_5.1.x86_64.rpm mod_python-3.3.1-12.el5.x86_64.rpm python-hashlib-20081119-5.el5.x86_64.rpm python-pycurl-7.15.5.1-4.el5.x86_64.rpm PyYAML-3.08-4.el5.x86_64.rpm repodata rh-cds-0.27-1.el5_5.noarch.rpm rh-rhua-0.91-1.el5_5.noarch.rpm rh-rhua-0.95-1.el5_5.noarch.rpm rh-rhui-tools-0.76-1.el5_5.noarch.rpm rpm-build-4.4.2.3-20.el5_5.1.x86_64.rpm [root@dhcp201-196 os]# pwd /var/lib/pulp-cds/content/dist/rhel/rhui/server/5Server/x86_64/rhui/1.2/os [root@dhcp201-196 os]# hostname dhcp201-196.englab.pnq.redhat.com
Found that we need to update nss and curl rpms even on the clients for the ssl to work. Just wondering whether we will have such dependencies also for rhel5 clients or in case of custom repos for example fedora clients. Need to check this out for rhel5 clients/others now. Does, this also mean that all the cloud images by default should be having these nss and curl changes (with their respective versions) requirements to work with RHUI 2.0 May be I am missing something, but currently it looks like every fedora or redhat release will be affected and the nss/curl rpm version numbers for that release must be specified for the client to work.
build: 0.188
It works for fedora, without any update of nss and curl. [root@localhost yum.repos.d]# cat rh-cloud.repo | head -n 10 [custom-10000] name=Custom Repositories - 10000 mirrorlist=file:///etc/yum.repos.d/rh-custom-10000.mirror enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release sslverify=1 sslclientkey=/etc/pki/entitlement/key.pem sslclientcert=/etc/pki/entitlement/product/content.crt sslcacert=/etc/pki/entitlement/ca.crt [root@localhost yum.repos.d]# cat rh-custom-10000.mirror #https://dhcp201-196.englab.pnq.redhat.com/rhuilb/pulp/repos//baseos/$basearch/fedora14/os https://dhcp201-101.englab.pnq.redhat.com/rhuilb/pulp/repos//baseos/$basearch/fedora14/os [root@localhost yum.repos.d]# rpm -ev gdb [root@localhost yum.repos.d]# yum install gdb Loaded plugins: langpacks, presto, refresh-packagekit Adding en_US to language list Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package gdb.x86_64 0:7.2-16.fc14 set to be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================================================= Package Arch Version Repository Size ================================================================================================================================================================================= Installing: gdb x86_64 7.2-16.fc14 custom-10000 2.2 M Transaction Summary ================================================================================================================================================================================= Install 1 Package(s) Total download size: 2.2 M Installed size: 5.2 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 2.2 M gdb-7.2-16.fc14.x86_64.rpm | 2.2 MB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Warning: RPMDB altered outside of yum. Installing : gdb-7.2-16.fc14.x86_64 1/1 Installed: gdb.x86_64 0:7.2-16.fc14 Complete!
works for rhel5 as well.
Verified in pulp build 0.190. The required version of nss/curl should be like: nss >= 3.12.9 curl => 7.19.7 I verified this requirement while installing pulp and pulp-cds. I got following, while Installing pulp Error: Package: grinder-0.0.103-1.el6.noarch (testing-rhel-pulp) Requires: nss >= 3.12.9 Installed: nss-3.12.7-2.el6.x86_64 (@anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0) nss = 3.12.7-2.el6 Error: Package: pulp-0.0.190-1.el6.noarch (testing-rhel-pulp) Requires: nss >= 3.12.9 Installed: nss-3.12.7-2.el6.x86_64 (@anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0) nss = 3.12.7-2.el6 Then I updated the repo to rhel6.1 to update these pkges, and this time Dependencies Resolved ==================================================================================================== Package Arch Version Repository Size ==================================================================================================== Installing: pulp noarch 0.0.190-1.el6 testing-rhel-pulp 583 k Updating: curl x86_64 7.19.7-26.el6 rhel6.1 191 k Installing for dependencies: PyYAML x86_64 3.09-5.el6 epel 158 k apr x86_64 1.3.9-3.el6 rhel-pnq 123 k <output truncated> : : : Updating for dependencies: libcurl x86_64 7.19.7-26.el6 rhel6.1 163 k nspr x86_64 4.8.7-1.el6 rhel6.1 110 k nss x86_64 3.12.9-9.el6 rhel6.1 766 k nss-softokn x86_64 3.12.9-3.el6 rhel6.1 170 k nss-softokn-freebl x86_64 3.12.9-3.el6 rhel6.1 122 k nss-sysinit x86_64 3.12.9-9.el6 rhel6.1 28 k nss-util x86_64 3.12.9-1.el6 rhel6.1 46 k Transaction Summary ==================================================================================================== Install 52 Package(s) Upgrade 8 Package(s) Total download size: 29 M Is this ok [y/N]:
Similar observation ( as in comment13 ) while installing pulp-cds: Installed: pulp-cds.noarch 0:0.0.190-1.el6 Dependency Installed: PyYAML.x86_64 0:3.09-5.el6 apr.x86_64 0:1.3.9-3.el6 apr-util.x86_64 0:1.3.9-3.el6_0.1 apr-util-ldap.x86_64 0:1.3.9-3.el6_0.1 createrepo.noarch 0:0.9.8-4.el6 deltarpm.x86_64 0:3.5-0.5.20090913git.el6 gofer.noarch 0:0.38-1.el6 grinder.noarch 0:0.0.103-1.el6 httpd.x86_64 0:2.2.15-9.el6 httpd-tools.x86_64 0:2.2.15-9.el6 libyaml.x86_64 0:0.1.3-1.el6 mailcap.noarch 0:2.1.31-1.1.el6 mod_python.x86_64 0:3.3.1-14.el6.1 mod_ssl.x86_64 1:2.2.15-9.el6 mod_wsgi.x86_64 0:3.2-1.el6 pulp-common.noarch 0:0.0.190-1.el6 python-deltarpm.x86_64 0:3.5-0.5.20090913git.el6 python-gofer.noarch 0:0.38-1.el6 python-qpid.noarch 0:0.10-1.el6 python-simplejson.x86_64 0:2.0.9-3.1.el6 Updated: curl.x86_64 0:7.19.7-26.el6 Dependency Updated: libcurl.x86_64 0:7.19.7-26.el6 nspr.x86_64 0:4.8.7-1.el6 nss.x86_64 0:3.12.9-9.el6 nss-softokn.x86_64 0:3.12.9-3.el6 nss-softokn-freebl.x86_64 0:3.12.9-3.el6 nss-sysinit.x86_64 0:3.12.9-9.el6 nss-util.x86_64 0:3.12.9-1.el6
Verified in build 0.190. The following rpms added as requirement (comment14). nss >= 3.12.9 curl => 7.19.7 And CDS sync is also working fine. So moving this to verified. ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= CDS Synchronization Status =- Last Refreshed: 14:40:00 (updated every 5 seconds, ctrl+c to exit) cds0021 ..................................................... [ UP ] cdss00115 ................................................... [ UP ] Next Sync Last Sync Last Result ------------------------------------------------------------------------------ cds0021 06-17-2011 14:50 06-17-2011 13:50 finished cdss00115 06-17-2011 14:14 06-17-2011 13:41 running Connected: dhcp193-163.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) =>
Rolling back to ASSIGNED since the downstream bug (698563) was marked as failing QA. Please check that bug and work with QE to further debug the issue.
Changing back to VERIFIED; QE upgraded grinder and marked the downstream bug as VERIFIED.
Closing with Community Release 15 pulp-0.0.223-4.