Description of problem: Unable synchronize CDSes having created, deleted and created a custom repository. Unable to confirm in another rhui deployment Version-Release number of selected component (if applicable): How reproducible: N/A Actual results: CDS synchronization fails Expected results: Working synchronization Additional info: ### Error details [root@ip-10-53-11-17 ~]# tail -50 /var/log/pulp-cds/gofer.log RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: . Please verify its path and try again 2012-04-19 13:11:27,740 [INFO][worker-0] _sync_repo() @ cdslib.py:318 - Synchronizing repo at [https://ip-10-226-7-207.eu-west-1.compute.internal//pulp/repos/gkrellm/i686] 2012-04-19 13:11:27,740 [INFO][worker-0] _sync_repo() @ cdslib.py:324 - Synchronizing repo [gkrellm-i686] from [https://ip-10-226-7-207.eu-west-1.compute.internal//pulp/repos/gkrellm/i686] to [/var/lib/pulp-cds/repos/gkrellm/i686] 2012-04-19 13:11:27,741 [DEBUG][worker-0] _sync_repo() @ cdslib.py:331 - Configuring repository for authentication 2012-04-19 13:11:29,128 [ERROR][worker-0] sync() @ cdslib.py:181 - Error performing repo sync Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/cds/cdslib.py", line 178, in sync self._sync_repo(base_url, repo) File "/usr/lib/python2.6/site-packages/pulp/cds/cdslib.py", line 359, in _sync_repo fetch.fetchYumRepo(repo_path, verify_options=verify_options) File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 144, in fetchYumRepo self.setupYumInfo() File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 108, in setupYumInfo info.setUp() File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 343, in setUp skip=self.skip) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 82, in __call__ return self.object(self, *args, **kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 267, in __call__ return self.__call(method, args, kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 243, in __call return self.__rmi(method.name, args, kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 136, in __rmi raise Exception(ex) Exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process retval = method(*args, **kwargs) File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 96, in getDownloadItems self.__getRepoData() File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 169, in __getRepoData for ftype in self.__getRepoXmlFileTypes(): File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 156, in __getRepoXmlFileTypes return self.repo.repoXML.fileTypes() File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1462, in <lambda> repoXML = property(fget=lambda self: self._getRepoXML(), File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1458, in _getRepoXML raise Errors.RepoError, msg RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: . Please verify its path and try again 2012-04-19 13:11:29,129 [INFO][worker-0] update_cluster_membership() @ cdslib.py:236 - Received cluster membership update; Cluster [rhui-cds-cluster], Members [ip-10-49-107-20.eu-west-1.compute.internal, ip-10-53-11-17.eu-west-1.compute.internal] 2012-04-19 13:11:29,130 [INFO][worker-0] update_cluster_membership() @ cdslib.py:247 - No changes needed to be made to cluster memberships
Created attachment 578726 [details] Screen log
Can not reproduce this bug in a 2.0.3. Bumping to 2.1 for further investigation.
What I was doing, when it happened: - there was the gkrellm-x86_64 custom repo "unprotected"; path: gkrellm/x86_64 - then I've added gkrellm-i386, protected by entitlement; path gkrellm/i386 - uploaded content into the gkrellm-i386 - checked content presence in /var/lib/pulp/repos/gkrellm - checked that in rhui-manager <main> -> n -> c lists that repo - removed both gkrellm-x86_64 and gkrellm-i386 repos - created gkrellm-x86_64 and gkrellm-i686 - tried to sync CDSes - error Since then can't sync CDSes anymore, but couldn't reproduce it in another deployment either.
Today I've noticed my identity certificate expired; this is probably the reason for the failure as generating new identity certificate solved the issue.
Created attachment 578906 [details] cds gofer log
Created attachment 578907 [details] rhua pulp log directory
### Screen Log of the identity certificate update solves the issue [root@ip-10-226-7-207 pulp]# rhui-manager status ip-10-49-107-20.eu-west-1.compute.internal .................. [ UP ] ip-10-49-107-20.eu-west-1.compute.internal .................. [ ERROR ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ UP ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ ERROR ] qRed Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [ ERROR ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64) [ ERROR ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386) [ ERROR ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [ ERROR ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ ERROR ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [ ERROR ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [ ERROR ] Identity certificate expiration date = 2012-04-18T10:47:05Z .......... [ ERROR ] Identity certificate at /etc/pki/rhui/identity.crt is expired! Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z .... [ OK ] 19 [root@ip-10-226-7-207 pulp]# [root@ip-10-226-7-207 ~]# rhui-manager status ip-10-49-107-20.eu-west-1.compute.internal .................. [ UP ] ip-10-49-107-20.eu-west-1.compute.internal .................. [ ERROR ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ UP ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ ERROR ] Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [ SUCCESS ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64) [ SUCCESS ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386) [ SUCCESS ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [ SUCCESS ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [ SUCCESS ] Identity certificate expiration date = 2012-04-18T10:47:05Z .......... [ ERROR ] Identity certificate at /etc/pki/rhui/identity.crt is expired! Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z .... [ OK ] 18 [root@ip-10-226-7-207 ~]# rhui-manager ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Home =- r manage repositories c manage content delivery servers (CDS) s synchronization status and scheduling e create entitlement certificates and client configuration RPMs n manage Red Hat entitlement certificates u manage users i manage identity certificate logout removes stored authentication credentials and exits < move to the previous screen ^, home move to the home screen /, clear clears the screen ?, help display help q, quit, exit exit Connected: ip-10-226-7-207.eu-west-1.compute.internal ------------------------------------------------------------------------------ rhui (home) => i ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Identity Certificate Management =- g generate a new identity certificate Connected: ip-10-226-7-207.eu-west-1.compute.internal ------------------------------------------------------------------------------ rhui (identity) => g Generating a new RHUI identity certificate will replace the one currently stored at /etc/pki/rhui/identity.crt. Proceed? [y/n]: y Enter the number of days the RHUI identity certificate will be valid. If the identity certificate ever expires, it will need to be regenerated using rhui-manager [Default: 3650]: ................................................................................................................................................+++ ............................+++ Successfully regenerated RHUI Identity certificate ------------------------------------------------------------------------------ rhui (identity) => < ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Home =- r manage repositories c manage content delivery servers (CDS) s synchronization status and scheduling e create entitlement certificates and client configuration RPMs n manage Red Hat entitlement certificates u manage users i manage identity certificate Connected: ip-10-226-7-207.eu-west-1.compute.internal ------------------------------------------------------------------------------ rhui (home) => s ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= Synchronization Status =- dr display repo sync summary vr view the details of the last repository sync sr sync an individual repository immediately dc display CDS sync summary vc view the details of the last CDS sync sc sync an individual CDS immediately Connected: ip-10-226-7-207.eu-west-1.compute.internal ------------------------------------------------------------------------------ rhui (sync) => sc Select one or more CDS instances to schedule to be synchronized before its scheduled time. The sync will happen as soon as possible depending on other tasks that may be executing in the RHUI. Only CDS instances that are not currently synchronizing are displayed. Last Result Next Sync CDS ------------------------------------------ - 1 : Error 04-20-2012 06:23 ip-10-49-107-20.eu-west-1.compute.internal - 2 : Error 04-20-2012 06:23 ip-10-53-11-17.eu-west-1.compute.internal Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: 1-2 Select one or more CDS instances to schedule to be synchronized before its scheduled time. The sync will happen as soon as possible depending on other tasks that may be executing in the RHUI. Only CDS instances that are not currently synchronizing are displayed. Last Result Next Sync CDS ------------------------------------------ x 1 : Error 04-20-2012 06:23 ip-10-49-107-20.eu-west-1.compute.internal x 2 : Error 04-20-2012 06:23 ip-10-53-11-17.eu-west-1.compute.internal Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: c The following CDS instances will be scheduled for synchronization: ip-10-49-107-20.eu-west-1.compute.internal ip-10-53-11-17.eu-west-1.compute.internal Proceed? (y/n) y Scheduling sync for ip-10-49-107-20.eu-west-1.compute.internal... ... successfully scheduled for the next available timeslot. Scheduling sync for ip-10-53-11-17.eu-west-1.compute.internal... ... successfully scheduled for the next available timeslot. ------------------------------------------------------------------------------ rhui (sync) => dc ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= CDS Synchronization Status =- Last Refreshed: 03:13:08 (updated every 5 seconds, ctrl+c to exit) ip-10-49-107-20.eu-west-1.compute.internal .................. [ UP ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ UP ] Next Sync Last Sync Last Result ------------------------------------------------------------------------------ ip-10-49-107-20.eu-west-1.compute.internal In Progress 04-20-2012 00:23 Error ip-10-53-11-17.eu-west-1.compute.internal In Progress 04-20-2012 00:23 Error Connected: ip-10-226-7-207.eu-west-1.compute.internal ------------------------------------------------------------------------------ [root@ip-10-226-7-207 log]# rhui-manager status ip-10-49-107-20.eu-west-1.compute.internal .................. [ UP ] ip-10-49-107-20.eu-west-1.compute.internal .................. [ SUCCESS ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ UP ] ip-10-53-11-17.eu-west-1.compute.internal ................... [ SUCCESS ] Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [ SUCCESS ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64) [ SUCCESS ] Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386) [ SUCCESS ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [ SUCCESS ] Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [ SUCCESS ] Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [ SUCCESS ] Identity certificate expiration date = 2022-04-18T07:12:29Z .......... [ OK ] Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z .... [ OK ] 0 [root@ip-10-226-7-207 log]#
Closing based on previous comments...