Bug 814377

Summary: Custom repository causes a CDS synchronization failure
Product: Red Hat Update Infrastructure for Cloud Providers Reporter: mkovacik
Component: RHUAAssignee: James Slagle <jslagle>
Status: CLOSED NOTABUG QA Contact: wes hayutin <whayutin>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.0.3CC: kbidarka, sghai, tsanders
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-04-20 07:46:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Screen log
none
cds gofer log
none
rhua pulp log directory none

Description mkovacik 2012-04-19 17:55:33 UTC 
Description of problem:
Unable synchronize CDSes having created, deleted and created a custom repository. Unable to confirm in another rhui deployment

Version-Release number of selected component (if applicable):


How reproducible:
N/A
  
Actual results:
CDS synchronization fails

Expected results:
Working synchronization

Additional info:

### Error details
[root@ip-10-53-11-17 ~]# tail -50 /var/log/pulp-cds/gofer.log 

RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: . Please verify its path and try again

2012-04-19 13:11:27,740 [INFO][worker-0] _sync_repo() @ cdslib.py:318 - Synchronizing repo at [https://ip-10-226-7-207.eu-west-1.compute.internal//pulp/repos/gkrellm/i686]
2012-04-19 13:11:27,740 [INFO][worker-0] _sync_repo() @ cdslib.py:324 - Synchronizing repo [gkrellm-i686] from [https://ip-10-226-7-207.eu-west-1.compute.internal//pulp/repos/gkrellm/i686] to [/var/lib/pulp-cds/repos/gkrellm/i686]
2012-04-19 13:11:27,741 [DEBUG][worker-0] _sync_repo() @ cdslib.py:331 - Configuring repository for authentication
2012-04-19 13:11:29,128 [ERROR][worker-0] sync() @ cdslib.py:181 - Error performing repo sync
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/cds/cdslib.py", line 178, in sync
    self._sync_repo(base_url, repo)
  File "/usr/lib/python2.6/site-packages/pulp/cds/cdslib.py", line 359, in _sync_repo
    fetch.fetchYumRepo(repo_path, verify_options=verify_options)
  File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 144, in fetchYumRepo
    self.setupYumInfo()
  File "/usr/lib/python2.6/site-packages/grinder/RepoFetch.py", line 108, in setupYumInfo
    info.setUp()
  File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 343, in setUp
    skip=self.skip)
  File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 82, in __call__
    return self.object(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 267, in __call__
    return self.__call(method, args, kwargs)
  File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 243, in __call
    return self.__rmi(method.name, args, kwargs)
  File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 136, in __rmi
    raise Exception(ex)
Exception: Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 429, in process
    retval = method(*args, **kwargs)

  File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 96, in getDownloadItems
    self.__getRepoData()

  File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 169, in __getRepoData
    for ftype in self.__getRepoXmlFileTypes():

  File "/usr/lib/python2.6/site-packages/grinder/YumInfo.py", line 156, in __getRepoXmlFileTypes
    return self.repo.repoXML.fileTypes()

  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1462, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),

  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1458, in _getRepoXML
    raise Errors.RepoError, msg

RepoError: Cannot retrieve repository metadata (repomd.xml) for repository: . Please verify its path and try again

2012-04-19 13:11:29,129 [INFO][worker-0] update_cluster_membership() @ cdslib.py:236 - Received cluster membership update; Cluster [rhui-cds-cluster], Members [ip-10-49-107-20.eu-west-1.compute.internal, ip-10-53-11-17.eu-west-1.compute.internal]
2012-04-19 13:11:29,130 [INFO][worker-0] update_cluster_membership() @ cdslib.py:247 - No changes needed to be made to cluster memberships
Comment 1 mkovacik 2012-04-19 17:56:43 UTC 
Created attachment 578726 [details]
Screen log
Comment 2 James Slagle 2012-04-19 19:37:38 UTC 
Can not reproduce this bug in a 2.0.3.  Bumping to 2.1 for further investigation.
Comment 3 mkovacik 2012-04-19 20:34:34 UTC 
What I was doing, when it happened:
- there was the gkrellm-x86_64 custom repo "unprotected"; path: gkrellm/x86_64
- then I've added gkrellm-i386, protected by entitlement; path gkrellm/i386
- uploaded content into the gkrellm-i386
- checked content presence in /var/lib/pulp/repos/gkrellm
- checked that in rhui-manager <main> -> n -> c lists that repo
- removed both gkrellm-x86_64 and gkrellm-i386 repos
- created gkrellm-x86_64 and gkrellm-i686
- tried to sync CDSes
- error
Since then can't sync CDSes anymore, but couldn't reproduce it in another deployment either.
Comment 4 mkovacik 2012-04-20 07:36:26 UTC 
Today I've noticed my identity certificate expired; this is probably the reason for the failure as generating new identity certificate solved the issue.
Comment 5 mkovacik 2012-04-20 07:37:39 UTC 
Created attachment 578906 [details]
cds gofer log
Comment 6 mkovacik 2012-04-20 07:39:06 UTC 
Created attachment 578907 [details]
rhua pulp log directory
Comment 7 mkovacik 2012-04-20 07:44:18 UTC 
### Screen Log of the identity certificate update solves the issue
[root@ip-10-226-7-207 pulp]# rhui-manager status
ip-10-49-107-20.eu-west-1.compute.internal .................. [  UP  ]

ip-10-49-107-20.eu-west-1.compute.internal .................. [  ERROR  ]
ip-10-53-11-17.eu-west-1.compute.internal ................... [  UP  ]

ip-10-53-11-17.eu-west-1.compute.internal ................... [  ERROR  ]

qRed Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [  ERROR  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64)  [  ERROR  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386)  [  ERROR  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [  ERROR  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [  ERROR  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [  ERROR  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [  ERROR  ]

Identity certificate expiration date = 2012-04-18T10:47:05Z  .......... [  ERROR  ]
Identity certificate at /etc/pki/rhui/identity.crt is expired!

Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z  .... [  OK  ]
19
[root@ip-10-226-7-207 pulp]# 
[root@ip-10-226-7-207 ~]# rhui-manager status
ip-10-49-107-20.eu-west-1.compute.internal .................. [  UP  ]

ip-10-49-107-20.eu-west-1.compute.internal .................. [  ERROR  ]
ip-10-53-11-17.eu-west-1.compute.internal ................... [  UP  ]

ip-10-53-11-17.eu-west-1.compute.internal ................... [  ERROR  ]

Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64)  [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386)  [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [  SUCCESS  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [  SUCCESS  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [  SUCCESS  ]

Identity certificate expiration date = 2012-04-18T10:47:05Z  .......... [  ERROR  ]
Identity certificate at /etc/pki/rhui/identity.crt is expired!

Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z  .... [  OK  ]
18
[root@ip-10-226-7-207 ~]# rhui-manager

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Home =-

   r   manage repositories
   c   manage content delivery servers (CDS)
   s   synchronization status and scheduling
   e   create entitlement certificates and client configuration RPMs
   n   manage Red Hat entitlement certificates
   u   manage users
   i   manage identity certificate

   logout
       removes stored authentication credentials and exits

   <   move to the previous screen
   ^, home
       move to the home screen
   /, clear
       clears the screen
   ?, help
       display help
   q, quit, exit
       exit

                         Connected: ip-10-226-7-207.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (home) => i

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Identity Certificate Management =-

   g   generate a new identity certificate

                         Connected: ip-10-226-7-207.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (identity) => g

Generating a new RHUI identity certificate will replace
the one currently stored at /etc/pki/rhui/identity.crt.  Proceed? [y/n]: y

Enter the number of days the RHUI identity certificate will be valid.
If the identity certificate ever expires, it will need to be
regenerated using rhui-manager [Default: 3650]:
................................................................................................................................................+++
............................+++
Successfully regenerated RHUI Identity certificate

------------------------------------------------------------------------------
rhui (identity) => <

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Home =-

   r   manage repositories
   c   manage content delivery servers (CDS)
   s   synchronization status and scheduling
   e   create entitlement certificates and client configuration RPMs
   n   manage Red Hat entitlement certificates
   u   manage users
   i   manage identity certificate

                         Connected: ip-10-226-7-207.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (home) => s

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= Synchronization Status =-

   dr  display repo sync summary
   vr  view the details of the last repository sync
   sr  sync an individual repository immediately
   dc  display CDS sync summary
   vc  view the details of the last CDS sync
   sc  sync an individual CDS immediately

                         Connected: ip-10-226-7-207.eu-west-1.compute.internal
------------------------------------------------------------------------------
rhui (sync) => sc

Select one or more CDS instances to schedule to be synchronized before its scheduled time.
The sync will happen as soon as possible depending on other tasks that may be executing
in the RHUI. Only CDS instances that are not currently synchronizing are displayed.

         Last Result   Next Sync         CDS
         ------------------------------------------
  -  1 : Error  04-20-2012 06:23  ip-10-49-107-20.eu-west-1.compute.internal
  -  2 : Error  04-20-2012 06:23  ip-10-53-11-17.eu-west-1.compute.internal
Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: 1-2

Select one or more CDS instances to schedule to be synchronized before its scheduled time.
The sync will happen as soon as possible depending on other tasks that may be executing
in the RHUI. Only CDS instances that are not currently synchronizing are displayed.

         Last Result   Next Sync         CDS
         ------------------------------------------
  x  1 : Error  04-20-2012 06:23  ip-10-49-107-20.eu-west-1.compute.internal
  x  2 : Error  04-20-2012 06:23  ip-10-53-11-17.eu-west-1.compute.internal
Enter value (1-2) to toggle selection, 'c' to confirm selections, or '?' for more commands: c


The following CDS instances will be scheduled for synchronization:
  ip-10-49-107-20.eu-west-1.compute.internal
  ip-10-53-11-17.eu-west-1.compute.internal
Proceed? (y/n) y


Scheduling sync for ip-10-49-107-20.eu-west-1.compute.internal...
... successfully scheduled for the next available timeslot.
Scheduling sync for ip-10-53-11-17.eu-west-1.compute.internal...
... successfully scheduled for the next available timeslot.

------------------------------------------------------------------------------
rhui (sync) => dc

------------------------------------------------------------------------------
             -= Red Hat Update Infrastructure Management Tool =-


-= CDS Synchronization Status =-

Last Refreshed: 03:13:08
(updated every 5 seconds, ctrl+c to exit)


ip-10-49-107-20.eu-west-1.compute.internal .................. [  UP  ]
ip-10-53-11-17.eu-west-1.compute.internal ................... [  UP  ]


Next Sync                    Last Sync                    Last Result
------------------------------------------------------------------------------
ip-10-49-107-20.eu-west-1.compute.internal
In Progress                  04-20-2012 00:23             Error

ip-10-53-11-17.eu-west-1.compute.internal
In Progress                  04-20-2012 00:23             Error


                         Connected: ip-10-226-7-207.eu-west-1.compute.internal
------------------------------------------------------------------------------
[root@ip-10-226-7-207 log]# rhui-manager status
ip-10-49-107-20.eu-west-1.compute.internal .................. [  UP  ]

ip-10-49-107-20.eu-west-1.compute.internal .................. [  SUCCESS  ]
ip-10-53-11-17.eu-west-1.compute.internal ................... [  UP  ]

ip-10-53-11-17.eu-west-1.compute.internal ................... [  SUCCESS  ]

Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) ............. [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-x86_64)  [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server - Supplementary (Source RPMs) from RHUI (6Server-i386)  [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-i386) ..... [  SUCCESS  ]

Red Hat Enterprise Linux 6 Server (RPMs) from RHUI (6Server-x86_64) ... [  SUCCESS  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386) ..... [  SUCCESS  ]

Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64) ... [  SUCCESS  ]

Identity certificate expiration date = 2022-04-18T07:12:29Z  .......... [  OK  ]

Entitlement CA certificate expiration date = 2012-07-25T16:36:58Z  .... [  OK  ]
0
[root@ip-10-226-7-207 log]#
Comment 8 mkovacik 2012-04-20 07:46:51 UTC 
Closing based on previous comments...