Bug 996198 - CDS sync should not attempt to sync older packages which have been removed from RHUA by way of 'num_old_packages_keep'
CDS sync should not attempt to sync older packages which have been removed fr...
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: CDS (Show other bugs)
Unspecified Unspecified
high Severity high
: ---
: 2.1.3
Assigned To: John Matthews
Ina Panova
Depends On:
Blocks: 1017409
  Show dependency treegraph
Reported: 2013-08-12 11:50 EDT by John Matthews
Modified: 2013-12-17 15:10 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
This update changes the behavior of a Content Delivery Server synchronization to allow a limit on the synchronization of older packages. The RHUA now sends two synchronization options to the Content Delivery Server when making a synchronization call. This allows the Content Delivery Server to match the behavior of the RHUA. Two configuration options have been added to the Content Delivery Server configuration to allow the Content Delivery Server to override options from the RHUA and further restrict how many older packages will be synchronized. This prevents errors on a RHUA when the RHUA is limiting older packages but the Content Delivery Server was unaware. The RHUA and Content Delivery Server are now more closely related in behavior. Additionally a Content Delivery Server now has the option to further reduce file storage requirements by only synchronizing the latest version of a package.
Story Points: ---
Clone Of:
: 1017409 (view as bug list)
Last Closed: 2013-12-17 15:10:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description John Matthews 2013-08-12 11:50:55 EDT
Description of problem:

A RHUA can be configured to not sync a configurable number of older packages, yet the CDS does not honor the same configuration.  This results in a large number of 404 messages being logged on RHUA.

The RHUA syncs repo metadata from CDN and preserves this metadata with preserve_metadata=True.  The CDS syncs the metadata from the RHUA, then uses the package metadata to determine what should be synced from RHUA.  In case of the RHUA not syncing old packages the CDS will still attempt to download these missing packages, resulting in excessive 404 messages being logged by Apache.

Possible fix is to update CDS to honor the configuration options for number of old packages.

We could consider having the RHUA configuration parameters passed to the CDS as part of the sync.  Perhaps allowing a CDS configuration file as an additional override, so certain CDSes could be configured for storing even less older packages.
Comment 1 John Matthews 2013-08-12 11:55:40 EDT
See bug 989506 for more information.
Comment 3 John Matthews 2013-10-09 11:25:41 EDT
We have modified the RHUA to send the below sync options to the CDS.

Additionally we have modified the CDS so it may override any options from the RHUA by looking for these options in "/etc/pulp/cds/cds.conf"

Fix is in pulp 0.0.263-37

Below are the associated commits.


Please note, we ran into an unexpected 'feature' with INIConfig on RHEL-6.  Namely, we had to rename the option 'remove_old_versions' to 'cds_remove_old_versions', this is only for /etc/pulp/cds/cds.conf.

Reason for rename is:
 - CDS runs as a Gofer plugin.  
 - Gofer uses INIConfig to parse the config file.
 - INIConfig honors DOS batch file syntax of 'rem' being a comment
   Therefore any lines beginning with 'rem' such as 'remove_old_versions. is interpreted to be a comment.
 - Our workaround is for the CDS to rename 'remove_old_versions' to 'cds_remove_old_versions'


Options explained:
 remove_old_versions:  is a boolean, true means we will limit older package synced, false means we will not.

 num_old_pkgs_keep: is an integer, if remove_old_versions=true, then we will look at this value and keep this many old versions of a package.  
  0 means we keep none (i.e. only latest package is downloaded, 0 old packages are kept).   
  1 means we download latest package AND we download 1 old package

Below are a few test scenario suggestions

1) RHUA, remove_old_versions=false
   CDS, remove_old_versions=false

2) RHUA, remove_old_versions=false
   CDS, remove_old_versions=true

3) RHUA, remove_old_versions=true, num_old_pkgs_keep=4
   CDS, remove_old_versions=true, num_old_pkg_keep=1

Note, the below scenario will result in many 404's on RHUA http log, as the CDS will attempt to sync packages that dont exist on the RHUA.
 RHUA, remove_old_versions=true
 CDS, remove_old_versions=false
Comment 5 Ina Panova 2013-11-04 06:27:15 EST
tested in pulp-0.0.263-38.git.1.7f842df.el6.noarch

Move to Verified.
Comment 7 errata-xmlrpc 2013-12-17 15:10:10 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.