Bug 996198 - CDS sync should not attempt to sync older packages which have been removed from RHUA by way of 'num_old_packages_keep'
Summary: CDS sync should not attempt to sync older packages which have been removed fr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: CDS
Version: 2.1.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.1.3
Assignee: John Matthews
QA Contact: Ina Panova
URL:
Whiteboard:
Depends On:
Blocks: 1017409
TreeView+ depends on / blocked
 
Reported: 2013-08-12 15:50 UTC by John Matthews
Modified: 2018-12-04 15:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
This update changes the behavior of a Content Delivery Server synchronization to allow a limit on the synchronization of older packages. The RHUA now sends two synchronization options to the Content Delivery Server when making a synchronization call. This allows the Content Delivery Server to match the behavior of the RHUA. Two configuration options have been added to the Content Delivery Server configuration to allow the Content Delivery Server to override options from the RHUA and further restrict how many older packages will be synchronized. This prevents errors on a RHUA when the RHUA is limiting older packages but the Content Delivery Server was unaware. The RHUA and Content Delivery Server are now more closely related in behavior. Additionally a Content Delivery Server now has the option to further reduce file storage requirements by only synchronizing the latest version of a package.
Clone Of:
: 1017409 (view as bug list)
Environment:
Last Closed: 2013-12-17 20:10:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1854 0 normal SHIPPED_LIVE Red Hat Update Infrastructure 2.1.3 bug fix update 2013-12-18 01:07:53 UTC

Description John Matthews 2013-08-12 15:50:55 UTC
Description of problem:

A RHUA can be configured to not sync a configurable number of older packages, yet the CDS does not honor the same configuration.  This results in a large number of 404 messages being logged on RHUA.

The RHUA syncs repo metadata from CDN and preserves this metadata with preserve_metadata=True.  The CDS syncs the metadata from the RHUA, then uses the package metadata to determine what should be synced from RHUA.  In case of the RHUA not syncing old packages the CDS will still attempt to download these missing packages, resulting in excessive 404 messages being logged by Apache.

Possible fix is to update CDS to honor the configuration options for number of old packages.

We could consider having the RHUA configuration parameters passed to the CDS as part of the sync.  Perhaps allowing a CDS configuration file as an additional override, so certain CDSes could be configured for storing even less older packages.

Comment 1 John Matthews 2013-08-12 15:55:40 UTC
See bug 989506 for more information.

Comment 3 John Matthews 2013-10-09 15:25:41 UTC
We have modified the RHUA to send the below sync options to the CDS.
 remove_old_versions
 num_old_pkgs_keep

Additionally we have modified the CDS so it may override any options from the RHUA by looking for these options in "/etc/pulp/cds/cds.conf"

Fix is in pulp 0.0.263-37

Below are the associated commits.

https://github.com/splice/pulp/commit/f85ef0477fd7bd24de810fc3788d9bc95b843079
https://github.com/splice/pulp/commit/33eb99928c42a0ef44850934238b6a3c8b784b7a
https://github.com/splice/pulp/commit/f9d840eb51d7b395af7636901d5f5d204f9e35bd
https://github.com/splice/pulp/commit/438e944f22d35393497023fb07cce23e8e9bd366
https://github.com/splice/pulp/commit/a9ce91818dae5e92c623b12ea571d6257869c819


Please note, we ran into an unexpected 'feature' with INIConfig on RHEL-6.  Namely, we had to rename the option 'remove_old_versions' to 'cds_remove_old_versions', this is only for /etc/pulp/cds/cds.conf.

Reason for rename is:
 - CDS runs as a Gofer plugin.  
 - Gofer uses INIConfig to parse the config file.
 - INIConfig honors DOS batch file syntax of 'rem' being a comment
   Therefore any lines beginning with 'rem' such as 'remove_old_versions. is interpreted to be a comment.
 - Our workaround is for the CDS to rename 'remove_old_versions' to 'cds_remove_old_versions'


QE:

Options explained:
 remove_old_versions:  is a boolean, true means we will limit older package synced, false means we will not.

 num_old_pkgs_keep: is an integer, if remove_old_versions=true, then we will look at this value and keep this many old versions of a package.  
  0 means we keep none (i.e. only latest package is downloaded, 0 old packages are kept).   
  1 means we download latest package AND we download 1 old package

Below are a few test scenario suggestions

1) RHUA, remove_old_versions=false
   CDS, remove_old_versions=false

2) RHUA, remove_old_versions=false
   CDS, remove_old_versions=true

3) RHUA, remove_old_versions=true, num_old_pkgs_keep=4
   CDS, remove_old_versions=true, num_old_pkg_keep=1


Note, the below scenario will result in many 404's on RHUA http log, as the CDS will attempt to sync packages that dont exist on the RHUA.
 RHUA, remove_old_versions=true
 CDS, remove_old_versions=false

Comment 5 Ina Panova 2013-11-04 11:27:15 UTC
tested in pulp-0.0.263-38.git.1.7f842df.el6.noarch

Move to Verified.

Comment 7 errata-xmlrpc 2013-12-17 20:10:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1854.html


Note You need to log in before you can comment on or make changes to this bug.