Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 996198

Summary:	CDS sync should not attempt to sync older packages which have been removed from RHUA by way of 'num_old_packages_keep'
Product:	Red Hat Update Infrastructure for Cloud Providers	Reporter:	John Matthews <jmatthew>
Component:	CDS	Assignee:	John Matthews <jmatthew>
Status:	CLOSED ERRATA	QA Contact:	Ina Panova <ipanova>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.1.2	CC:	ahumbe, cbillett, dgao, ipanova, melewis, tsanders, whayutin
Target Milestone:	---
Target Release:	2.1.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	This update changes the behavior of a Content Delivery Server synchronization to allow a limit on the synchronization of older packages. The RHUA now sends two synchronization options to the Content Delivery Server when making a synchronization call. This allows the Content Delivery Server to match the behavior of the RHUA. Two configuration options have been added to the Content Delivery Server configuration to allow the Content Delivery Server to override options from the RHUA and further restrict how many older packages will be synchronized. This prevents errors on a RHUA when the RHUA is limiting older packages but the Content Delivery Server was unaware. The RHUA and Content Delivery Server are now more closely related in behavior. Additionally a Content Delivery Server now has the option to further reduce file storage requirements by only synchronizing the latest version of a package.	Story Points:	---
Clone Of:
Clones:	1017409 (view as bug list)		Environment:
Last Closed:	2013-12-17 20:10:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1017409

Description John Matthews 2013-08-12 15:50:55 UTC

Description of problem:

A RHUA can be configured to not sync a configurable number of older packages, yet the CDS does not honor the same configuration.  This results in a large number of 404 messages being logged on RHUA.

The RHUA syncs repo metadata from CDN and preserves this metadata with preserve_metadata=True.  The CDS syncs the metadata from the RHUA, then uses the package metadata to determine what should be synced from RHUA.  In case of the RHUA not syncing old packages the CDS will still attempt to download these missing packages, resulting in excessive 404 messages being logged by Apache.

Possible fix is to update CDS to honor the configuration options for number of old packages.

We could consider having the RHUA configuration parameters passed to the CDS as part of the sync.  Perhaps allowing a CDS configuration file as an additional override, so certain CDSes could be configured for storing even less older packages.

Comment 1 John Matthews 2013-08-12 15:55:40 UTC

See bug 989506 for more information.

Comment 3 John Matthews 2013-10-09 15:25:41 UTC

We have modified the RHUA to send the below sync options to the CDS.
 remove_old_versions
 num_old_pkgs_keep

Additionally we have modified the CDS so it may override any options from the RHUA by looking for these options in "/etc/pulp/cds/cds.conf"

Fix is in pulp 0.0.263-37

Below are the associated commits.

https://github.com/splice/pulp/commit/f85ef0477fd7bd24de810fc3788d9bc95b843079
https://github.com/splice/pulp/commit/33eb99928c42a0ef44850934238b6a3c8b784b7a
https://github.com/splice/pulp/commit/f9d840eb51d7b395af7636901d5f5d204f9e35bd
https://github.com/splice/pulp/commit/438e944f22d35393497023fb07cce23e8e9bd366
https://github.com/splice/pulp/commit/a9ce91818dae5e92c623b12ea571d6257869c819


Please note, we ran into an unexpected 'feature' with INIConfig on RHEL-6.  Namely, we had to rename the option 'remove_old_versions' to 'cds_remove_old_versions', this is only for /etc/pulp/cds/cds.conf.

Reason for rename is:
 - CDS runs as a Gofer plugin.  
 - Gofer uses INIConfig to parse the config file.
 - INIConfig honors DOS batch file syntax of 'rem' being a comment
   Therefore any lines beginning with 'rem' such as 'remove_old_versions. is interpreted to be a comment.
 - Our workaround is for the CDS to rename 'remove_old_versions' to 'cds_remove_old_versions'


QE:

Options explained:
 remove_old_versions:  is a boolean, true means we will limit older package synced, false means we will not.

 num_old_pkgs_keep: is an integer, if remove_old_versions=true, then we will look at this value and keep this many old versions of a package.  
  0 means we keep none (i.e. only latest package is downloaded, 0 old packages are kept).   
  1 means we download latest package AND we download 1 old package

Below are a few test scenario suggestions

1) RHUA, remove_old_versions=false
   CDS, remove_old_versions=false

2) RHUA, remove_old_versions=false
   CDS, remove_old_versions=true

3) RHUA, remove_old_versions=true, num_old_pkgs_keep=4
   CDS, remove_old_versions=true, num_old_pkg_keep=1


Note, the below scenario will result in many 404's on RHUA http log, as the CDS will attempt to sync packages that dont exist on the RHUA.
 RHUA, remove_old_versions=true
 CDS, remove_old_versions=false

Comment 5 Ina Panova 2013-11-04 11:27:15 UTC

tested in pulp-0.0.263-38.git.1.7f842df.el6.noarch

Move to Verified.

Comment 7 errata-xmlrpc 2013-12-17 20:10:10 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1854.html