Bug 1004001 - [RFE] Allow whitelist/blacklist for filtering yum repo syncs
[RFE] Allow whitelist/blacklist for filtering yum repo syncs
Status: CLOSED WONTFIX
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
Unspecified
Unspecified Unspecified
medium Severity medium (vote)
: Unspecified
: --
Assigned To: satellite6-bugs
Katello QA List
: FutureFeature, Reopened, Triaged
: 1157857 (view as bug list)
Depends On: 1003999
Blocks: 260381 754576 sat6-pulp-future 1293641
  Show dependency treegraph
 
Reported: 2013-09-03 12:57 EDT by Mike McCune
Modified: 2017-07-26 15:47 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1003999
Environment:
Last Closed: 2016-10-06 08:27:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mike McCune 2013-09-03 12:57:51 EDT
+++ This bug was initially created as a clone of Bug #1003999 +++

Filtering in pulp currently only effects published Content Views but there is no way to apply a filtering algorithm to a yum repository feed.

There are often cases where repositories contain large sets of packages that are never used (architectures, subdirectories, etc), eg:

http://linux.dell.com/repo/hardware/latest/platform_independent/rh60_64/

where the user may only want a certain subset of the large repo.

The customer should have the ability to specify on the repo a set of filters which prevent packages from being synced thus saving disk space and sync time.
Comment 1 Christina Plummer 2013-10-22 09:53:04 EDT
Another example is Oracle - the public repos for Oracle Linux include source RPMs in the same directory as the binary/noarch RPMs, e.g.

http://public-yum.oracle.com/repo/OracleLinux/OL5/latest/x86_64/

When I synced this repo, over half of the space consumed was from the source RPMs:

$ sudo du -Lks /var/lib/pulp/published/http/repos/ol5/x86_64/
7759976 /var/lib/pulp/published/http/repos/ol5/x86_64/
$ sudo du -Lks /var/lib/pulp/published/http/repos/ol5/x86_64/*.src.rpm | awk '{SUM+=$1} END{print SUM}'
3686144
Comment 2 Christina Plummer 2013-10-22 10:50:02 EDT
The other use that I would like to see is for cloning local repos - i.e. "syncing" from one repo to another in order to create a "promote to production" process.

One model developed using Pulp v1 is described in a Usenix paper from 2011 [1].  This would enable "less risky" (i.e. the majority of) packages from an upstream distributor to be promoted automatically to the internal production repositories, but packages that require additional testing (e.g. kernel, or applications like mysql or httpd) can be filtered from the automatic sync.  That way, the majority of updates can be pushed automatically to clients in a timely manner (or pulled via a standard "yum update"), while reducing the risk of introducting unexpected issues.  This model enables target package sets to be managed in one place, in the repository itself, rather than through excludes on each individual client.  For example, you might have 3 repositories:

"Live" = repo synced daily from upstream distributor
"Unstable" = repo synced daily from "Live", excluding $risky_pkgs
"Stable" = repo synced daily from "Unstable", excluding $risky_pkgs

$risky_pkgs would be manually promoted from Live -> Unstable -> Stable, for example weekly.

Similarly, different teams might choose to set different policies (maybe they want all kernel updates as soon as they are available, but they want to make sure that Python gets tested with their custom app first), so their $risky_pkgs_teamA list would be different.  It enables all teams within an organization to set their own policies while still inheriting certain organization-wide policy (e.g. that packages must be at least a day old before being installed anywhere).

It is not clear to me why the features of "cloning" and "sync filters" were removed between Pulp v1 and v2.

[1] https://www.usenix.org/legacy/events/lisa11/tech/full_papers/Pierre.pdf
Comment 3 Christina Plummer 2013-10-22 11:21:47 EDT
Based on the above use cases, I'd like to see sync filters for at least the following criteria:
 - content type (e.g. RPM, source RPM)
 - architecture (e.g. x86_64)
 - package name match (e.g. 'kernel*' or 'kmod*')
 - date package was added to the repo (e.g. "before 20131001" or "before -7days")

Thanks.
Comment 4 Randy Barlow 2014-12-02 11:20:12 EST
*** Bug 1157857 has been marked as a duplicate of this bug. ***
Comment 5 Andrea Giardini 2014-12-16 07:18:38 EST
Can we make a point on this and decide how we want to implement this?
I'd like to see this in Pulp since in my case is a "blocking" feature

Should we follow the old v1 approach? (Create a filter, link a filter to a repo)
Comment 6 Brian Bouterse 2015-02-18 20:11:50 EST
Moved to https://pulp.plan.io/issues/206
Comment 7 RHEL Product and Program Management 2015-03-04 06:24:27 EST
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 8 RHEL Product and Program Management 2015-03-18 15:23:21 EDT
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 9 pulp-infra@redhat.com 2015-10-08 10:01:17 EDT
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 10 pulp-infra@redhat.com 2016-01-27 09:01:36 EST
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2016-01-28 13:01:37 EST
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Note You need to log in before you can comment on or make changes to this bug.