Bug 1335621 - [RFE] Ignore warnings when syncing repos and SRC packages are missing [NEEDINFO]
Summary: [RFE] Ignore warnings when syncing repos and SRC packages are missing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Repositories
Version: 6.2.0
Hardware: Unspecified
OS: Linux
high
medium vote
Target Milestone: Released
Assignee: Partha Aji
QA Contact: jcallaha
URL:
Whiteboard:
: 1591358 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-12 17:09 UTC by Dylan Gross
Modified: 2019-10-07 17:18 UTC (History)
11 users (show)

Fixed In Version: tfm-rubygem-katello-3.8.0,pulp-rpm-2.17.0
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:36:19 UTC
bkearney: needinfo? (bkearney)


Attachments (Terms of Use)
verification screenshot (83.41 KB, image/png)
2019-01-16 20:35 UTC, jcallaha
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:36:28 UTC
Pulp Redmine 3440 Normal CLOSED - CURRENTRELEASE It is not possible to skip SRPMs during sync, only together with RPMs 2018-05-02 14:32:18 UTC
Foreman Issue Tracker 22802 None None None 2018-03-06 21:57:01 UTC
Foreman Issue Tracker 22803 None None None 2018-03-06 22:01:00 UTC

Description Dylan Gross 2016-05-12 17:09:29 UTC
Description of problem:

When pulp syncs a repo, if the SRC package is not found in the Repo, this will cause the repo sync task to complete, but end with a warning.   In Satellite 5.x, there was a deliberate change to ignore the warnings when SRC rpms could not be found (BZ#626749)

Version-Release number of selected component (if applicable):

   Satellite 6.2.0 (Beta) and earlier.


How reproducible:   Always



Steps to Reproduce:

   1. Add several rpm files to a web server that will be accessible to Satellite 6 for syncing

 # find test_repo/ -type f
test_repo/rpm/src/yum-3.4.3-0.src.rpm
test_repo/rpm/noarch/yum-3.4.3-0.noarch.rpm

2. Create the repo metadata
 # createrepo -v test_repo/

3. Ensure the metadata contains the location for both rpms
 # zcat test_repo/repodata/primary.xml.gz  | grep "location"
<location href="rpm/noarch/yum-3.4.3-0.noarch.rpm"/>
<location href="rpm/src/yum-3.4.3-0.src.rpm"/>

4. Remove the source rpm from the filesystem
 # rm test_repo/rpm/src/yum-3.4.3-0.src.rpm 

 # find test_repo/ -type f
test_repo/rpm/noarch/yum-3.4.3-0.noarch.rpm
test_repo/repodata/primary.xml.gz
test_repo/repodata/other.xml.gz
test_repo/repodata/repomd.xml
test_repo/repodata/filelists.xml.gz

5. Add the repo to a product in Satellite 6
New product -> test_repo_sync

6. Add the repo to the new product
New repository
  name: test_repo
  type: yum
  url:  http://satellite.example.com/repo/custom/test_repo


7. Sync the repo
Product Test_repo_sync -> repositories -> test_repo -> sync now

Errors:  Failed to download 1 package
An error occurred during the sync 
{"url"=>"http://satellite.example.com/repo/custom/test_repo/rpm/src/yum-3.4.3-0.src.rpm", "errors"=>["Not Found"]}


Actual results:

   Repo sync task completes, but with a Result = Warning and backtraces in the Error tab and an exception similar to .....

   RuntimeError: An error occurred during the sync 
{"url"=>"http://satellite.example.com/bash-4.1.2-33.el6.src.rpm", "errors"=>["Not Found"]}
 

Expected results:

    No Result = Warning if the SRC rpm is 404 Not Found.   (Though it is expected if this happens for a non-Src rpm.


Additional info:

    In using Satellite in conjunction with some 3rd party repos (For example SuSE's SMT application), the option to not sync src RPMs is given.   When Satellite then syncs against these repos, all tasks syncing against a repo like this end with Result == Warning.  Whereas in Satellite 5.x, this appears to have deliberately changed to ignore missing SRC rpms with BZ#626749

Comment 3 Michael Hrivnak 2016-06-13 17:29:22 UTC
Pulp's rpm importer has a "skip" feature that would resolve this. You can tell it which types of content to skip during sync, and "srpm" is one of the choices. Katello could make that setting available.

http://pulp-rpm.readthedocs.io/en/2.8-release/tech-reference/yum-plugins.html#configuration-parameters

If you want to try downloading srpms, and just silence any errors in case any are not found, that is not something pulp can do. My recommendation would be for the user interface (katello in this case) to just now show the warnings for those files.

Comment 4 Partha Aji 2018-03-06 21:16:45 UTC
Regarding comment #3 I do not see a "srpm" skip option available for yum repos. Even according to the documentation 
https://docs.pulpproject.org/plugins/pulp_rpm/tech-reference/yum-plugins.html#yum-importer  the skip option is listed without srpm
""

skip
    List of content types to be skipped during the repository synchronization. If unspecified, all types will be synchronized. Valid values are: rpm, drpm, distribution, erratum; default is [].
""

more over pulp wont even accept that in pulp-admin
""
$ pulp rpm repo create --repo-id=j123 --feed=http://<fqdn>/upstream-repo --skip=srpm      

Validation failed for argument [--skip]: Types must be a comma-separated list                        using only the following values: rpm, drpm, distribution, erratum                                   
""

I was able to set "srpm" as importer option via runcible which directly interacts with the pulp api. Even then the sync call on the repo synced srpm instead of skipping.

Comment 5 Partha Aji 2018-03-06 21:56:58 UTC
Connecting redmine issue http://projects.theforeman.org/issues/22802 from this bug

Comment 6 Partha Aji 2018-03-06 22:00:58 UTC
Connecting redmine issue http://projects.theforeman.org/issues/22803 from this bug

Comment 7 pulp-infra@redhat.com 2018-03-06 22:02:10 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2018-03-06 22:02:14 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2018-03-26 14:32:13 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2018-04-05 21:41:10 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2018-04-09 15:32:37 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2018-04-24 17:02:06 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2018-05-02 14:32:19 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 15 pm-sat@redhat.com 2018-06-11 20:18:28 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/22803 has been resolved.

Comment 16 Brad Buckingham 2018-06-29 20:05:38 UTC
*** Bug 1591358 has been marked as a duplicate of this bug. ***

Comment 17 jcallaha 2019-01-16 20:34:46 UTC
Verified in Satellite 6.5.0 Snap 11

The feature introduced was to add the ability to skip syncing of specified content types. 
for this bug, I have selected source rpms.

Setup:
1. Download rpms and source rpms into one directory
# wgetdir http://my.secret.host/x86_64/baseos/os/Packages/
# wgetdir http://my.secret.host/x86_64/baseos/source/SRPMS/
# ls | grep .rpm | wc -l
5006
# ls | grep src.rpm | wc -l
961
    This results in 4045 regular rpms.

2. Turn the rpms and source rpms into a repository
# createrepo .

3. Host the repository
podman run -d -p 1337:80 -v $(pwd):/usr/local/apache2/htdocs/:Z httpd

4. Create a new repository
set the content type to yum
set the url as http://<my.repo.host>:1337/
select source rpms to be ignored.

5. Synchronize the repository

Results:
The repo sync'd correctly ignoring all source rpms
The number of rpms were 4045 with 0 source rpms after sync.
See attached screenshot for verification.

Comment 18 jcallaha 2019-01-16 20:35:47 UTC
Created attachment 1521114 [details]
verification screenshot

Comment 20 Bryan Kearney 2019-03-08 14:47:28 UTC
test2

Comment 23 errata-xmlrpc 2019-05-14 12:36:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.