Bug 1492333 - Content Views dont copy SRPMs at all
Summary: Content Views dont copy SRPMs at all
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Views
Version: 6.2.11
Hardware: All
OS: Linux
high
high
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: Sachin Ghai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-16 13:27 UTC by Pavel Moravec
Modified: 2021-07-30 07:49 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
srpms count on UI under published cv (32.78 KB, image/png)
2017-12-13 11:55 UTC, Sachin Ghai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 21154 0 High Closed Content Views dont copy SRPMs at all 2021-02-08 13:58:23 UTC
Red Hat Bugzilla 1491638 0 high CLOSED Satellite 6: SRPM repos are not published via http 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 3186961 0 None None None 2017-09-16 15:35:53 UTC

Internal Links: 1491638

Description Pavel Moravec 2017-09-16 13:27:09 UTC
Description of problem:
Adding a Source RPM repo (with SRPMs and errata) to a Content View and publishing this C.V., the resulting C.V. repository is empty.

Moreover, due to https://bugzilla.redhat.com/show_bug.cgi?id=1491646, the resulting repo is never published (and Capsule sync fails).

Checking dynflow steps for C.V. publish:
- SRPMs are not copied at all
- errata are copied, but purged away at the end (since no underlying units - the missing SRPMs - are in the repo, so purging the just-added errata)


Version-Release number of selected component (if applicable):
Sat 6.2.11 (but imho any)


How reproducible:
100%


Steps to Reproduce:
1. Sync a Source RPM repo
2. Add it to a Content View and Publish
3. Set a Content Host to this C.V.
4. run "subscription-manager refresh; yum install some-SRPM" on the Content Host


Actual results:
yum fails on parsing metadata with error

<some-path>/source/SRPMS/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found


Expected results:
yum succeeds


Additional info:

Comment 2 Pavel Moravec 2017-09-16 14:32:21 UTC
Manual workaround fore C.V.s without filters - regrettably it fails on publishing the repos :(


1) identify (e.g. from dynflow console of CV publish, from steps Actions::Pulp::Repository::CopyErrata) the source_pulp_id and target_pulp_id. There should be _twice_ as many these steps as # of repos in the CVs are. For _each_ such step, store the repo names in source_pulp_id / target_pulp_id bash variables.


2) associate erratum, SRPMs, metadata and distribution between source and target repos - URI and POST params taken from katello debug logs:

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

for unit in erratum srpm yum_repo_metadata_file distribution; do
  curl -i -H "Content-Type: application/json" -X POST -d "{\"source_repo_id\":\"$source_repo_id\",\"criteria\":{\"type_ids\":[\"$unit\"],\"filters\":{}}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$target_repo_id/actions/associate/
done



3) Wait until the associate tasks are completed - below command should not list the tasks / UUIDs printed in 2) output:

pulp-admin -u admin -p $pulpAdminPassword tasks list


4) Once units association is completed / tasks are gone (usually happens within a second, but for large repos, it can be longer), publish the repo:

curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"$repo\",\"override_config\":{\"force_full\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$target_repo_id/actions/publish/


5) Again check via pulp-admin when the task finishes. Since that time, client should be able to access that repo and its content.


As written, it fails to publish the repo with exception:

2017-09-16T16:23:39.356378+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144) Task pulp.server.managers.repo.publish.publish[189b404a-f701-4edb-ae4e-b176924b3897] raised unexpected: MissingResource({'resource_id': {'repo_id': u'Default_Organization-Library-MRG_src-Red_Hat_Enterprise_MRG_Messaging_3_for_RHEL_7-Red_Hat_Enterprise_MRG_Messaging_3_for_RHEL_7_Source_RPMs_x86_64_7Server', 'distributor_id': u''}},)
2017-09-16T16:23:39.356557+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144) Traceback (most recent call last):
2017-09-16T16:23:39.356724+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
2017-09-16T16:23:39.356901+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     R = retval = fun(*args, **kwargs)
2017-09-16T16:23:39.357063+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 473, in __call__
2017-09-16T16:23:39.357222+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     return super(Task, self).__call__(*args, **kwargs)
2017-09-16T16:23:39.357375+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
2017-09-16T16:23:39.357549+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     return super(PulpTask, self).__call__(*args, **kwargs)
2017-09-16T16:23:39.357712+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
2017-09-16T16:23:39.357864+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     return self.run(*args, **kwargs)
2017-09-16T16:23:39.358012+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 961, in publish
2017-09-16T16:23:39.358173+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     dist = model.Distributor.objects.get_or_404(repo_id=repo_id, distributor_id=dist_id)
2017-09-16T16:23:39.358323+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)   File "/usr/lib/python2.7/site-packages/pulp/server/db/querysets.py", line 116, in get_or_404
2017-09-16T16:23:39.358477+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144)     raise pulp_exceptions.MissingResource(**kwargs)
2017-09-16T16:23:39.358634+02:00 pmoravec-sat62-rhel7 pulp: celery.worker.job:ERROR: (21540-90144) MissingResource: Missing resource(s): repo_id=Default_Organization-Library-MRG_src-Red_Hat_Enterprise_MRG_Messaging_3_for_RHEL_7-Red_Hat_Enterprise_MRG_Messaging_3_for_RHEL_7_Source_RPMs_x86_64_7Server, distributor_id=

(why distributor_id is empty???)

Comment 4 Pavel Moravec 2017-09-17 20:23:29 UTC
.. and here is a working version of the workaround:

bash script:


pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
source_repo_id=$1
target_repo_id=$2

for unit in erratum srpm yum_repo_metadata_file distribution; do
	curl -i -H "Content-Type: application/json" -X POST -d "{\"source_repo_id\":\"$source_repo_id\",\"criteria\":{\"type_ids\":[\"$unit\"],\"filters\":{}}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$target_repo_id/actions/associate/
done

pulp-admin -u admin -p $pulpAdminPassword tasks list

curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"${target_repo_id}_clone\",\"override_config\":{\"force_full\":true, \"source_repo_id\":\"$source_repo_id\", \"source_distributor_id\":\"$source_repo_id\"}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$target_repo_id/actions/publish/


Run it with 2 arguments source_repo_id and target_repo_id for each "CopyErrata" of source repository in the dynflow task, in the same ordering like in the task.

In case pulp-admin in the middle prints a running task "associate units", re-run the latest cmd manually.

After each execution of the script, ensure the publish task finished, before running next round for next pair of repos.

Comment 5 Brad Buckingham 2017-09-28 19:50:13 UTC
Created redmine issue http://projects.theforeman.org/issues/21154 from this bug

Comment 6 Satellite Program 2017-10-09 18:20:22 UTC
Upstream bug assigned to jsherril

Comment 7 Satellite Program 2017-10-09 18:20:25 UTC
Upstream bug assigned to jsherril

Comment 8 Satellite Program 2017-11-30 21:17:08 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21154 has been resolved.

Comment 9 Sachin Ghai 2017-12-13 11:54:24 UTC
Verified with sat6.3 snap28.

I can see synced SRPMS in content-view. UI shows the count after publish/promote. Please see the attached screenshot.

Comment 10 Sachin Ghai 2017-12-13 11:55:08 UTC
Created attachment 1367302 [details]
srpms count on UI under published cv

Comment 11 Sachin Ghai 2017-12-13 11:59:31 UTC
To test the fix, I synced sat6 srms and published them in two CVs a) cv_srpms b) cv_rhel73


Here are the list of source rpms published in a CV:

/var/lib/pulp/published/yum/master/yum_distributor/1-cv_srpms-v1_0-db74d53d-be62-451a-9dae-62c644009aa2/1513154587.51/Packages/t/tfm-rubygem-hammer_cli-0.5.1.13-2.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_srpms-v1_0-db74d53d-be62-451a-9dae-62c644009aa2/1513154587.51/Packages/t/tfm-rubygem-activerecord-session_store-0.1.2-1.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_srpms-v1_0-db74d53d-be62-451a-9dae-62c644009aa2/1513154587.51/Packages/t/tfm-rubygem-angular-rails-templates-0.1.2-4.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_srpms-v1_0-db74d53d-be62-451a-9dae-62c644009aa2/1513154587.51/Packages/t/tfm-rubygem-sprockets-rails-2.3.3-1.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_srpms-v1_0-db74d53d-be62-451a-9dae-62c644009aa2/1513154587.51/Packages/t/tfm-rubygem-rbovirt-0.0.37-1.el7sat.src.rpm


A few more in another CV:


/var/lib/pulp/published/yum/master/yum_distributor/1-cv_rhel73-v3_0-db74d53d-be62-451a-9dae-62c644009aa2/1513156318.1/Packages/r/rubygem-clamp-0.6.2-2.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_rhel73-v3_0-db74d53d-be62-451a-9dae-62c644009aa2/1513156318.1/Packages/r/rubygem-rack-1.4.1-13.el7sat.src.rpm
/var/lib/pulp/published/yum/master/yum_distributor/1-cv_rhel73-v3_0-db74d53d-be62-451a-9dae-62c644009aa2/1513156318.1/Packages/r/ruby-augeas-0.5.0-1.el7.src.rpm



[root@cloud-qe-14 661a8abb8a17787c68fa65f383ffda96f52adbc814fa082934b8ba208ccdbe]# ll
total 1876
-rw-r--r--. 1 apache apache 1918580 Dec 13 03:47 katello-installer-base-3.0.0.88-1.el7sat.src.rpm
[root@cloud-qe-14 661a8abb8a17787c68fa65f383ffda96f52adbc814fa082934b8ba208ccdbe]# pwd
/var/lib/pulp/content/units/srpm/d7/661a8abb8a17787c68fa65f383ffda96f52adbc814fa082934b8ba208ccdbe

Comment 12 Sachin Ghai 2017-12-13 12:02:51 UTC
Later, I tried to sync the contents to capsule to see if srpms got synced to external capsule:

capsule sync was successfully completed. here are the srpms from capsule.

[root@cloud-qe-06 ~]# cd /var/lib/pulp/content/units/srpm/2d/b449d3d12f90605ed66e8f9e25f258338acd9fc6972cac762086b124b7668f/
[root@cloud-qe-06 b449d3d12f90605ed66e8f9e25f258338acd9fc6972cac762086b124b7668f]# ll
total 1884
-rw-r--r--. 1 apache apache 1926473 Dec 13 03:48 katello-installer-base-3.0.0.95-1.el7sat.src.rpm
[root@cloud-qe-06 b449d3d12f90605ed66e8f9e25f258338acd9fc6972cac762086b124b7668f]# 


p]# cd /var/lib/pulp/published/yum/master/yum_distributor/1-cv_rhel73-Dev-db74d53d-be62-451a-9dae-62c644009aa2/1513165243.1/Packages/p/
[root@cloud-qe-06 p]# ls
pulp-2.8.3.3-1.el7sat.src.rpm         pulp-ostree-1.1.3.3-1.el7sat.src.rpm                         pyserial-2.6-5.el7.src.rpm                 python-mongoengine-0.10.5-2.el7sat.src.rpm

Comment 13 Sachin Ghai 2017-12-13 12:06:19 UTC
I provisioned a content host and registered it with same cv that includes the srpms. here is the output of repolist.


[root@sghaisrpmtest ~]# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                                                                           repo name                                                                                                  status
!Default_Organization_63capsule_63capsule_rhel7                                                   63capsule_rhel7                                                                                               176
!rhel-7-server-rpms/7Server/x86_64                                                                Red Hat Enterprise Linux 7 Server (RPMs)                                                                   17,802
!rhel-7-server-satellite-6.2-source-rpms/x86_64                                                   Red Hat Satellite 6.2 (for RHEL 7 Server) (Source RPMs)                                                       441
repolist: 18,419


~]# yumdownloader --source foreman
Loaded plugins: product-id, subscription-manager
Default_Organization_63capsule_63capsule_rhel7                                                                                                                                              | 2.5 kB  00:00:00     
rhel-7-server-rpms                                                                                                                                                                          | 2.0 kB  00:00:00     
rhel-7-server-satellite-6.2-source-rpms                                                                                                                                                     | 2.1 kB  00:00:00     
foreman-1.11.0.85-1.el7sat.src.rpm                                                                                                                                                          | 3.6 MB  00:00:00     
[root@sghaisrpmtest ~]# ls *src.rpm
foreman-1.11.0.85-1.el7sat.src.rpm

Comment 14 Satellite Program 2018-02-21 16:54:37 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.