Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1264560 - As a user, my rpm sync finishes quickly when upstream metadata hasn't changed
Summary: As a user, my rpm sync finishes quickly when upstream metadata hasn't changed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: Tazim Kolhar
URL: http://projects.theforeman.org/issues...
Whiteboard:
: 1269833 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-18 19:41 UTC by Mike McCune
Modified: 2021-04-06 18:04 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Satellite was performing sync post-processing even if a repository did not change. The code was updated to skip this processing if it is not necessary.
Clone Of:
Environment:
Last Closed: 2016-01-21 07:42:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 2 0 High CLOSED - CURRENTRELEASE As a user, my rpm sync finishes quickly when upstream metadata hasn't changed 2016-02-08 19:01:33 UTC
Red Hat Product Errata RHBA-2016:0052 0 normal SHIPPED_LIVE Satellite 6.1.6 bug fix update 2016-01-21 12:40:53 UTC

Description Mike McCune 2015-09-18 19:41:27 UTC 
The repomd.xml file has its own timestamp in the XML, and then each file it references has its own timestamp. The treeinfo file contains its own timestamp, which already gets checked for freshness. We should check the repomd.xml timestamp and skip the rest of the sync if it hasn't changed. This will reduce the sync time for a big repo from possibly 1-2 minutes to a small number of seconds when the upstream repo hasn't changed. This will also help us be smarter about when the distribution files get re-downloaded.
Deliverables:

    rpm sync checks the timestamp in repomd.xml, and skips all unit types but "distribution" if it hasn't changed since the last sync.
    tell Katello team (specifically jsherrill) about this change once it's in Pulp. They might be able to remove some code on their end once this is in place.
    release notes
Comment 3 pulp-infra@redhat.com 2015-10-02 13:42:06 UTC 
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
Comment 4 pulp-infra@redhat.com 2015-10-02 13:42:09 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 5 Bryan Kearney 2015-10-09 19:52:23 UTC 
*** Bug 1269833 has been marked as a duplicate of this bug. ***
Comment 7 jcallaha 2015-11-04 17:43:32 UTC 
Failed QA in Satellite 6.1.4. Sync still takes ~3 minutes with no new packages.

Started at: 2015-11-04 17:00:29 UTC
Ended at: 2015-11-04 17:03:32 UTC

  RELEASE: Red Hat Enterprise Linux Server release 7.1 (Maipo)
  FOREMAN: 1.7.2
SATELLITE: 6.1.4
     RUBY: ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
   PUPPET: 3.6.2
Comment 8 jcallaha 2015-11-04 17:50:38 UTC 
Additionally, the repository being re-sync'd was Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server.
Comment 9 Mike McCune 2015-11-04 19:57:54 UTC 
So, this bug is 2 parts, we got the Pulp part included in 6.1.4 but there still needs to be work done in Katello to detect when no-new metadata was synced so we can skip reindexing and other tasks.

There will be improvement in 6.1.4 but not the entire resolution as needed.

Going to move this to 6.1.6 and have the Katello side of things addressed.
Comment 10 Justin Sherrill 2015-11-13 14:17:49 UTC 
Created redmine issue http://projects.theforeman.org/issues/12470 from this bug
Comment 11 Chris Peters 2015-12-17 14:33:46 UTC 
[SATELLITE-6.1.0 2b3202a] Fixes #12470 - skip post-sync actions if no new content was synced
 Author: Justin Sherrill <jsherril>
 Date: Fri Nov 13 09:43:01 2015 -0500
 9 files changed, 86 insertions(+), 22 deletions(-)
 create mode 100644 app/lib/actions/katello/repository/index_content.rb
 create mode 100644 app/lib/actions/middleware/execute_if_contents_changed.rb
Comment 12 Tazim Kolhar 2016-01-05 12:08:27 UTC 
Hi,

 please provide verification steps.

thanks and regards,
Tazim
Comment 13 Mike McCune 2016-01-12 23:59:36 UTC 
This caused a regression:

https://bugzilla.redhat.com/show_bug.cgi?id=1297308
Comment 16 Tazim Kolhar 2016-01-16 16:10:50 UTC 
FAILEDQA:

for sat6.1.6 compose 5
# rpm -qa | grep foreman
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch
foreman-vmware-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3.5-1.el7sat.noarch
foreman-selinux-1.7.2.17-1.el7sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.14-1.el7sat.noarch
foreman-ovirt-1.7.2.50-1.el7sat.noarch
foreman-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman_docker-1.2.0.24-1.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.15.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_docker-0.0.3.10-1.el7sat.noarch
foreman-debug-1.7.2.50-1.el7sat.noarch
foreman-proxy-1.7.2.8-1.el7sat.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch
foreman-discovery-image-3.0.5-3.el7sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch
foreman-libvirt-1.7.2.50-1.el7sat.noarch
foreman-gce-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman-0.1.4.15-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.23-1.el7sat.noarch
foreman-postgresql-1.7.2.50-1.el7sat.noarch
foreman-compute-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman-redhat_access-0.2.4-1.el7sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.10-1.el7sat.noarch


Tried to sync RHEL6.6 repo with no new packages : 
but still it took ~3 mins

Id: c287bf96-9010-4229-852d-925ddbaea7a8
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Started at: 2016-01-16 15:55:47 UTC
Ended at: 2016-01-16 15:58:21 UTC
State: stopped
Result: success
Params: repository 'rhel_repo'; product 'rhel6'; organization 'Default Organization'
Parent task
100.0% Complete

100%
Output:
No new packages.


As per my understanding, it should take less time since no new packages to
sync

Thanks and Regards,
Tazim 
No new packages.
Comment 17 Tazim Kolhar 2016-01-18 16:24:56 UTC 
Spoke to the developers,
got to the conclusion that

this much time is expected to download the metadata

Id: c287bf96-9010-4229-852d-925ddbaea7a8
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Started at: 2016-01-16 15:55:47 UTC
Ended at: 2016-01-16 15:58:21 UTC
State: stopped
Result: success
Params: repository 'rhel_repo'; product 'rhel6'; organization 'Default Organization'
Parent task
100.0% Complete
Comment 19 Pradeep Kumar Surisetty 2016-01-20 04:46:25 UTC 
Issue resolved for preprocess. Post process it takes same amount of time still

Other Bz:  https://bugzilla.redhat.com/show_bug.cgi?id=1279490
Comment 21 errata-xmlrpc 2016-01-21 07:42:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0052
Comment 22 pulp-infra@redhat.com 2016-02-08 19:01:34 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Note You need to log in before you can comment on or make changes to this bug.