Bug 1264560 - As a user, my rpm sync finishes quickly when upstream metadata hasn't changed
As a user, my rpm sync finishes quickly when upstream metadata hasn't changed
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
Unspecified
Unspecified Unspecified
unspecified Severity medium (vote)
: 6.1.6
: --
Assigned To: Justin Sherrill
Tazim Kolhar
http://projects.theforeman.org/issues...
: Triaged
: 1269833 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-18 15:41 EDT by Mike McCune
Modified: 2017-07-26 16:46 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Satellite was performing sync post-processing even if a repository did not change. The code was updated to skip this processing if it is not necessary.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-21 02:42:19 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Pulp Redmine 2 High CLOSED - CURRENTRELEASE As a user, my rpm sync finishes quickly when upstream metadata hasn't changed 2016-02-08 14:01 EST

  None (edit)
Description Mike McCune 2015-09-18 15:41:27 EDT
The repomd.xml file has its own timestamp in the XML, and then each file it references has its own timestamp. The treeinfo file contains its own timestamp, which already gets checked for freshness. We should check the repomd.xml timestamp and skip the rest of the sync if it hasn't changed. This will reduce the sync time for a big repo from possibly 1-2 minutes to a small number of seconds when the upstream repo hasn't changed. This will also help us be smarter about when the distribution files get re-downloaded.
Deliverables:

    rpm sync checks the timestamp in repomd.xml, and skips all unit types but "distribution" if it hasn't changed since the last sync.
    tell Katello team (specifically jsherrill) about this change once it's in Pulp. They might be able to remove some code on their end once this is in place.
    release notes
Comment 3 pulp-infra@redhat.com 2015-10-02 09:42:06 EDT
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
Comment 4 pulp-infra@redhat.com 2015-10-02 09:42:09 EDT
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 5 Bryan Kearney 2015-10-09 15:52:23 EDT
*** Bug 1269833 has been marked as a duplicate of this bug. ***
Comment 7 jcallaha 2015-11-04 12:43:32 EST
Failed QA in Satellite 6.1.4. Sync still takes ~3 minutes with no new packages.

Started at: 2015-11-04 17:00:29 UTC
Ended at: 2015-11-04 17:03:32 UTC

  RELEASE: Red Hat Enterprise Linux Server release 7.1 (Maipo)
  FOREMAN: 1.7.2
SATELLITE: 6.1.4
     RUBY: ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
   PUPPET: 3.6.2
Comment 8 jcallaha 2015-11-04 12:50:38 EST
Additionally, the repository being re-sync'd was Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server.
Comment 9 Mike McCune 2015-11-04 14:57:54 EST
So, this bug is 2 parts, we got the Pulp part included in 6.1.4 but there still needs to be work done in Katello to detect when no-new metadata was synced so we can skip reindexing and other tasks.

There will be improvement in 6.1.4 but not the entire resolution as needed.

Going to move this to 6.1.6 and have the Katello side of things addressed.
Comment 10 Justin Sherrill 2015-11-13 09:17:49 EST
Created redmine issue http://projects.theforeman.org/issues/12470 from this bug
Comment 11 Chris Peters 2015-12-17 09:33:46 EST
[SATELLITE-6.1.0 2b3202a] Fixes #12470 - skip post-sync actions if no new content was synced
 Author: Justin Sherrill <jsherril@redhat.com>
 Date: Fri Nov 13 09:43:01 2015 -0500
 9 files changed, 86 insertions(+), 22 deletions(-)
 create mode 100644 app/lib/actions/katello/repository/index_content.rb
 create mode 100644 app/lib/actions/middleware/execute_if_contents_changed.rb
Comment 12 Tazim Kolhar 2016-01-05 07:08:27 EST
Hi,

 please provide verification steps.

thanks and regards,
Tazim
Comment 13 Mike McCune 2016-01-12 18:59:36 EST
This caused a regression:

https://bugzilla.redhat.com/show_bug.cgi?id=1297308
Comment 16 Tazim Kolhar 2016-01-16 11:10:50 EST
FAILEDQA:

for sat6.1.6 compose 5
# rpm -qa | grep foreman
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-1.0-1.noarch
ruby193-rubygem-foreman_hooks-0.3.7-2.el7sat.noarch
foreman-vmware-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman_tasks-0.0.3.5-1.el7sat.noarch
foreman-selinux-1.7.2.17-1.el7sat.noarch
ruby193-rubygem-foreman_bootdisk-4.0.2.14-1.el7sat.noarch
foreman-ovirt-1.7.2.50-1.el7sat.noarch
foreman-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman_docker-1.2.0.24-1.el7sat.noarch
ruby193-rubygem-foreman-tasks-0.6.15.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_bootdisk-0.1.2.7-1.el7sat.noarch
rubygem-hammer_cli_foreman_docker-0.0.3.10-1.el7sat.noarch
foreman-debug-1.7.2.50-1.el7sat.noarch
foreman-proxy-1.7.2.8-1.el7sat.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-client-1.0-1.noarch
hp-dl180g6-01.rhts.eng.bos.redhat.com-foreman-proxy-client-1.0-1.noarch
foreman-discovery-image-3.0.5-3.el7sat.noarch
ruby193-rubygem-foreman_gutterball-0.0.1.9-1.el7sat.noarch
foreman-libvirt-1.7.2.50-1.el7sat.noarch
foreman-gce-1.7.2.50-1.el7sat.noarch
rubygem-hammer_cli_foreman-0.1.4.15-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-2.0.0.23-1.el7sat.noarch
foreman-postgresql-1.7.2.50-1.el7sat.noarch
foreman-compute-1.7.2.50-1.el7sat.noarch
ruby193-rubygem-foreman-redhat_access-0.2.4-1.el7sat.noarch
rubygem-hammer_cli_foreman_discovery-0.0.1.10-1.el7sat.noarch


Tried to sync RHEL6.6 repo with no new packages : 
but still it took ~3 mins

Id: c287bf96-9010-4229-852d-925ddbaea7a8
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Started at: 2016-01-16 15:55:47 UTC
Ended at: 2016-01-16 15:58:21 UTC
State: stopped
Result: success
Params: repository 'rhel_repo'; product 'rhel6'; organization 'Default Organization'
Parent task
100.0% Complete

100%
Output:
No new packages.


As per my understanding, it should take less time since no new packages to
sync

Thanks and Regards,
Tazim 
No new packages.
Comment 17 Tazim Kolhar 2016-01-18 11:24:56 EST
Spoke to the developers,
got to the conclusion that

this much time is expected to download the metadata

Id: c287bf96-9010-4229-852d-925ddbaea7a8
Label: Actions::Katello::Repository::Sync
Name: Synchronize
Owner: admin
Started at: 2016-01-16 15:55:47 UTC
Ended at: 2016-01-16 15:58:21 UTC
State: stopped
Result: success
Params: repository 'rhel_repo'; product 'rhel6'; organization 'Default Organization'
Parent task
100.0% Complete
Comment 19 Pradeep Kumar Surisetty 2016-01-19 23:46:25 EST
Issue resolved for preprocess. Post process it takes same amount of time still

Other Bz:  https://bugzilla.redhat.com/show_bug.cgi?id=1279490
Comment 21 errata-xmlrpc 2016-01-21 02:42:19 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0052
Comment 22 pulp-infra@redhat.com 2016-02-08 14:01:34 EST
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Note You need to log in before you can comment on or make changes to this bug.