Bug 1388296 - Regression: Syncing large Library of content from Satellite to Capsule takes hours even if no content changes
Regression: Syncing large Library of content from Satellite to Capsule takes ...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Capsule - Content (Show other bugs)
6.2.3
Unspecified Unspecified
high Severity high (vote)
: 6.2.9
: --
Assigned To: John Mitsch
jcallaha
http://projects.theforeman.org/issues...
: PrioBumpField, Regression, Reopened, Triaged
: 1418543 (view as bug list)
Depends On: 1383378 1395777
Blocks: 1353215 1394354
  Show dependency treegraph
 
Reported: 2016-10-25 00:37 EDT by Mike McCune
Modified: 2018-05-25 11:10 EDT (History)
29 users (show)

See Also:
Fixed In Version: pulp-2.8.7.7-1,rubygem-katello-3.0.0.119-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1394354 (view as bug list)
Environment:
Last Closed: 2017-05-01 09:55:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
1388296-6.2.8-hotfix-2.tar (6.93 MB, application/x-tar)
2017-04-05 12:17 EDT, Mike McCune
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 18032 None None None 2017-01-11 11:06 EST
Pulp Redmine 1724 Normal CLOSED - CURRENTRELEASE Publish should be a no-op if no units and no settings have changed since the last successful publish 2017-02-15 10:33 EST
Pulp Redmine 1928 High CLOSED - CURRENTRELEASE Publish should be operational if override config values were specified 2017-03-15 12:33 EDT
Pulp Redmine 2081 Normal CLOSED - CURRENTRELEASE All of repo's distributors are updated at end of publish 2017-03-15 12:33 EDT
Pulp Redmine 2648 Normal CLOSED - COMPLETE Backport #1724, Publish should be a no-op if no units and no settings have changed since the last successful publish 2017-03-30 19:08 EDT

  None (edit)
Description Mike McCune 2016-10-25 00:37:15 EDT
In 6.1 you can re-sync a large Library from Satellite to Capsule in under an hour

With 6.2 it can take 6+ hours to re-sync even if there are no content changes.

We need to get this back in line with 6.1's performance.
Comment 3 Zach Huntington-Meath 2016-11-22 11:47:15 EST

*** This bug has been marked as a duplicate of bug 1391704 ***
Comment 4 Ivan Necas 2016-11-24 16:12:46 EST
I don't think it's duplicite, as this talks about the whole capsule sync on environment, while https://bugzilla.redhat.com/show_bug.cgi?id=1391704 talks about one repo sync/public.
Comment 5 John Mitsch 2017-01-11 11:04:42 EST
Created redmine issue http://projects.theforeman.org/issues/18032 from this bug
Comment 6 John Mitsch 2017-02-02 15:15:22 EST
Myself and another developer were not able to reproduce this on 6.2.7. An initial sync took ~40 mins and a resync (with no new repos) took ~3 mins. 

If you are seeing poor performance on NOOP (resyncing with no or few updates) synchronizations between Satellite and Capsule, feel free to reopen
Comment 11 Brad Buckingham 2017-02-14 10:46:35 EST
*** Bug 1418543 has been marked as a duplicate of this bug. ***
Comment 12 pulp-infra@redhat.com 2017-02-15 10:33:34 EST
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2017-02-15 10:33:41 EST
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 15 pm-sat@redhat.com 2017-02-17 10:14:00 EST
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18032 has been resolved.
Comment 16 John Mitsch 2017-02-23 09:49 EST
Created attachment 1256938 [details]
katello hotfix
Comment 17 John Mitsch 2017-02-23 09:49 EST
Created attachment 1256939 [details]
pulp hotfix
Comment 19 Bryan Kearney 2017-02-23 14:01:32 EST
Dependent bugs are in 6.2.8 already.
Comment 20 pm-sat@redhat.com 2017-02-23 16:09:52 EST
Please add verifications steps for this bug to help QE verify
Comment 21 John Mitsch 2017-02-24 08:53:47 EST
Steps to verify for QE:

We want to check two things: the revision numbers of a repo in pulp aren't changing after a CVV promote/publish, and a capsule re-sync is fast when no content has changed (and after a CV publish/promote)

1) Sync a capsule to library with many repos (ideally 1000+). Then resync the capsule again, noting how long the 2nd sync took. 
2) Create a content view with a large amount of the repos from library
3) check the revision number in pulp for some of those repos
i.e grep revision /var/lib/pulp/published/yum/https/repos/<org>/Library/<content_view>/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml
4) Publish a new content view version
5) Check the revision numbers again and assert they haven't changed
6) re-sync the capsule and make sure it isn't taking much longer that the 2nd sync from step 1
7) Publish a new Content View Version and this time check "force yum metadata regeneration"
8) Check the revision numbers again and this time they should have changed


let me know if you have any questions
Comment 22 Mike McCune 2017-03-01 11:28:37 EST
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.7 ONLY ===

1. Download attached files
2. Stop services on both Capsule and Satellite
  katello-service stop
3. Extract both tarballs on the Satellite server
  tar xvf hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2
  tar xvf pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 
4. Extract the only the pulp tarball on the Capsule servers
  tar xvf pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 
5. Update the packages on both the Satellite and Capsule servers
  yum update *.rpm (be sure that the untared rpms are the only ones in the cwd)
6. Upgrade Satellite
  satellite-installer --scenario satellite --upgrade
  wait for this to finish and the Satellite to be running, then:
7. Upgrade the capsules
  satellite-installer --scenario capsule --upgrade
  
  
To rollback any changes you can use yum history to view the yum transactions and  yum history undo <yum_transaction_number>

You may run into this error on the capsules when re-running the installer:
Upgrade Step: start_httpd...
Redirecting to /bin/systemctl start  httpd.service
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Some services failed to start: httpd

in httpd logs:
Feb 22 18:33:54 capsule627.example.com httpd[25756]: AH00526: Syntax error on line 28 of /etc/httpd/conf.d/pulp.conf:
Feb 22 18:33:54 capsule627.example.com httpd[25756]: Name duplicates previous WSGI daemon definition.

workaround:
Manually edit /etc/httpd/conf.d/pulp.conf to only include the first commented out line and restart httpd` and re-run satellite-installer --scenario capsule --upgrade. Be sure to backup the original pulp.conf file to be safe.
Comment 23 Mike McCune 2017-03-02 17:14:02 EST
=== HOTFIX DOWNLOAD URL ==

You can fetch the 2 tarballs as mentioned in prior comment from:

 * http://people.redhat.com/~mmccune/hotfix/1388296/hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2

 * http://people.redhat.com/~mmccune/hotfix/1388296/pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2


verify sums:

$ sha256sum hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2
c928375ae05d1e81228e2bd126fcd242ca12b52cc4dd8e7cbd0c9fe70ef71e49  hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2
$ sha256sum pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2
506ec90744999d2c9f5c68f2dd5bb68cf3a2987d3fb363f1fa386d62c964a266  pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2
Comment 24 Johan Bergström 2017-03-06 04:34:05 EST
Btw, this hotfix reverses this suggested workaround here: https://bugzilla.redhat.com/show_bug.cgi?id=1370582

It needs to be re-applied if you had it previously.
Comment 36 pulp-infra@redhat.com 2017-03-15 12:33:06 EDT
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 37 pulp-infra@redhat.com 2017-03-15 12:33:13 EDT
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 38 pulp-infra@redhat.com 2017-03-15 12:33:55 EDT
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 39 pulp-infra@redhat.com 2017-03-15 12:34:02 EDT
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 43 Johan Bergström 2017-03-21 05:56:05 EDT
Seems like normal upgrade path (2.6.7+hotfix -> 2.6.8) is not working, how do I proceed?

Upgrade Step: update_http_conf...

Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: migrate_pulp...
28093

Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image]
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Found the following type definitions that were not present in the update collection [node, repository]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 226, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 293, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 78, in migrate_database
    raise DataError(msg)
DataError: The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23.


Upgrade step migrate_pulp failed. Check logs for more information.
Comment 44 Johan Bergström 2017-03-21 10:12:59 EDT
Managed to work past this by reverting pulp migration 24 with the following commands in pulp_database:

db.migration_trackers.update({"name":"pulp.server.db.migrations"}, {$set : {"version":23}})
db.repo_distributors.update({}, {$unset: {"last_updated": ""}}, true, true)
db.repo_distributors.update({}, {$unset: {"last_override_config": ""}}, true, true)

Then run the upgrade.
Comment 46 pulp-infra@redhat.com 2017-03-29 21:33:01 EDT
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 47 pulp-infra@redhat.com 2017-03-29 21:33:09 EDT
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 48 pulp-infra@redhat.com 2017-03-30 19:08:35 EDT
The Pulp upstream bug status is at CLOSED - COMPLETE. Updating the external tracker on this bug.
Comment 49 Mike McCune 2017-04-05 12:16:19 EDT
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.8 ONLY ===

This is an update to a previously released 6.2.8 hotfix that was improperly packaged.


1. Download attached file 1388296-6.2.8-hotfix-2.tar
2. Stop services on both Capsule and Satellite
  katello-service stop
3. mkdir /var/www/html/pub/1388296-hotfix
4. Copy and extract the tarball to /var/www/html/pub/1388296-hotfix
5. restorecon -r /var/www/html/pub/1388296-hotfix
6. cp /var/www/html/pub/1388296-hotfix/satellite6-hotfix.repo /etc/yum.repos.d/
7. start httpd: systemctl start httpd.service
8. Update the packages on both the Satellite and Capsule servers
  yum update 
9. Upgrade Satellite
  satellite-installer --upgrade
10. repeat above procedure on Capsules
Comment 50 Mike McCune 2017-04-05 12:17 EDT
Created attachment 1269030 [details]
1388296-6.2.8-hotfix-2.tar
Comment 51 Mike McCune 2017-04-06 15:23:11 EDT
NOTE:

The above hotfix includes fixes for 3 bugs:

* Bug 1410649 - Sync fails due to error - PulpExecutionException: Importer indicated a failed response
* Bug 1404345 - Sync failure - DocumentTooLarge 
* Bug 1388296 - Regression: Syncing large Library of content from Satellite to Capsule takes hours even if no content changes
Comment 53 jcallaha 2017-04-12 15:08:08 EDT
Test results from a system with:
453 repositories
855,529 packages

Initial Sync - 11h 17m
Started at: 2017-04-11 18:56:09 UTC
Ended at: 2017-04-12 06:39:50 UTC

Second Sync - 2h 25m
Started at: 2017-04-12 13:52:19 UTC
Ended at: 2017-04-12 16:37:08 UTC

Third Sync - 2h 29m
Started at: 2017-04-12 14:36:11 UTC
Ended at: 2017-04-12 18:05:49 UTC
Comment 54 jcallaha 2017-04-12 15:48:30 EDT
Verified in Satellite 6.2.9 Snap 2. 
I also covered the additional revision version checks upon content view publishing.

Initial repomd revision
-bash-4.2# grep revision repodata/repomd.xml 
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492021344</revision>

Republish without forcing metadata regeneration
-bash-4.2# grep revision repodata/repomd.xml 
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492021344</revision>

Republish with forced metadata regeneration
-bash-4.2# grep revision repodata/repomd.xml 
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492022235</revision>
Comment 55 Julio Entrena Perez 2017-04-13 04:58:23 EDT
Does this cover puppet repositories too, or only rpm repositories?

Testing with the hotfix in comment 50 all Actions::Pulp::Consumer::SyncCapsule steps complete quickly but one that takes +5,000 seconds, which is for a puppet repository.
Comment 56 Pavel Moravec 2017-04-13 05:46:55 EDT
(In reply to Julio Entrena Perez from comment #55)
> Does this cover puppet repositories too, or only rpm repositories?
> 
> Testing with the hotfix in comment 50 all
> Actions::Pulp::Consumer::SyncCapsule steps complete quickly but one that
> takes +5,000 seconds, which is for a puppet repository.

To make the question more wide / generic: what types of repositories it improves? rpm, puppet, iso, KS, docker,..?
Comment 57 John Mitsch 2017-04-13 08:02:16 EDT
This enhancement is just for rpm repositories
Comment 60 errata-xmlrpc 2017-05-01 09:55:46 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1191

Note You need to log in before you can comment on or make changes to this bug.