In 6.1 you can re-sync a large Library from Satellite to Capsule in under an hour With 6.2 it can take 6+ hours to re-sync even if there are no content changes. We need to get this back in line with 6.1's performance.
*** This bug has been marked as a duplicate of bug 1391704 ***
I don't think it's duplicite, as this talks about the whole capsule sync on environment, while https://bugzilla.redhat.com/show_bug.cgi?id=1391704 talks about one repo sync/public.
Created redmine issue http://projects.theforeman.org/issues/18032 from this bug
Myself and another developer were not able to reproduce this on 6.2.7. An initial sync took ~40 mins and a resync (with no new repos) took ~3 mins. If you are seeing poor performance on NOOP (resyncing with no or few updates) synchronizations between Satellite and Capsule, feel free to reopen
*** Bug 1418543 has been marked as a duplicate of this bug. ***
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18032 has been resolved.
Created attachment 1256938 [details] katello hotfix
Created attachment 1256939 [details] pulp hotfix
Dependent bugs are in 6.2.8 already.
Please add verifications steps for this bug to help QE verify
Steps to verify for QE: We want to check two things: the revision numbers of a repo in pulp aren't changing after a CVV promote/publish, and a capsule re-sync is fast when no content has changed (and after a CV publish/promote) 1) Sync a capsule to library with many repos (ideally 1000+). Then resync the capsule again, noting how long the 2nd sync took. 2) Create a content view with a large amount of the repos from library 3) check the revision number in pulp for some of those repos i.e grep revision /var/lib/pulp/published/yum/https/repos/<org>/Library/<content_view>/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml 4) Publish a new content view version 5) Check the revision numbers again and assert they haven't changed 6) re-sync the capsule and make sure it isn't taking much longer that the 2nd sync from step 1 7) Publish a new Content View Version and this time check "force yum metadata regeneration" 8) Check the revision numbers again and this time they should have changed let me know if you have any questions
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.7 ONLY === 1. Download attached files 2. Stop services on both Capsule and Satellite katello-service stop 3. Extract both tarballs on the Satellite server tar xvf hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2 tar xvf pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 4. Extract the only the pulp tarball on the Capsule servers tar xvf pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 5. Update the packages on both the Satellite and Capsule servers yum update *.rpm (be sure that the untared rpms are the only ones in the cwd) 6. Upgrade Satellite satellite-installer --scenario satellite --upgrade wait for this to finish and the Satellite to be running, then: 7. Upgrade the capsules satellite-installer --scenario capsule --upgrade To rollback any changes you can use yum history to view the yum transactions and yum history undo <yum_transaction_number> You may run into this error on the capsules when re-running the installer: Upgrade Step: start_httpd... Redirecting to /bin/systemctl start httpd.service Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. Some services failed to start: httpd in httpd logs: Feb 22 18:33:54 capsule627.example.com httpd[25756]: AH00526: Syntax error on line 28 of /etc/httpd/conf.d/pulp.conf: Feb 22 18:33:54 capsule627.example.com httpd[25756]: Name duplicates previous WSGI daemon definition. workaround: Manually edit /etc/httpd/conf.d/pulp.conf to only include the first commented out line and restart httpd` and re-run satellite-installer --scenario capsule --upgrade. Be sure to backup the original pulp.conf file to be safe.
=== HOTFIX DOWNLOAD URL == You can fetch the 2 tarballs as mentioned in prior comment from: * http://people.redhat.com/~mmccune/hotfix/1388296/hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2 * http://people.redhat.com/~mmccune/hotfix/1388296/pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 verify sums: $ sha256sum hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2 c928375ae05d1e81228e2bd126fcd242ca12b52cc4dd8e7cbd0c9fe70ef71e49 hotfix_katello-3.0.0.95-2.BZ1388296.el7sat.noarch.tar.bz2 $ sha256sum pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2 506ec90744999d2c9f5c68f2dd5bb68cf3a2987d3fb363f1fa386d62c964a266 pulp-hotfix-2.8.7.5-2.BZ_1388296.el7.tar.bz2
Btw, this hotfix reverses this suggested workaround here: https://bugzilla.redhat.com/show_bug.cgi?id=1370582 It needs to be re-applied if you had it previously.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Seems like normal upgrade path (2.6.7+hotfix -> 2.6.8) is not working, how do I proceed? Upgrade Step: update_http_conf... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: migrate_pulp... 28093 Attempting to connect to localhost:27017 Attempting to connect to localhost:27017 Write concern for Mongo connection: {} Loading content types. Loading type descriptors [] Parsing type descriptors Validating type descriptor syntactic integrity Validating type descriptor semantic integrity Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module Loading unit model: docker_blob = pulp_docker.plugins.models:Blob Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest Loading unit model: docker_image = pulp_docker.plugins.models:Image Loading unit model: docker_tag = pulp_docker.plugins.models:Tag Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory Loading unit model: iso = pulp_rpm.plugins.db.models:ISO Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile Updating the database with types [] Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image] Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image] Found the following type definitions that were not present in the update collection [node, repository] Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 226, in main return _auto_manage_db(options) File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 293, in _auto_manage_db migrate_database(options) File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 78, in migrate_database raise DataError(msg) DataError: The database for migration package pulp.server.db.migrations is at version 24, which is larger than the latest version available, 23. Upgrade step migrate_pulp failed. Check logs for more information.
Managed to work past this by reverting pulp migration 24 with the following commands in pulp_database: db.migration_trackers.update({"name":"pulp.server.db.migrations"}, {$set : {"version":23}}) db.repo_distributors.update({}, {$unset: {"last_updated": ""}}, true, true) db.repo_distributors.update({}, {$unset: {"last_override_config": ""}}, true, true) Then run the upgrade.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - COMPLETE. Updating the external tracker on this bug.
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.8 ONLY === This is an update to a previously released 6.2.8 hotfix that was improperly packaged. 1. Download attached file 1388296-6.2.8-hotfix-2.tar 2. Stop services on both Capsule and Satellite katello-service stop 3. mkdir /var/www/html/pub/1388296-hotfix 4. Copy and extract the tarball to /var/www/html/pub/1388296-hotfix 5. restorecon -r /var/www/html/pub/1388296-hotfix 6. cp /var/www/html/pub/1388296-hotfix/satellite6-hotfix.repo /etc/yum.repos.d/ 7. start httpd: systemctl start httpd.service 8. Update the packages on both the Satellite and Capsule servers yum update 9. Upgrade Satellite satellite-installer --upgrade 10. repeat above procedure on Capsules
Created attachment 1269030 [details] 1388296-6.2.8-hotfix-2.tar
NOTE: The above hotfix includes fixes for 3 bugs: * Bug 1410649 - Sync fails due to error - PulpExecutionException: Importer indicated a failed response * Bug 1404345 - Sync failure - DocumentTooLarge * Bug 1388296 - Regression: Syncing large Library of content from Satellite to Capsule takes hours even if no content changes
Test results from a system with: 453 repositories 855,529 packages Initial Sync - 11h 17m Started at: 2017-04-11 18:56:09 UTC Ended at: 2017-04-12 06:39:50 UTC Second Sync - 2h 25m Started at: 2017-04-12 13:52:19 UTC Ended at: 2017-04-12 16:37:08 UTC Third Sync - 2h 29m Started at: 2017-04-12 14:36:11 UTC Ended at: 2017-04-12 18:05:49 UTC
Verified in Satellite 6.2.9 Snap 2. I also covered the additional revision version checks upon content view publishing. Initial repomd revision -bash-4.2# grep revision repodata/repomd.xml <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492021344</revision> Republish without forcing metadata regeneration -bash-4.2# grep revision repodata/repomd.xml <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492021344</revision> Republish with forced metadata regeneration -bash-4.2# grep revision repodata/repomd.xml <repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><revision>1492022235</revision>
Does this cover puppet repositories too, or only rpm repositories? Testing with the hotfix in comment 50 all Actions::Pulp::Consumer::SyncCapsule steps complete quickly but one that takes +5,000 seconds, which is for a puppet repository.
(In reply to Julio Entrena Perez from comment #55) > Does this cover puppet repositories too, or only rpm repositories? > > Testing with the hotfix in comment 50 all > Actions::Pulp::Consumer::SyncCapsule steps complete quickly but one that > takes +5,000 seconds, which is for a puppet repository. To make the question more wide / generic: what types of repositories it improves? rpm, puppet, iso, KS, docker,..?
This enhancement is just for rpm repositories
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1191