+++ This bug was initially created as a clone of Bug #1705099 +++ Description of problem: Katello uses wrong CA cert file to verify the ueber certificate which caused the ueber certificate to regenerate every time Satellite performing a Capsule content sync. This issue only happen if the Satellite is using custom SSL certificate. Regeneration of the ueber certificate has a very bad effect. It is causing Satellite to update all importer and distributor configurations in the Capsule. Updating the importer/distributor will cause Pulp to do force full sync and publish. Therefore causing optimized capsule sync to be meaningless. In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/lib/actions/katello/capsule_content/sync.rb def plan(smart_proxy, options = {}) action_subject(smart_proxy) capsule_content = ::Katello::CapsuleContent.new(smart_proxy) capsule_content.ping_pulp capsule_content.verify_ueber_certs <========== Verify ueber certifcate In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/services/cert/certs.rb def self.verify_ueber_cert(organization) ueber_cert = OpenSSL::X509::Certificate.new(self.ueber_cert(organization)[:cert]) cert_store = OpenSSL::X509::Store.new cert_store.add_file Setting[:ssl_ca_file] <========== Is "/etc/foreman/proxy_ca.pem" which can be a custom SSL certificate. "SETTINGS[:katello][:candlepin][:ca_cert_file]" should be used for verification organization.regenerate_ueber_cert unless cert_store.verify ueber_cert end Steps to Reproduce: 1) Have a Satellite that uses custom SSL certificate and at least have a Capsule. 2) Add some repos to a CV. Publish and promote the CV. 3) Go to Infrastructure -> Capsule -> capsule hostname -> perform optimized sync. 4) In Satellite 6.4.2, you should see many set of UpdateImporter/UpdateDistributor tasks equal to the number of the Content view repos you sync. 5) Perform optimized capsule sync multiple times. You still see same number of UpdateImporter/UpdateDistributor tasks. Repos take long time to sync and all celery process are consuming ~100% cpu time. Expected result: If Satellite is performing optimized Capsule sync, it is expected to see many of the below messages in the /var/log/messages. pulp_rpm.plugins.importers.yum.sync:INFO: [7fa268a6] upstream repo metadata has not changed. Skipping steps. celery.app.trace:INFO: [78c89ac9] Task pulp.server.managers.repo.publish.publish[78c89ac9-e195-4b62-a057-a371eff543cc] succeeded in 0.024296627962s: {'exception': None, 'repo_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'traceback': None, 'started': '2019-05-01T12:28:02Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2019, 5, 1, 12, 28, 2, 62334, tzinfo=<isodate.tzinfo.Utc object at 0x7f7445e03510>), 'error_message': None, 'distributor_type_id': 'puppet_install_distributor', 'distributor_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'summary': 'Skipped: Repository content has not changed since last publish.', 'result': 'skipped', 'id': '5cc990d27399db03fc538eb3', 'details': 'Skipped: Repository content has not changed since last publish.'} Actual Result: /var/log/messages is full of the following messages even after performing optimized capsule sync multiple times: pulp_rpm.plugins.importers.yum.sync:INFO: [0b8f1af6] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [5c6cb69e] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [e66f214e] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [27eb0633] Generating metadata databases. --- Additional comment from on 2019-05-01T13:30:58Z Since this bug report was entered in Red Hat Bugzilla, the 'sat-backlog' flag has been set to ? to ensure that it is properly evaluated for release. --- Additional comment from on 2019-05-01T13:30:58Z Since this issue was entered in Red Hat Bugzilla, the pm_ack has been set to + automatically for the next planned release. --- Additional comment from on 2019-05-03T00:24:00Z I added a pull request in the upstream case https://github.com/Katello/katello/pull/8098 --- Additional comment from on 2019-05-09T19:29:41Z Connecting redmine issue https://projects.theforeman.org/issues/26721 from this bug --- Additional comment from on 2019-05-10T20:05:52Z Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/26721 has been resolved. --- Additional comment from on 2019-05-14T00:00:32Z Hi Since the patch has been merged. Can we have a hotfix for Satellite 6.4? Thanks. Regards Hao --- Additional comment from on 2019-05-14T02:04:56Z Hi, Can we get information on which future release the fix is targeted? Per, Doc Text: This has been fixed upstream. A future release will contain this fix. Regards, Josephine Alviso GSS - APAC Brisbane --- Additional comment from on 2019-05-14T04:57:10Z Hey Hao, I've been on site and talked to the Customer who has also asked for a hotfix. Once we have identified that a hotfix is available, I'll be happy as the TAM to submit the required paperwork. With thanks Ché ___________________________________________________________ Ché Patterson, RHCE Technical Account Manager Red Hat Red Hat Asia Pacific Pty Ltd Level 11 40 Marcus Clark Street Canberra Australia che@redhat T: 61261452823 ___________________________________________________________ --- Additional comment from on 2019-05-15T02:00:35Z Hi Team Please note that the slow optimized Capsule sync may happen once more time for each Capsule after applying the hotfix if customer has triggered any capsule sync (including auto trigger by content view publish) before applying the hotfix. This caused the regeneration of new ueber certificates so all Capsules needs to update the latest ueber certificates for all their repos on the next Capsule sync. 1) In dynflow console, if you still seeing non empty response for all repos like below. That means the Capsule need to update its importer for new ueber certificate. Actions::Pulp::Repository::RefreshRun (success) [ 3.01s / 3.01s] Output: --- responses: - spawned_tasks: - _href: "/pulp/api/v2/tasks/70a63e55-8227-478b-9ead-a055020bf9e9/" task_id: 70a63e55-8227-478b-9ead-a055020bf9e9 result: error: - spawned_tasks: - _href: "/pulp/api/v2/tasks/26d8d063-f1bd-44af-bac2-331f95b9013a/" task_id: 26d8d063-f1bd-44af-bac2-331f95b9013a result: error: pulp_tasks: [] 2) Let the capsule sync to finish and then trigger the optimized capsule sync for the same capsule again. This time, check the dynflow console again and should see empty responses now. This verified the fix. 3: Actions::Pulp::Repository::RefreshRun (success) [ 3.11s / 3.11s ] Output: --- responses: [] pulp_tasks: [] Thanks. Regards Hao --- Additional comment from on 2019-05-16T20:30:35Z Performance issue - requesting 6.4.z as well. --- Additional comment from on 2019-05-20T01:52:08Z Hotfix is available for Satellite 6.4.3 --- Additional comment from on 2019-05-20T01:53:52Z Created attachment 1571053 [details] Hotfix for Satellite 6.4.3 --- Additional comment from on 2019-05-20T01:56:30Z Things to know after applying hotfix. see comment #9
*** Bug 1716577 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1581