Description of problem: Katello uses wrong CA cert file to verify the ueber certificate which caused the ueber certificate to regenerate every time Satellite performing a Capsule content sync. This issue only happen if the Satellite is using custom SSL certificate. Regeneration of the ueber certificate has a very bad effect. It is causing Satellite to update all importer and distributor configurations in the Capsule. Updating the importer/distributor will cause Pulp to do force full sync and publish. Therefore causing optimized capsule sync to be meaningless. In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/lib/actions/katello/capsule_content/sync.rb def plan(smart_proxy, options = {}) action_subject(smart_proxy) capsule_content = ::Katello::CapsuleContent.new(smart_proxy) capsule_content.ping_pulp capsule_content.verify_ueber_certs <========== Verify ueber certifcate In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/services/cert/certs.rb def self.verify_ueber_cert(organization) ueber_cert = OpenSSL::X509::Certificate.new(self.ueber_cert(organization)[:cert]) cert_store = OpenSSL::X509::Store.new cert_store.add_file Setting[:ssl_ca_file] <========== Is "/etc/foreman/proxy_ca.pem" which can be a custom SSL certificate. "SETTINGS[:katello][:candlepin][:ca_cert_file]" should be used for verification organization.regenerate_ueber_cert unless cert_store.verify ueber_cert end Steps to Reproduce: 1) Have a Satellite that uses custom SSL certificate and at least have a Capsule. 2) Add some repos to a CV. Publish and promote the CV. 3) Go to Infrastructure -> Capsule -> capsule hostname -> perform optimized sync. 4) In Satellite 6.4.2, you should see many set of UpdateImporter/UpdateDistributor tasks equal to the number of the Content view repos you sync. 5) Perform optimized capsule sync multiple times. You still see same number of UpdateImporter/UpdateDistributor tasks. Repos take long time to sync and all celery process are consuming ~100% cpu time. Expected result: If Satellite is performing optimized Capsule sync, it is expected to see many of the below messages in the /var/log/messages. pulp_rpm.plugins.importers.yum.sync:INFO: [7fa268a6] upstream repo metadata has not changed. Skipping steps. celery.app.trace:INFO: [78c89ac9] Task pulp.server.managers.repo.publish.publish[78c89ac9-e195-4b62-a057-a371eff543cc] succeeded in 0.024296627962s: {'exception': None, 'repo_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'traceback': None, 'started': '2019-05-01T12:28:02Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2019, 5, 1, 12, 28, 2, 62334, tzinfo=<isodate.tzinfo.Utc object at 0x7f7445e03510>), 'error_message': None, 'distributor_type_id': 'puppet_install_distributor', 'distributor_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'summary': 'Skipped: Repository content has not changed since last publish.', 'result': 'skipped', 'id': '5cc990d27399db03fc538eb3', 'details': 'Skipped: Repository content has not changed since last publish.'} Actual Result: /var/log/messages is full of the following messages even after performing optimized capsule sync multiple times: pulp_rpm.plugins.importers.yum.sync:INFO: [0b8f1af6] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [5c6cb69e] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [e66f214e] Generating metadata databases. pulp_rpm.plugins.importers.yum.sync:INFO: [27eb0633] Generating metadata databases.
Connecting redmine issue https://projects.theforeman.org/issues/26721 from this bug
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/26721 has been resolved.
This doesn't look correctly aligned to 6.6, putting back in POST
FailedQA. @satellite-6.6.0-5.beta.el7sat.noarch tfm-rubygem-katello-3.12.0.7-1.el7sat.noarch Fix not delivered: # grep 'cert_store.add_file' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-*/app/services/cert/certs.rb cert_store.add_file Setting[:ssl_ca_file]
Placing back in to POST. Hi Evgeni, Can you it looks like this one may have missed the builds? Can you take a peak and help ensure it lands in a future snap? I do see the changes in upstream katello. If you need anything, please do let us know. Thanks!
VERIFIED. @satellite-6.6.0-6.el7sat.noarch tfm-rubygem-katello-3.12.0.18-1.el7sat.noarch by the following manual reproducer: 1) Have Satellite & Capsule installed with custom certs 2) Sync a bunch of large yum repos (rhel7/rhel6 server) on Satellite with the external capsule 3) Trigger consequent capsule sync tasks and note the times # time hammer capsule content synchronize --id 2 >>> times [ 45min, 23sec, 25sec, 23sec ... ] the fix resolves the perf issue (after 1st full sync doing optimized syncs)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:3172