Bug 1705099

Description of problem:

Katello uses wrong CA cert file to verify the ueber certificate which caused the ueber certificate to regenerate every time Satellite performing a Capsule content sync. This issue only happen if the Satellite is using custom SSL certificate.

Regeneration of the ueber certificate has a very bad effect. It is causing Satellite to update all importer and distributor configurations in the Capsule. Updating the importer/distributor will cause Pulp to do force full sync and publish. Therefore causing optimized capsule sync to be meaningless.

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/lib/actions/katello/capsule_content/sync.rb

def plan(smart_proxy, options = {})
  action_subject(smart_proxy)
  capsule_content = ::Katello::CapsuleContent.new(smart_proxy)
  capsule_content.ping_pulp
  capsule_content.verify_ueber_certs  <========== Verify ueber certifcate

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/services/cert/certs.rb

def self.verify_ueber_cert(organization)
  ueber_cert = OpenSSL::X509::Certificate.new(self.ueber_cert(organization)[:cert])
  cert_store = OpenSSL::X509::Store.new
  cert_store.add_file Setting[:ssl_ca_file]  <========== Is "/etc/foreman/proxy_ca.pem" which can be a custom SSL certificate. "SETTINGS[:katello][:candlepin][:ca_cert_file]" should be used for verification
  organization.regenerate_ueber_cert unless cert_store.verify ueber_cert
end

Steps to Reproduce:
1) Have a Satellite that uses custom SSL certificate and at least have a Capsule.
2) Add some repos to a CV. Publish and promote the CV.
3) Go to Infrastructure -> Capsule -> capsule hostname -> perform optimized sync.
4) In Satellite 6.4.2, you should see many set of UpdateImporter/UpdateDistributor tasks equal to the number of the Content view repos you sync.
5) Perform optimized capsule sync multiple times. You still see same number of UpdateImporter/UpdateDistributor tasks. Repos take long time to sync and all celery process are consuming ~100% cpu time.


Expected result:
If Satellite is performing optimized Capsule sync, it is expected to see many of the below messages in the /var/log/messages.

pulp_rpm.plugins.importers.yum.sync:INFO: [7fa268a6] upstream repo metadata has not changed. Skipping steps.
celery.app.trace:INFO: [78c89ac9] Task pulp.server.managers.repo.publish.publish[78c89ac9-e195-4b62-a057-a371eff543cc] succeeded in 0.024296627962s: {'exception': None, 'repo_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'traceback': None, 'started': '2019-05-01T12:28:02Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2019, 5, 1, 12, 28, 2, 62334, tzinfo=<isodate.tzinfo.Utc object at 0x7f7445e03510>), 'error_message': None, 'distributor_type_id': 'puppet_install_distributor', 'distributor_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'summary': 'Skipped: Repository content has not changed since last publish.', 'result': 'skipped', 'id': '5cc990d27399db03fc538eb3', 'details': 'Skipped: Repository content has not changed since last publish.'}


Actual Result:
/var/log/messages is full of the following messages even after performing optimized capsule sync multiple times:

pulp_rpm.plugins.importers.yum.sync:INFO: [0b8f1af6] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [5c6cb69e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [e66f214e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [27eb0633] Generating metadata databases.