Bug 1716578

+++ This bug was initially created as a clone of Bug #1705099 +++

Description of problem:

Katello uses wrong CA cert file to verify the ueber certificate which caused the ueber certificate to regenerate every time Satellite performing a Capsule content sync. This issue only happen if the Satellite is using custom SSL certificate.

Regeneration of the ueber certificate has a very bad effect. It is causing Satellite to update all importer and distributor configurations in the Capsule. Updating the importer/distributor will cause Pulp to do force full sync and publish. Therefore causing optimized capsule sync to be meaningless.

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/lib/actions/katello/capsule_content/sync.rb

def plan(smart_proxy, options = {})
  action_subject(smart_proxy)
  capsule_content = ::Katello::CapsuleContent.new(smart_proxy)
  capsule_content.ping_pulp
  capsule_content.verify_ueber_certs  <========== Verify ueber certifcate

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/services/cert/certs.rb

def self.verify_ueber_cert(organization)
  ueber_cert = OpenSSL::X509::Certificate.new(self.ueber_cert(organization)[:cert])
  cert_store = OpenSSL::X509::Store.new
  cert_store.add_file Setting[:ssl_ca_file]  <========== Is "/etc/foreman/proxy_ca.pem" which can be a custom SSL certificate. "SETTINGS[:katello][:candlepin][:ca_cert_file]" should be used for verification
  organization.regenerate_ueber_cert unless cert_store.verify ueber_cert
end

Steps to Reproduce:
1) Have a Satellite that uses custom SSL certificate and at least have a Capsule.
2) Add some repos to a CV. Publish and promote the CV.
3) Go to Infrastructure -> Capsule -> capsule hostname -> perform optimized sync.
4) In Satellite 6.4.2, you should see many set of UpdateImporter/UpdateDistributor tasks equal to the number of the Content view repos you sync.
5) Perform optimized capsule sync multiple times. You still see same number of UpdateImporter/UpdateDistributor tasks. Repos take long time to sync and all celery process are consuming ~100% cpu time.


Expected result:
If Satellite is performing optimized Capsule sync, it is expected to see many of the below messages in the /var/log/messages.

pulp_rpm.plugins.importers.yum.sync:INFO: [7fa268a6] upstream repo metadata has not changed. Skipping steps.
celery.app.trace:INFO: [78c89ac9] Task pulp.server.managers.repo.publish.publish[78c89ac9-e195-4b62-a057-a371eff543cc] succeeded in 0.024296627962s: {'exception': None, 'repo_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'traceback': None, 'started': '2019-05-01T12:28:02Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2019, 5, 1, 12, 28, 2, 62334, tzinfo=<isodate.tzinfo.Utc object at 0x7f7445e03510>), 'error_message': None, 'distributor_type_id': 'puppet_install_distributor', 'distributor_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'summary': 'Skipped: Repository content has not changed since last publish.', 'result': 'skipped', 'id': '5cc990d27399db03fc538eb3', 'details': 'Skipped: Repository content has not changed since last publish.'}


Actual Result:
/var/log/messages is full of the following messages even after performing optimized capsule sync multiple times:

pulp_rpm.plugins.importers.yum.sync:INFO: [0b8f1af6] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [5c6cb69e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [e66f214e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [27eb0633] Generating metadata databases.

--- Additional comment from  on 2019-05-01T13:30:58Z 

Since this bug report was entered in Red Hat Bugzilla, the 'sat-backlog' flag has been set to ? to ensure that it is properly evaluated for release.

--- Additional comment from  on 2019-05-01T13:30:58Z 

Since this issue was entered in Red Hat Bugzilla, the pm_ack has been set to + automatically for the next planned release.

--- Additional comment from  on 2019-05-03T00:24:00Z 

I added a pull request in the upstream case

https://github.com/Katello/katello/pull/8098

--- Additional comment from  on 2019-05-09T19:29:41Z 

Connecting redmine issue https://projects.theforeman.org/issues/26721 from this bug

--- Additional comment from  on 2019-05-10T20:05:52Z 

Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/26721 has been resolved.

--- Additional comment from  on 2019-05-14T00:00:32Z 

Hi

Since the patch has been merged. Can we have a hotfix for Satellite 6.4?

Thanks.

Regards
Hao

--- Additional comment from  on 2019-05-14T02:04:56Z 

Hi,

Can we get information on which future release the fix is targeted? Per,
  Doc Text: This has been fixed upstream. A future release will contain this fix.

Regards,

Josephine Alviso
GSS - APAC Brisbane

--- Additional comment from  on 2019-05-14T04:57:10Z 

Hey Hao, I've been on site and talked to the Customer who has also asked for a hotfix. Once we have identified that a hotfix is available, I'll be happy as the TAM to submit the required paperwork.

With thanks

Ché
___________________________________________________________

Ché Patterson, RHCE
Technical Account Manager
Red Hat Red Hat Asia Pacific Pty Ltd
Level 11

40 Marcus Clark Street Canberra Australia
che@redhat    T: 61261452823  
___________________________________________________________

--- Additional comment from  on 2019-05-15T02:00:35Z 

Hi Team

Please note that the slow optimized Capsule sync may happen once more time for each Capsule after applying the hotfix if customer has triggered any capsule sync (including auto trigger by content view publish) before applying the hotfix. This caused the regeneration of new ueber certificates so all Capsules needs to update the latest ueber certificates for all their repos on the next Capsule sync.


1) In dynflow console, if you still seeing non empty response for all repos like below. That means the Capsule need to update its importer for new ueber certificate.
Actions::Pulp::Repository::RefreshRun (success) [ 3.01s / 3.01s]
Output:

---
responses:
- spawned_tasks:
  - _href: "/pulp/api/v2/tasks/70a63e55-8227-478b-9ead-a055020bf9e9/"
    task_id: 70a63e55-8227-478b-9ead-a055020bf9e9
  result: 
  error: 
- spawned_tasks:
  - _href: "/pulp/api/v2/tasks/26d8d063-f1bd-44af-bac2-331f95b9013a/"
    task_id: 26d8d063-f1bd-44af-bac2-331f95b9013a
  result: 
  error: 
pulp_tasks: []

2) Let the capsule sync to finish and then trigger the optimized capsule sync for the same capsule again. This time, check the dynflow console again and should see empty responses now. This verified the fix.


3: Actions::Pulp::Repository::RefreshRun (success) [ 3.11s / 3.11s ]
Output:

---
responses: []
pulp_tasks: []


Thanks.

Regards
Hao

--- Additional comment from  on 2019-05-16T20:30:35Z 

Performance issue - requesting 6.4.z as well.

--- Additional comment from  on 2019-05-20T01:52:08Z 

Hotfix is available for Satellite 6.4.3

--- Additional comment from  on 2019-05-20T01:53:52Z 

Created attachment 1571053 [details]
Hotfix for Satellite 6.4.3

--- Additional comment from  on 2019-05-20T01:56:30Z 

Things to know after applying hotfix. see comment #9