Bug 1705099 - Regeneration of ueber certificate is causing optimized capsule sync to perform force full sync every time.
Summary: Regeneration of ueber certificate is causing optimized capsule sync to perfor...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Capsule - Content
Version: 6.4.2
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: 6.6.0
Assignee: Hao Chang Yu
QA Contact: Lukas Pramuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-01 13:30 UTC by Hao Chang Yu
Modified: 2019-11-07 12:02 UTC (History)
16 users (show)

Fixed In Version: tfm-rubygem-katello-3.12.0.9-1
Doc Type: Known Issue
Doc Text:
This has been fixed upstream. A future release will contain this fix.
Clone Of:
: 1716577 1716578 1717700 (view as bug list)
Environment:
Last Closed: 2019-10-22 12:47:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 26721 'High' 'Closed' 'Regeneration of ueber certificate is causing optimized capsule sync to perform force full sync every time' 2019-12-04 02:30:22 UTC
Red Hat Product Errata RHSA-2019:3172 None None None 2019-10-22 12:47:28 UTC

Description Hao Chang Yu 2019-05-01 13:30:56 UTC
Description of problem:

Katello uses wrong CA cert file to verify the ueber certificate which caused the ueber certificate to regenerate every time Satellite performing a Capsule content sync. This issue only happen if the Satellite is using custom SSL certificate.

Regeneration of the ueber certificate has a very bad effect. It is causing Satellite to update all importer and distributor configurations in the Capsule. Updating the importer/distributor will cause Pulp to do force full sync and publish. Therefore causing optimized capsule sync to be meaningless.

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/lib/actions/katello/capsule_content/sync.rb

def plan(smart_proxy, options = {})
  action_subject(smart_proxy)
  capsule_content = ::Katello::CapsuleContent.new(smart_proxy)
  capsule_content.ping_pulp
  capsule_content.verify_ueber_certs  <========== Verify ueber certifcate

In /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.56/app/services/cert/certs.rb

def self.verify_ueber_cert(organization)
  ueber_cert = OpenSSL::X509::Certificate.new(self.ueber_cert(organization)[:cert])
  cert_store = OpenSSL::X509::Store.new
  cert_store.add_file Setting[:ssl_ca_file]  <========== Is "/etc/foreman/proxy_ca.pem" which can be a custom SSL certificate. "SETTINGS[:katello][:candlepin][:ca_cert_file]" should be used for verification
  organization.regenerate_ueber_cert unless cert_store.verify ueber_cert
end

Steps to Reproduce:
1) Have a Satellite that uses custom SSL certificate and at least have a Capsule.
2) Add some repos to a CV. Publish and promote the CV.
3) Go to Infrastructure -> Capsule -> capsule hostname -> perform optimized sync.
4) In Satellite 6.4.2, you should see many set of UpdateImporter/UpdateDistributor tasks equal to the number of the Content view repos you sync.
5) Perform optimized capsule sync multiple times. You still see same number of UpdateImporter/UpdateDistributor tasks. Repos take long time to sync and all celery process are consuming ~100% cpu time.


Expected result:
If Satellite is performing optimized Capsule sync, it is expected to see many of the below messages in the /var/log/messages.

pulp_rpm.plugins.importers.yum.sync:INFO: [7fa268a6] upstream repo metadata has not changed. Skipping steps.
celery.app.trace:INFO: [78c89ac9] Task pulp.server.managers.repo.publish.publish[78c89ac9-e195-4b62-a057-a371eff543cc] succeeded in 0.024296627962s: {'exception': None, 'repo_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'traceback': None, 'started': '2019-05-01T12:28:02Z', '_ns': 'repo_publish_results', 'completed': datetime.datetime(2019, 5, 1, 12, 28, 2, 62334, tzinfo=<isodate.tzinfo.Utc object at 0x7f7445e03510>), 'error_message': None, 'distributor_type_id': 'puppet_install_distributor', 'distributor_id': '1-hao_main_cv-Library-puppet-d4028fb0-c8fc-4236-a10f-255ad509db9d', 'summary': 'Skipped: Repository content has not changed since last publish.', 'result': 'skipped', 'id': '5cc990d27399db03fc538eb3', 'details': 'Skipped: Repository content has not changed since last publish.'}


Actual Result:
/var/log/messages is full of the following messages even after performing optimized capsule sync multiple times:

pulp_rpm.plugins.importers.yum.sync:INFO: [0b8f1af6] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [5c6cb69e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [e66f214e] Generating metadata databases.
pulp_rpm.plugins.importers.yum.sync:INFO: [27eb0633] Generating metadata databases.

Comment 4 Jonathon Turel 2019-05-09 19:29:41 UTC
Connecting redmine issue https://projects.theforeman.org/issues/26721 from this bug

Comment 5 Bryan Kearney 2019-05-10 20:05:52 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/26721 has been resolved.

Comment 15 Mike McCune 2019-06-06 22:33:33 UTC
This doesn't look correctly aligned to 6.6, putting back in POST

Comment 17 Lukas Pramuk 2019-07-15 15:18:10 UTC
FailedQA.

@satellite-6.6.0-5.beta.el7sat.noarch
tfm-rubygem-katello-3.12.0.7-1.el7sat.noarch


Fix not delivered:

# grep 'cert_store.add_file' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-*/app/services/cert/certs.rb
      cert_store.add_file Setting[:ssl_ca_file]

Comment 18 Brad Buckingham 2019-07-23 19:57:18 UTC
Placing back in to POST.

Hi Evgeni,

Can you it looks like this one may have missed the builds?  Can you take a peak and help ensure it lands in a future snap?

I do see the changes in upstream katello.  If you need anything, please do let us know.   Thanks!

Comment 23 Lukas Pramuk 2019-09-04 14:22:52 UTC
VERIFIED.

@satellite-6.6.0-6.el7sat.noarch
tfm-rubygem-katello-3.12.0.18-1.el7sat.noarch

by the following manual reproducer:


1) Have Satellite & Capsule installed with custom certs

2) Sync a bunch of large yum repos (rhel7/rhel6 server) on Satellite with the external capsule

3) Trigger consequent capsule sync tasks and note the times

# time hammer capsule content synchronize --id 2

>>> times [ 45min, 23sec, 25sec, 23sec ... ] the fix resolves the perf issue (after 1st full sync doing optimized syncs)

Comment 25 errata-xmlrpc 2019-10-22 12:47:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3172


Note You need to log in before you can comment on or make changes to this bug.