Bug 1787400 - puppet-access_insights_client causes excessive load on Satellite
Summary: puppet-access_insights_client causes excessive load on Satellite
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: RH Cloud - Insights
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.7.0
Assignee: wclark
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-02 17:29 UTC by wclark
Modified: 2023-03-24 16:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1791499 (view as bug list)
Environment:
Last Closed: 2020-04-14 13:38:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 28652 0 High Rejected puppet-access_insights_client causes excessive load on Katello 2020-06-29 18:05:46 UTC
Github redhataccess puppet-access_insights_client pull 6 0 None closed dont use ensure => latest 2020-06-29 18:05:46 UTC

Description wclark 2020-01-02 17:29:21 UTC
Description of problem:

The puppet module puppet-access_insights_client generates excessive load in large Satellite deployments, by using `ensure => latest` rather than using `ensure => present` and relying on normal patching operations to update the package. In one customer case this was causing constant 100% CPU Utilization on Satellite.

The relevant RPM package is redhat-access-insights-puppet, which is shipped in Satellite repositories but (I believe) owned by Insights engineering.

The issue affects all currently shipped versions of the redhat-access-insights-puppet RPM although the logic is modified on the latest version of the module, such that only certain RHEL versions use the `ensure => latest` logic.

This issue would be resolved upstream via the open PR:

   https://github.com/redhataccess/puppet-access_insights_client/pull/6 

Version-Release number of selected component (if applicable):

redhat-access-insights-puppet RPM (all currently shipped versions)

How reproducible:
100%

For the latest shipped version of the RPM, the issue occurs on clients satisfying the below condition which results in using the manifest old.pp:

    if (versioncmp($::operatingsystemrelease, '6.10') < 0 or
      (versioncmp($::operatingsystemrelease, '7.0') >= 0 and versioncmp($::operatingsystemrelease, '7.5') < 0)) {
      $class_name = 'old'

old.pp declares the package with `ensure => latest`, causing extraneous calls to Satellite checking for package updates. At large scale, the issue results in excessive utilization of Satellite resources.

Steps to Reproduce:

Using the 256GB RAM tuning profile for Satellite, running on hardware comparable to a 'very large' Satellite deployment, Install this puppet module on roughly 15,000+ Content Hosts with a mix of RHEL versions.

Actual results:
Puppet module is checking for updates to the insights-client RPM each time it runs, causing constant, excessive traffic on Satellite that results in 100% CPU utilization at scale

Expected results:
Puppet module ensures that the package is present, and relies on normally scheduled patching operations to update the insights-client RPM.

Comment 6 jcallaha 2020-03-16 17:54:42 UTC
Verified in Satellite 6.7 Snap 16

The solution was to update the old manifest to check that packages were installed instead of latest.

cat /usr/share/puppet/modules/access_insights_client/manifests/old.pp
...
) {
  package { $package_name:
    ensure => installed,
  }
...

Comment 7 Bryan Kearney 2020-04-14 13:38:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454


Note You need to log in before you can comment on or make changes to this bug.