Bug 1787400

Summary:	puppet-access_insights_client causes excessive load on Satellite
Product:	Red Hat Satellite	Reporter:	wclark
Component:	RH Cloud - Insights	Assignee:	wclark
Status:	CLOSED ERRATA	QA Contact:	jcallaha
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.6.0	CC:	dsynk, ecerqueira, ehelms, pcreech, zhunting
Target Milestone:	6.7.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1791499 (view as bug list)		Environment:
Last Closed:	2020-04-14 13:38:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description wclark 2020-01-02 17:29:21 UTC

Description of problem:

The puppet module puppet-access_insights_client generates excessive load in large Satellite deployments, by using `ensure => latest` rather than using `ensure => present` and relying on normal patching operations to update the package. In one customer case this was causing constant 100% CPU Utilization on Satellite.

The relevant RPM package is redhat-access-insights-puppet, which is shipped in Satellite repositories but (I believe) owned by Insights engineering.

The issue affects all currently shipped versions of the redhat-access-insights-puppet RPM although the logic is modified on the latest version of the module, such that only certain RHEL versions use the `ensure => latest` logic.

This issue would be resolved upstream via the open PR:

   https://github.com/redhataccess/puppet-access_insights_client/pull/6 

Version-Release number of selected component (if applicable):

redhat-access-insights-puppet RPM (all currently shipped versions)

How reproducible:
100%

For the latest shipped version of the RPM, the issue occurs on clients satisfying the below condition which results in using the manifest old.pp:

    if (versioncmp($::operatingsystemrelease, '6.10') < 0 or
      (versioncmp($::operatingsystemrelease, '7.0') >= 0 and versioncmp($::operatingsystemrelease, '7.5') < 0)) {
      $class_name = 'old'

old.pp declares the package with `ensure => latest`, causing extraneous calls to Satellite checking for package updates. At large scale, the issue results in excessive utilization of Satellite resources.

Steps to Reproduce:

Using the 256GB RAM tuning profile for Satellite, running on hardware comparable to a 'very large' Satellite deployment, Install this puppet module on roughly 15,000+ Content Hosts with a mix of RHEL versions.

Actual results:
Puppet module is checking for updates to the insights-client RPM each time it runs, causing constant, excessive traffic on Satellite that results in 100% CPU utilization at scale

Expected results:
Puppet module ensures that the package is present, and relies on normally scheduled patching operations to update the insights-client RPM.

Comment 6 jcallaha 2020-03-16 17:54:42 UTC

Verified in Satellite 6.7 Snap 16

The solution was to update the old manifest to check that packages were installed instead of latest.

cat /usr/share/puppet/modules/access_insights_client/manifests/old.pp
...
) {
  package { $package_name:
    ensure => installed,
  }
...

Comment 7 Bryan Kearney 2020-04-14 13:38:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1454