Bug 1288152 - Errata Install to Content Host takes too long and doesn't scale well
Errata Install to Content Host takes too long and doesn't scale well
Status: ON_QA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.4
Unspecified Unspecified
unspecified Severity high (vote)
: Beta
: --
Assigned To: Adam Price
jcallaha
: PrioBumpPM, Triaged
: 1290867 1418174 1466985 (view as bug list)
Depends On:
Blocks: 1353215 1399395 1343706
  Show dependency treegraph
 
Reported: 2015-12-03 12:01 EST by Brad Buckingham
Modified: 2017-11-27 01:53 EST (History)
33 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1343706 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
foreman-debug: installing errata on single host (707.49 KB, application/x-xz)
2015-12-03 12:05 EST, Brad Buckingham
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2802141 None None None 2016-12-11 10:58 EST
Foreman Issue Tracker 15366 None None None 2016-06-09 11:34 EDT
Pulp Redmine 1462 High CLOSED - CURRENTRELEASE Errata Install to Content Host takes too long and doesn't scale well 2016-06-27 13:01 EDT

  None (edit)
Description Brad Buckingham 2015-12-03 12:01:24 EST
Description of problem:

When applying (installing) errata on a Content Host from the server, it takes a very long time.

For example, installing ~556 errata on a single content host was observed to take ~44 minutes. Most of that time (~36 minutes) was during the 'initiating the install' phase of the task (i.e. executing the pulp consumer content install).  While this may not sound too bad a first glance, it won't scale well as the behavior is linear.  As a result, if there were 100 content hosts, that same action could take ~3 days.

Version-Release number of selected component (if applicable):

pulp-server-2.6.0.17-1.el7sat.noarch
pulp-rpm-plugins-2.6.0.17-1.el7sat.noarch
ruby193-rubygem-katello-2.2.0.76-1.el7sat.noarch

How reproducible:
always

Steps to Reproduce:
1. install Satellite 6.1.4
2. import a manifest
3. enable and sync the RHEL 6Server RPM repo
4. create a content view containing the RHEL 6Server RPM repo and publish it
5. register a RHEL 6.5 client to the above content view
6. initiate an errata install from the UI 
   (Hosts -> Content Hosts -> select the host -> Errata -> scroll through
    all of the errata, select them and 'Apply Selected')

Actual results:

Observe that the errata are installed on the content host; however, it takes a very long time to initiate and execute the task.

Expected results:

We need to investigate why it is taking so long to initiate the errata install and look for ways to optimize it to improve performance.

Additional info:

Attaching a foreman debug from the server where the above scenario was executed.
Comment 1 Brad Buckingham 2015-12-03 12:05 EST
Created attachment 1101863 [details]
foreman-debug: installing errata on single host
Comment 4 Michael Hrivnak 2015-12-23 16:35:46 EST
Done.

Do you have any other info on what the bottleneck was? CPU? Memory? Disk IO?
Comment 5 pulp-infra@redhat.com 2015-12-23 17:00:24 EST
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2015-12-23 17:00:27 EST
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 7 Brad Buckingham 2016-01-08 14:51:00 EST
I do not have additional data at this time.
Comment 8 Jeff Ortel 2016-01-08 15:00:32 EST
Based on https://bugzilla.redhat.com/show_bug.cgi?id=1269509, it looks like this issue has already been resolved in 6.1.5.
Comment 9 Jeff Ortel 2016-01-08 15:21:24 EST
My mistake.  This is server-side and unrelated to: 1269509.
Comment 10 pulp-infra@redhat.com 2016-01-19 16:00:22 EST
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 11 pulp-infra@redhat.com 2016-01-29 12:00:25 EST
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2016-03-21 15:30:26 EDT
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 14 Dennis Kliban 2016-04-25 12:00:00 EDT
The problem stems from the fact that Pulp takes each errata and turns it into a list of packages. As the number of errata grows, the amount of time it takes to translate them into package lists grows. 

The proper solution is to smarten up the Katello agent/pulpplugin for Gofer to be aware of errata as a content type. This way Pulp can completely avoid having to generate lists of packages. Yum is very good at figuring out what packages belong to what errata. 

I have tested this solution with yum 3.4.3 on RHEL7.

I have also tested this with yum 3.2.22 and yum-security 1.1.16 on RHEL 5.10.

I have confirmed that Yum 3.2.22 ans yum-security 1.1.16 have shipped with RHEL 5 since update 5.
Comment 16 pulp-infra@redhat.com 2016-05-16 08:31:03 EDT
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 17 Dennis Kliban 2016-05-17 13:19:50 EDT
The fix for this issue requires:

  - the agent on each content host to be updated
  - yum to be up to date on each content host (https://bugzilla.redhat.com/show_bug.cgi?id=1246026)
  - yum-plugin-security package to be installed on RHEL 5 and 6 content hosts
Comment 18 pulp-infra@redhat.com 2016-05-17 13:30:56 EDT
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 19 pulp-infra@redhat.com 2016-05-26 16:31:43 EDT
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 22 pulp-infra@redhat.com 2016-05-31 13:31:25 EDT
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 23 Mike McCune 2016-06-07 13:52:46 EDT
This is going to require adding a:


Requires: yum-plugin-security

on the katello-agent-package for RHEL5 and RHEL6. We do not need that Requires on RHEL7 as this is built into yum.
Comment 25 Adam Price 2016-06-09 11:34:28 EDT
Created redmine issue http://projects.theforeman.org/issues/15366 from this bug
Comment 26 pulp-infra@redhat.com 2016-06-17 13:31:33 EDT
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 27 pulp-infra@redhat.com 2016-06-27 13:01:54 EDT
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 32 Andrew Kofink 2017-01-24 10:38:28 EST
A possible workaround for anyone who encounters this error is to use remote execution rather than Katello agent to install errata. This requires a bit of setup with asymmetric encryption keys (SSH private/public keys between capsule and clients). Errata installation is performed the same way as before, except the user selects the dropdown arrow on the "Apply Selected" button, then selects "via remote execution" in the Content Hosts > Errata page.
Comment 33 Justin Sherrill 2017-01-24 10:39:09 EST
*** Bug 1290867 has been marked as a duplicate of this bug. ***
Comment 35 John Mitsch 2017-02-28 11:21:57 EST
*** Bug 1418174 has been marked as a duplicate of this bug. ***
Comment 37 David Davis 2017-07-03 10:02:57 EDT
*** Bug 1466985 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.