Bug 1288152 - Errata Install to Content Host takes too long and doesn't scale well
Summary: Errata Install to Content Host takes too long and doesn't scale well
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.1.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Adam Price
QA Contact: jcallaha
URL:
Whiteboard:
: 1290867 1418174 1466985 (view as bug list)
Depends On:
Blocks: 1353215 1399395 1343706
TreeView+ depends on / blocked
 
Reported: 2015-12-03 17:01 UTC by Brad Buckingham
Modified: 2021-12-10 14:33 UTC (History)
40 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1343706 (view as bug list)
Environment:
Last Closed: 2018-02-21 16:54:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
foreman-debug: installing errata on single host (707.49 KB, application/x-xz)
2015-12-03 17:05 UTC, Brad Buckingham
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 15366 0 Urgent Closed Errata Install to Content Host takes too long and doesn't scale well 2021-02-11 20:47:46 UTC
Pulp Redmine 1462 0 High CLOSED - CURRENTRELEASE Errata Install to Content Host takes too long and doesn't scale well 2016-06-27 17:01:53 UTC
Red Hat Knowledge Base (Solution) 2802141 0 None None None 2016-12-11 15:58:57 UTC

Description Brad Buckingham 2015-12-03 17:01:24 UTC
Description of problem:

When applying (installing) errata on a Content Host from the server, it takes a very long time.

For example, installing ~556 errata on a single content host was observed to take ~44 minutes. Most of that time (~36 minutes) was during the 'initiating the install' phase of the task (i.e. executing the pulp consumer content install).  While this may not sound too bad a first glance, it won't scale well as the behavior is linear.  As a result, if there were 100 content hosts, that same action could take ~3 days.

Version-Release number of selected component (if applicable):

pulp-server-2.6.0.17-1.el7sat.noarch
pulp-rpm-plugins-2.6.0.17-1.el7sat.noarch
ruby193-rubygem-katello-2.2.0.76-1.el7sat.noarch

How reproducible:
always

Steps to Reproduce:
1. install Satellite 6.1.4
2. import a manifest
3. enable and sync the RHEL 6Server RPM repo
4. create a content view containing the RHEL 6Server RPM repo and publish it
5. register a RHEL 6.5 client to the above content view
6. initiate an errata install from the UI 
   (Hosts -> Content Hosts -> select the host -> Errata -> scroll through
    all of the errata, select them and 'Apply Selected')

Actual results:

Observe that the errata are installed on the content host; however, it takes a very long time to initiate and execute the task.

Expected results:

We need to investigate why it is taking so long to initiate the errata install and look for ways to optimize it to improve performance.

Additional info:

Attaching a foreman debug from the server where the above scenario was executed.

Comment 1 Brad Buckingham 2015-12-03 17:05:07 UTC
Created attachment 1101863 [details]
foreman-debug: installing errata on single host

Comment 4 Michael Hrivnak 2015-12-23 21:35:46 UTC
Done.

Do you have any other info on what the bottleneck was? CPU? Memory? Disk IO?

Comment 5 pulp-infra@redhat.com 2015-12-23 22:00:24 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2015-12-23 22:00:27 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 7 Brad Buckingham 2016-01-08 19:51:00 UTC
I do not have additional data at this time.

Comment 8 Jeff Ortel 2016-01-08 20:00:32 UTC
Based on https://bugzilla.redhat.com/show_bug.cgi?id=1269509, it looks like this issue has already been resolved in 6.1.5.

Comment 9 Jeff Ortel 2016-01-08 20:21:24 UTC
My mistake.  This is server-side and unrelated to: 1269509.

Comment 10 pulp-infra@redhat.com 2016-01-19 21:00:22 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2016-01-29 17:00:25 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2016-03-21 19:30:26 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 14 Dennis Kliban 2016-04-25 16:00:00 UTC
The problem stems from the fact that Pulp takes each errata and turns it into a list of packages. As the number of errata grows, the amount of time it takes to translate them into package lists grows. 

The proper solution is to smarten up the Katello agent/pulpplugin for Gofer to be aware of errata as a content type. This way Pulp can completely avoid having to generate lists of packages. Yum is very good at figuring out what packages belong to what errata. 

I have tested this solution with yum 3.4.3 on RHEL7.

I have also tested this with yum 3.2.22 and yum-security 1.1.16 on RHEL 5.10.

I have confirmed that Yum 3.2.22 ans yum-security 1.1.16 have shipped with RHEL 5 since update 5.

Comment 16 pulp-infra@redhat.com 2016-05-16 12:31:03 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 17 Dennis Kliban 2016-05-17 17:19:50 UTC
The fix for this issue requires:

  - the agent on each content host to be updated
  - yum to be up to date on each content host (https://bugzilla.redhat.com/show_bug.cgi?id=1246026)
  - yum-plugin-security package to be installed on RHEL 5 and 6 content hosts

Comment 18 pulp-infra@redhat.com 2016-05-17 17:30:56 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 19 pulp-infra@redhat.com 2016-05-26 20:31:43 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2016-05-31 17:31:25 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 23 Mike McCune 2016-06-07 17:52:46 UTC
This is going to require adding a:


Requires: yum-plugin-security

on the katello-agent-package for RHEL5 and RHEL6. We do not need that Requires on RHEL7 as this is built into yum.

Comment 25 Adam Price 2016-06-09 15:34:28 UTC
Created redmine issue http://projects.theforeman.org/issues/15366 from this bug

Comment 26 pulp-infra@redhat.com 2016-06-17 17:31:33 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 27 pulp-infra@redhat.com 2016-06-27 17:01:54 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 32 Andrew Kofink 2017-01-24 15:38:28 UTC
A possible workaround for anyone who encounters this error is to use remote execution rather than Katello agent to install errata. This requires a bit of setup with asymmetric encryption keys (SSH private/public keys between capsule and clients). Errata installation is performed the same way as before, except the user selects the dropdown arrow on the "Apply Selected" button, then selects "via remote execution" in the Content Hosts > Errata page.

Comment 33 Justin Sherrill 2017-01-24 15:39:09 UTC
*** Bug 1290867 has been marked as a duplicate of this bug. ***

Comment 35 John Mitsch 2017-02-28 16:21:57 UTC
*** Bug 1418174 has been marked as a duplicate of this bug. ***

Comment 37 David Davis 2017-07-03 14:02:57 UTC
*** Bug 1466985 has been marked as a duplicate of this bug. ***

Comment 40 Pablo Hess 2018-02-16 15:27:07 UTC
I've been able to reproduce this issue on Sat 6.2.14.

This Satellite was installed on 6.2.13 and then upgraded to 6.2.14 with no issues.

Then I provisioned 4 hosts (VMs) from Satellite with RHEL 7.3 GA, no 'yum update' during the install. Next, I switched every one of them to RHEL 7Server so they'd have many errata to apply.


One of the hosts then had these errata applicable:

     - 60(sec) 185(bugfix) 26(enh).  I had installed a few additional packages to this host only, prior to this test.

The remaining 3 hosts had these errata:
     - 34(sec) 123(bugfix) 19(enh).


On the Content Hosts page I selected these 4 hosts and did a bulk action to apply all errata.

The 4 tasks emerging from this bulk action (1 task per host) took 35, 77, 82, and 89 minutes to complete. Pulp displayed >100% CPU use for 45 minutes.

This Satellite has 4 cores, total 8 threads. 16 GB RAM. All storage is on a pretty fast sata SSD. Disk I/O was never the bottleneck for this Errata Applicability calculation.

No swapping occurred on Satellite during this whole process.

Comment 41 Satellite Program 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.