Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1483677

Summary: Optimize package profile task processing time
Product: Red Hat Satellite Reporter: sthirugn <sthirugn>
Component: Content ManagementAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: high    
Version: 6.2.11CC: andrew.schofield, aperotti, bbuckingham, bkearney, cdonnell, cduryee, chsimon, ddo, dkaylor, fgarciad, hprakash, igreen, inecas, jhutar, jsherril, kabbott, ktordeur, mbacovsk, mmccune, mtenheuv, oshtaier, pdwyer, pghadge, pmoravec, psuriset, rankumar, rfurlan, sbadhwar, swadeley, tbrisker, vivpatil, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1485976 (view as bug list) Environment:
Last Closed: 2017-12-20 17:17:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1485976    
Attachments:
Description Flags
rhel6_hotfix
none
rhel7_hotfix
none
el7 hotfix
none
el6 hotfix
none
el7 hotfix
none
el6 hotfix
none
Sat 6.2.12 el7 hotfix
none
Sat 6.2.12 el6 hotfix
none
verification screenshot 1
none
verification screenshot 2 none

Description sthirugn@redhat.com 2017-08-21 17:18:59 UTC
Description of problem:
Optimize package profile task processing time

Version-Release number of selected component (if applicable):
Satellite 6.2.11

How reproducible:
Always

Steps to Reproduce:
1. It was found that on a heavily loaded Satellite box, the package profile task processing time can be longer which leads to consequent host registration failures.

Actual results:
package profile update tasks take a long time to run

Expected results:
package profile update task processing should be optimized to run faster.

Additional info:

Comment 1 Brad Buckingham 2017-08-21 17:31:58 UTC
Created redmine issue http://projects.theforeman.org/issues/20682 from this bug

Comment 3 Brad Buckingham 2017-08-21 17:41:45 UTC
Some initial changes for this are in the following upstream PR:
https://github.com/Katello/katello/pull/6912

Note: this PR does not currently address cleaning up the duplicates that may exist within the katello_installed_packages table.  Those duplicates would have been caused by having entries added for the same package on N different hosts.

Comment 4 Satellite Program 2017-08-21 18:13:18 UTC
Upstream bug assigned to bbuckingham

Comment 5 Satellite Program 2017-08-21 18:13:21 UTC
Upstream bug assigned to bbuckingham

Comment 7 sthirugn@redhat.com 2017-08-25 21:33:46 UTC
Created attachment 1318342 [details]
rhel6_hotfix

Comment 8 sthirugn@redhat.com 2017-08-25 21:34:09 UTC
Created attachment 1318343 [details]
rhel7_hotfix

Comment 9 sthirugn@redhat.com 2017-08-25 21:51:31 UTC
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.11 ONLY ===

1. Download attached file tfm-rubygem-katello-hotfix-rhel6.tar.xz or tfm-rubygem-katello-hotfix-rhel7.tar.xz depending on your Satellite host OS version
2. Stop service on Satellite
  katello-service stop
3. Extract the tarball and install the packages in Satellite
4. Upgrade Satellite (hotfix has table index creation)
  satellite-installer --upgrade
5. [optional] To clean duplicate installed packages:
  foreman-rake katello:clean_installed_packages

Comment 10 Satellite Program 2017-09-08 18:12:55 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20682 has been resolved.

Comment 11 Justin Sherrill 2017-09-08 18:49:29 UTC
Created attachment 1323848 [details]
el7 hotfix

Comment 12 Justin Sherrill 2017-09-08 18:50:06 UTC
Created attachment 1323849 [details]
el6 hotfix

Comment 13 Justin Sherrill 2017-09-08 19:12:27 UTC
Added new hotfix that adds an additional feature:

After running the "foreman-rake clean_installed_packages" script, a new setting will be flipped to true.  The "bulk_query_installed_packages" setting.  this will flip the queries to using a more efficient bulk query. Same instructions as before.

Comment 18 Justin Sherrill 2017-09-11 16:11:05 UTC
Created attachment 1324512 [details]
el7 hotfix

Comment 19 Justin Sherrill 2017-09-11 16:12:25 UTC
Created attachment 1324513 [details]
el6 hotfix

Comment 20 Justin Sherrill 2017-09-11 16:14:02 UTC
New hotfixes, uploaded.  Simply update from the existing ones using yum

Comment 28 sthirugn@redhat.com 2017-09-15 17:43:08 UTC
*** Bug 1483136 has been marked as a duplicate of this bug. ***

Comment 40 Martin Bacovsky 2017-10-26 14:40:38 UTC
Created attachment 1343800 [details]
Sat 6.2.12 el7 hotfix

Comment 41 Martin Bacovsky 2017-10-26 14:42:30 UTC
Created attachment 1343801 [details]
Sat 6.2.12 el6 hotfix

Comment 42 Mike McCune 2017-10-28 21:50:54 UTC
=== HOTFIX INSTRUCTIONS FOR SATELLITE 6.2.12 ONLY ===

1. Download attached file sat6.2.12-el7-hotfix.tgz or sat6.2.12-el6-hotfix.tgz depending on your Satellite host OS version

2. Stop service on Satellite
  katello-service stop

3. Extract the tarball and install the packages in Satellite

4. Upgrade Satellite (hotfix has table index creation) or optionally if you do wish to not run the full upgrade procedure you may run:

 foreman-rake db:migrate 

otherwise run:

 satellite-installer --upgrade

5. [optional] To clean duplicate installed packages:
  foreman-rake katello:clean_installed_packages

This last step 5 will reduce the # of rows in your database table and improve Satellite performance.

Comment 43 Mike McCune 2017-10-29 03:20:17 UTC
Update to step (4) if you do the optional route of db:migrate:

"""
4. Upgrade Satellite (hotfix has table index creation) or optionally if you do wish to not run the full upgrade procedure you may run:

 katello-service start
 foreman-rake db:migrate 

otherwise run:

 satellite-installer --upgrade
"""

Comment 44 Ilan Green 2017-10-31 09:52:00 UTC
Question
Should the hotfix be applied only to the Satellite server?
No need to apply it to the Capsules?

Thanks

Comment 45 Pavel Moravec 2017-10-31 11:17:43 UTC
(In reply to Ilan Green from comment #44)
> Question
> Should the hotfix be applied only to the Satellite server?
> No need to apply it to the Capsules?
> 
> Thanks

Only to Satellite, since the patched RPMs are katello ones and there is no katello on Capsules.

Comment 50 Satellite Program 2017-11-17 23:16:12 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21691 has been resolved.

Comment 52 jcallaha 2017-12-12 21:39:38 UTC
Verified in Satellite 6.2.13 Snap 1.

Several customer db upgrades have successfully been completed to 6.2.13, with moderate (no more than 1 hour) increases in upgrade time. Subsequent upgrades did not show the same time increase.

Outside of the upgrades, I'm seeing consistently short package profile processing time. Typical times range from around 1 to 4 seconds. See attached verification screenshot for specific processing times. Stressing the satellite pushed the processing time up to around 30 seconds. See second attached screenshot.

Comment 53 jcallaha 2017-12-12 21:40:13 UTC
Created attachment 1366979 [details]
verification screenshot 1

Comment 54 jcallaha 2017-12-12 21:40:40 UTC
Created attachment 1366980 [details]
verification screenshot 2

Comment 59 errata-xmlrpc 2017-12-20 17:17:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3491