Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1507498 - [upgrades] clean_backend_objects takes 5+ hours on large Satellite (50k+ hosts)
Summary: [upgrades] clean_backend_objects takes 5+ hours on large Satellite (50k+ hosts)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Partha Aji
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: 1533259
TreeView+ depends on / blocked
 
Reported: 2017-10-30 12:35 UTC by Mike McCune
Modified: 2019-09-26 14:04 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
During some upgrades, the clean_backend_objects step can take over five hours, or time out completely. If this occurs during the 6.3 Beta, contact Satellite Engineering through the beta e-mail list for assistance.
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:17 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 21569 0 Normal Closed clean backend object takes a long time to run on a foreman instance with thousands of hosts 2020-09-03 12:40:36 UTC

Description Mike McCune 2017-10-30 12:35:53 UTC 
During the upgrade of a database with a large # of clients the clean_backend_objects step ran for around 5 hours and then eventually died:

foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
Upgrade Step: clean_backend_objects (this may take a while) ...


[DEBUG 2017-10-27 13:28:32 main] foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
[ INFO 2017-10-27 13:28:32 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-10-27 18:25:40 main] rake aborted!
[DEBUG 2017-10-27 18:25:40 main] Errno::ECONNRESET: Connection reset by peer - SSL_connect


during the execution there was a lot of load on the Satellite as we made many API calls into Candlepin to check for missing data.

We need to improve the performance of this task so it doesn't cause undo 6.3 upgrade pain.
Comment 3 Partha Aji 2017-11-03 05:52:03 UTC 
Connecting redmine issue http://projects.theforeman.org/issues/21569 from this bug
Comment 4 Satellite Program 2017-11-17 23:11:25 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21569 has been resolved.
Comment 8 Sachin Ghai 2017-12-13 10:14:14 UTC 
Verified with upgrade from 6.2z to sat6.3 snap28 using same db as used in comment6. The time taken by clean_backend_objects is approx :100 minutes.  However, with earlier builds same db took 1hour 40 mins.

# cat /var/log/foreman-installer/satellite.log | grep clean_backend_objects
[ INFO 2017-12-13 04:02:24 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-12-13 04:04:01 main] foreman-rake katello:clean_backend_objects COMMIT=true finished successfully!
Comment 9 Sachin Ghai 2017-12-13 13:53:50 UTC 
Just to correct and clarify comment8.  The time taken by clean_backend_objects with 6.2.z -> 6.3 snap28 upgrade is approx :2 minutes.. not 100 minutes. So this reduces the overall upgrade time.
Comment 11 Andrew Dahms 2018-02-16 03:31:24 UTC 
Setting the 'requires_doc_text' flag to '-'.
Comment 12 Satellite Program 2018-02-21 16:54:17 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336
Note You need to log in before you can comment on or make changes to this bug.