Bug 1507498

Summary:	[upgrades] clean_backend_objects takes 5+ hours on large Satellite (50k+ hosts)
Product:	Red Hat Satellite	Reporter:	Mike McCune <mmccune>
Component:	Upgrades	Assignee:	Partha Aji <paji>
Status:	CLOSED ERRATA	QA Contact:	Katello QA List <katello-qa-list>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.3.0	CC:	adahms, bbuckingham, cwelton, egolov, ehelms, inecas, mbacovsk, paji, pcreech, rjerrido, sghai
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	During some upgrades, the clean_backend_objects step can take over five hours, or time out completely. If this occurs during the 6.3 Beta, contact Satellite Engineering through the beta e-mail list for assistance.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-21 16:54:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1533259

Description Mike McCune 2017-10-30 12:35:53 UTC

During the upgrade of a database with a large # of clients the clean_backend_objects step ran for around 5 hours and then eventually died:

foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
Upgrade Step: clean_backend_objects (this may take a while) ...


[DEBUG 2017-10-27 13:28:32 main] foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
[ INFO 2017-10-27 13:28:32 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-10-27 18:25:40 main] rake aborted!
[DEBUG 2017-10-27 18:25:40 main] Errno::ECONNRESET: Connection reset by peer - SSL_connect


during the execution there was a lot of load on the Satellite as we made many API calls into Candlepin to check for missing data.

We need to improve the performance of this task so it doesn't cause undo 6.3 upgrade pain.

Comment 3 Partha Aji 2017-11-03 05:52:03 UTC

Connecting redmine issue http://projects.theforeman.org/issues/21569 from this bug

Comment 4 Satellite Program 2017-11-17 23:11:25 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21569 has been resolved.

Comment 8 Sachin Ghai 2017-12-13 10:14:14 UTC

Verified with upgrade from 6.2z to sat6.3 snap28 using same db as used in comment6. The time taken by clean_backend_objects is approx :100 minutes.  However, with earlier builds same db took 1hour 40 mins.

# cat /var/log/foreman-installer/satellite.log | grep clean_backend_objects
[ INFO 2017-12-13 04:02:24 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-12-13 04:04:01 main] foreman-rake katello:clean_backend_objects COMMIT=true finished successfully!

Comment 9 Sachin Ghai 2017-12-13 13:53:50 UTC

Just to correct and clarify comment8.  The time taken by clean_backend_objects with 6.2.z -> 6.3 snap28 upgrade is approx :2 minutes.. not 100 minutes. So this reduces the overall upgrade time.

Comment 11 Andrew Dahms 2018-02-16 03:31:24 UTC

Setting the 'requires_doc_text' flag to '-'.

Comment 12 Satellite Program 2018-02-21 16:54:17 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336