Bug 2221291

Summary:	Sat 6.14 is slow to sync and index repositories
Product:	Red Hat Satellite	Reporter:	Samir Jha <sajha>
Component:	Repositories	Assignee:	Samir Jha <sajha>
Status:	VERIFIED ---	QA Contact:	Sam Bible <sbible>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.14.0	CC:	ahumbe, rlavi, sbible, zhunting
Target Milestone:	6.14.0	Keywords:	Performance, Regression, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-katello-4.9.0.5-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Samir Jha 2023-07-07 18:54:35 UTC

We have new logic dueing repo indexing which looks for orphaned content units after sync and tries to delete them. However, this introduces a lot of slowness in syncs in env with a large number of content units.

https://community.theforeman.org/t/katello-4-9-is-very-slow/34038/9

We need to optimize this operation:
https://github.com/Katello/katello/blob/f9bf7d0065c55e202275de84b9c69b5ebbca9ed0/app/models/katello/repository.rb#L938

and possibly scope it by repo being synced or call this as an async task in finalize to not impact sync performance.

Please clone and make sure we have this flow downstream.

Comment 1 Samir Jha 2023-07-07 18:54:39 UTC

Created from redmine issue https://projects.theforeman.org/issues/36563

Comment 2 Samir Jha 2023-07-07 18:54:42 UTC

Upstream bug assigned to None

Comment 3 Brad Buckingham 2023-07-12 18:29:34 UTC

Moving to POST as upstream PR merged. :)

Comment 5 Sam Bible 2023-07-20 18:45:11 UTC

Verified on: 6.14 - Snap 8

Steps to verify:
1. Spin up 2 Satellite Instances, one of Snap 8, and another of an earlier Snap, like 5
2. Following the instructions here: https://github.com/Katello/katello/pull/10638, create large amount of orphaned content on both Satellites
2. Enable Red Hat Satellite Client 6 for RHEL 8 x86_64 RPMs on both instances.
3. Sync this repo on both instances.

Expected Results:
The repo should sync almost immediately (5seconds) on the newer Satellite, and take a long time (10 mins) on the other Satellite, as it's also cleaning up the Orphaned Content
Checking Katello::Rpm.count in foreman-rake console on both instances should show that the orphaned content created in step 2 is still present on the newer Satellite.

Actual Results:
The repo syncs almost immediately on the newer Satellite, and takes a long time on the other Satellite, as it's also cleaning up the Orphaned Content
Katello::Rpm.count shows that the orphaned content is still present on the newer Satellite.