Bug 1816464

Summary:	Decreased performance in GenerateApplicability in 6.6
Product:	Red Hat Satellite	Reporter:	Mike McCune <mmccune>
Component:	Pulp	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED ERRATA	QA Contact:	Lucie Vrtelova <lvrtelov>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.6.0	CC:	bkearney, bmbouter, dkliban, egolov, ehelms, ggainey, ipanova, jjeffers, lvrtelov, pcreech, rchan, satellite6-bugs, ttereshc
Target Milestone:	6.7.4	Keywords:	Performance, PrioBumpField, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	pulp-2.21.0.3-1,pulp-2.21.0.4-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-09-30 13:12:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mike McCune 2020-03-24 03:47:37 UTC

Back in the 6.3 era we had a bug where the Pulp monthly maintenance was not being ran:

https://bugzilla.redhat.com/show_bug.cgi?id=1609928

We are seeing reports that GenerateApplicability performance seems to be worse in 6.6 so I did some digging. Part of the diagnosis routine for the above bug was to check the 2 collections and note if there was a significant size difference:

db.repo_profile_applicability
db.consumers

I've checked with an older customer 6.5 database and get the following counts:

> db.repo_profile_applicability.find().size()
61526
>  db.consumers.find().size()
54726
>

Checking with another customer with similar sized database on 6.6 shows a greater than 10x ratio which was causing performance issues in older versions:

> db.repo_profile_applicability.find().size()
863830
>  db.consumers.find().size()
66891
>

I checked and the montly maintenance is being ran and running it manually returns very quickly (5 seconds):

 celery.worker.strategy:INFO: Received task: pulp.server.maintenance.monthly.queue_monthly_maintenance[5a5aefbd-dd52-42b5-8d38-2c3b65053f27]
 celery.worker.strategy:INFO: Received task: pulp.server.maintenance.monthly.monthly_maintenance[7c4e5227-9572-439d-99e0-9ad938c9fc9c]
 celery.app.trace:INFO: [5a5aefbd] Task pulp.server.maintenance.monthly.queue_monthly_maintenance[5a5aefbd-dd52-42b5-8d38-2c3b65053f27] succeeded in 0.0373372323811s: None
 pulp.server.managers.consumer.applicability:INFO: [7c4e5227] Orphaned consumer profiles to process: 2253
 pulp.server.managers.consumer.applicability:INFO: [7c4e5227] Orphaned consumer profiles processed: 2253
 celery.app.trace:INFO: [7c4e5227] Task pulp.server.maintenance.monthly.monthly_maintenance[7c4e5227-9572-439d-99e0-9ad938c9fc9c] succeeded in 5.05477944389s: None

yet the size of repo_profile_applicability remains large.

Comment 3 Mike McCune 2020-03-24 03:49:53 UTC

On 3/23/20 9:07 AM, Tatiana Tereshchenko wrote:
> In 6.6, applicability calculation for modularity was introduced.
> It means that profile counter should be roughly doubled in comparison to 
> the consumer one.
> 
> A situation when the applicability profile is orphaned but not cleaned 
> up is when consumer system has changes in RPMs installed but not in 
> modules and vice versa.
> I wouldn't expect this to happen that often so the numbers of 
> applicability increases that much, except the non-modular repositories 
> case which should be handled separately.

> On the system which you shared, there are many modular applicability 
> profiles for consumers with no modularity content, maybe this case is 
> not handled correctly during the clean up stage.
>

Comment 4 pulp-infra@redhat.com 2020-03-28 12:53:34 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2020-03-28 12:53:35 UTC

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2020-05-08 19:30:40 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2020-05-08 22:14:11 UTC

All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 11 pulp-infra@redhat.com 2020-05-19 14:31:52 UTC

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 12 pulp-infra@redhat.com 2020-05-19 16:32:42 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 22 errata-xmlrpc 2020-09-30 13:12:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.7.4 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4127

Comment 23 Red Hat Bugzilla 2023-09-15 00:30:31 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days