Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2238906 - Long running postgres threads during content-export
Summary: Long running postgres threads during content-export
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.11.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.15.0
Assignee: satellite6-bugs
QA Contact: Shweta Singh
URL:
Whiteboard:
: 2228592 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-09-14 09:12 UTC by Keith Williams
Modified: 2024-04-23 17:14 UTC (History)
14 users (show)

Fixed In Version: pulp-rpm-3.17.20, pulp_rpm-3.18.19, pulp_rpm-3.19.11, pulp_rpm-3.22.6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2250344 2257299 (view as bug list)
Environment:
Last Closed: 2024-04-23 17:14:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
RHEL 7 Hotfix RPM for Satellite 6.11.5 (402.84 KB, application/x-rpm)
2023-10-30 21:00 UTC, Ian Ballou 		no flags Details
RHEL 8 Hotfix RPM for Satellite 6.11.5 (387.37 KB, application/x-rpm)
2023-10-30 21:03 UTC, Ian Ballou 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github pulp pulp_rpm issues 3259 0 None closed Implementation of artifacts_for_version is slow 2023-10-09 12:34:04 UTC
Red Hat Issue Tracker SAT-20082 0 None None None 2023-09-14 09:13:43 UTC
Red Hat Product Errata RHSA-2024:2010 0 None None None 2024-04-23 17:14:28 UTC

Description Keith Williams 2023-09-14 09:12:01 UTC 
Description of problem:
During large (>500GB) content exports, there is a massive increase in time taken for the export and this is due to a very long running postgres thread prior to the export files being written.

This BZ is raised in relation to support case #03609607.

During a 512GB export, 21hours was spent by the following processes before any export files are written:
100% CPU util for: "postgres: parallel worker for PID 12890"
PID 12890: "postgres: pulp pulpcore ::1946170) SELECT"


Version-Release number of selected component (if applicable):
6.11.5.4

How reproducible:
Everytime

Steps to Reproduce:
Testing has been done on a 512GB export that contains:
Repositories: 27 (both Red Hat and EPEL)
RPM Packages: ~166,000
Size of export: 512GB
Command run: time hammer content-export complete cersion --content-view testexport --version 2 --organization-id 1 --chunk-size-gb 200



Actual results:
Time to create: ~25.5hrs

Expected results:
Ideally less than 6hrs to create the export.


Additional info:
Please see the support case #03609607 which has a lot more contextual information.
The customer require being able to produce +2.1TB exports in a timely manner.
Testing of exporting a 2.1TB export that contains: 
Repositories: 151 (both Red Hat and 3rd party repos)
RPM Packages: >200,000
Size of export: 2.1TB
Time to create: ~37hrs

Why we think this needs to be looked at is due to the time taken for a smaller export of 300GB:
Repositories: 9 (both Red Hat and EPEL)
RPM Packages: ~130,000
Size of export: 306GB
Time to create: 3hrs 25mins
There is a non-linear growth in the time taken for larger exports.
Comment 2 Daniel Alley 2023-09-28 02:52:46 UTC 
Hao and I have been looking at this issue. We have found that it is actually somewhat tricky to reproduce, you need a certain combination of repos and a certain scale to get there.

For instance, exporting all of these repos will allow the issue to be reproduced:

Red Hat Enterprise Linux 6 Server Kickstart x86_64 6.10 
Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server 
Red Hat Enterprise Linux 8 for x86_64 - AppStream Kickstart 8.8 
Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8 
Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart 8.8 
Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8 

But removing any one of them will not.  I'm not claiming it's the only combination that will trigger this, just that it's what we stumbled across.  It may not have anything to do with the specific repos but rather the scale, number of RPMs, etc.

Hao uncovered that the likely culprit is the implementation of "artifacts_for_version" in pulp_rpm.  A tweaked implementation yielded significant benefits in a simple python shell test, and we're currently working on testing it with an actual export and working out the best implementation.
Comment 13 Daniel Alley 2023-10-16 13:59:26 UTC 
Updating the fixed-in-version as the additional patch provides another ~30% improvement due to improved queries (or more specifically, less queries).
Comment 14 Daniel Alley 2023-10-16 18:24:50 UTC 
These 3 BZs ought to be tested together as they involve essentially the same setup process

https://bugzilla.redhat.com/show_bug.cgi?id=2238906 (this one)
https://bugzilla.redhat.com/show_bug.cgi?id=2228592
https://bugzilla.redhat.com/show_bug.cgi?id=2244521
Comment 15 Daniel Alley 2023-10-18 13:04:24 UTC 
*** Bug 2228592 has been marked as a duplicate of this bug. ***
Comment 16 Brad Buckingham 2023-10-30 11:29:29 UTC 
Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set.
Comment 17 Ian Ballou 2023-10-30 21:00:30 UTC 
Created attachment 1996266 [details]
RHEL 7 Hotfix RPM for Satellite 6.11.5

A hotfix is now available for Satellite 6.11.5 running on RHEL 7

HOTFIX INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.11.5 server

2. Obtain the Hotfix RPM for Satellite 6.11.5 on RHEL 7 from this attachment

3. # yum install ./tfm-pulpcore-python3-pulp-rpm-3.17.20-2.HOTFIXRHBZ2238906.el7pc.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart
Comment 18 Ian Ballou 2023-10-30 21:03:28 UTC 
Created attachment 1996267 [details]
RHEL 8 Hotfix RPM for Satellite 6.11.5

A hotfix is now available for Satellite 6.11.5 running on RHEL 8

HOTFIX INSTALL INSTRUCTIONS:

1. Take a complete backup or snapshot of Satellite 6.11.5 server

2. Obtain the Hotfix RPM for Satellite 6.11.5 on RHEL 8 from this attachment

3. # dnf install ./python38-pulp-rpm-3.17.20-2.HOTFIXRHBZ2238906.el8pc.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart
Comment 21 Joniel Pasqualetto 2023-11-16 15:07:16 UTC 
Hello

Any chance we could get hotfixes for other Satellite versions other than 6.11?
Comment 22 Daniel Alley 2023-11-16 15:12:57 UTC 
Ideally users that are already on 6.12 would upgrade to 6.13 or 6.14, which would gain them these patches (and many other related ones that also benefit performance).
Comment 23 Joniel Pasqualetto 2023-11-16 16:42:38 UTC 
I wouldn't mind asking my customer to upgrade, if the fixes were released with errata. Unfortunately, that's not the case for this BZ (or the others you linked to be tested together on https://bugzilla.redhat.com/show_bug.cgi?id=2238906#c14 )

At this point, upgrading won't help much.
Comment 24 Daniel Alley 2023-11-17 01:21:52 UTC 
I could have sworn it was already out, but it seems it's just in line for the next z-stream.  OK. 

It's not a "simple" cherrypick because the performance issues are spread across a couple different BZs or many different commits addressing different root problems for one BZ.  The way we handled the hotfix for 6.11 was to ship upstream pulp releases and probably we would do the same here?
Comment 28 Shweta Singh 2024-02-07 18:22:24 UTC 
Verified.

Tested on Satellite 6.15.0 Snap 8.0

Verification Steps: 
1. Enable and sync some  large repos with immediate download policy(~200GB)
2. Create a content view and publish it
3. Perform the content export looking at the time: `time hammer content-export complete library --organization-id 1`

Observation:
The content export time has been significantly reduced.
Comment 31 errata-xmlrpc 2024-04-23 17:14:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010
Note You need to log in before you can comment on or make changes to this bug.