Bug 2238906
| Summary: | Long running postgres threads during content-export | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Keith Williams <keiwilli> | ||||||
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Shweta Singh <shwsingh> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 6.11.5 | CC: | ahumbe, dalley, dkliban, ggainey, hyu, iballou, jpasqual, juqiao, osousa, pcreech, rchan, rlavi, shwsingh, zhunting | ||||||
| Target Milestone: | 6.15.0 | Keywords: | Performance, Triaged | ||||||
| Target Release: | Unused | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | pulp-rpm-3.17.20, pulp_rpm-3.18.19, pulp_rpm-3.19.11, pulp_rpm-3.22.6 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 2250344 2257299 (view as bug list) | Environment: | |||||||
| Last Closed: | 2024-04-23 17:14:26 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Keith Williams
2023-09-14 09:12:01 UTC
Hao and I have been looking at this issue. We have found that it is actually somewhat tricky to reproduce, you need a certain combination of repos and a certain scale to get there. For instance, exporting all of these repos will allow the issue to be reproduced: Red Hat Enterprise Linux 6 Server Kickstart x86_64 6.10 Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server Red Hat Enterprise Linux 8 for x86_64 - AppStream Kickstart 8.8 Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8 Red Hat Enterprise Linux 8 for x86_64 - BaseOS Kickstart 8.8 Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8 But removing any one of them will not. I'm not claiming it's the only combination that will trigger this, just that it's what we stumbled across. It may not have anything to do with the specific repos but rather the scale, number of RPMs, etc. Hao uncovered that the likely culprit is the implementation of "artifacts_for_version" in pulp_rpm. A tweaked implementation yielded significant benefits in a simple python shell test, and we're currently working on testing it with an actual export and working out the best implementation. Updating the fixed-in-version as the additional patch provides another ~30% improvement due to improved queries (or more specifically, less queries). These 3 BZs ought to be tested together as they involve essentially the same setup process https://bugzilla.redhat.com/show_bug.cgi?id=2238906 (this one) https://bugzilla.redhat.com/show_bug.cgi?id=2228592 https://bugzilla.redhat.com/show_bug.cgi?id=2244521 *** Bug 2228592 has been marked as a duplicate of this bug. *** Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set. Created attachment 1996266 [details]
RHEL 7 Hotfix RPM for Satellite 6.11.5
A hotfix is now available for Satellite 6.11.5 running on RHEL 7
HOTFIX INSTALL INSTRUCTIONS:
1. Take a complete backup or snapshot of Satellite 6.11.5 server
2. Obtain the Hotfix RPM for Satellite 6.11.5 on RHEL 7 from this attachment
3. # yum install ./tfm-pulpcore-python3-pulp-rpm-3.17.20-2.HOTFIXRHBZ2238906.el7pc.noarch.rpm --disableplugin=foreman-protector
4. # satellite-maintain service restart
Created attachment 1996267 [details]
RHEL 8 Hotfix RPM for Satellite 6.11.5
A hotfix is now available for Satellite 6.11.5 running on RHEL 8
HOTFIX INSTALL INSTRUCTIONS:
1. Take a complete backup or snapshot of Satellite 6.11.5 server
2. Obtain the Hotfix RPM for Satellite 6.11.5 on RHEL 8 from this attachment
3. # dnf install ./python38-pulp-rpm-3.17.20-2.HOTFIXRHBZ2238906.el8pc.noarch.rpm --disableplugin=foreman-protector
4. # satellite-maintain service restart
Hello Any chance we could get hotfixes for other Satellite versions other than 6.11? Ideally users that are already on 6.12 would upgrade to 6.13 or 6.14, which would gain them these patches (and many other related ones that also benefit performance). I wouldn't mind asking my customer to upgrade, if the fixes were released with errata. Unfortunately, that's not the case for this BZ (or the others you linked to be tested together on https://bugzilla.redhat.com/show_bug.cgi?id=2238906#c14 ) At this point, upgrading won't help much. I could have sworn it was already out, but it seems it's just in line for the next z-stream. OK. It's not a "simple" cherrypick because the performance issues are spread across a couple different BZs or many different commits addressing different root problems for one BZ. The way we handled the hotfix for 6.11 was to ship upstream pulp releases and probably we would do the same here? Verified. Tested on Satellite 6.15.0 Snap 8.0 Verification Steps: 1. Enable and sync some large repos with immediate download policy(~200GB) 2. Create a content view and publish it 3. Perform the content export looking at the time: `time hammer content-export complete library --organization-id 1` Observation: The content export time has been significantly reduced. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:2010 |