Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2003764

Summary: CV-promotion with-filters causes pulp code to do repoclosure once per-rpm-copied, which is Very Nonperformant.
Product: Red Hat Satellite Reporter: Grant Gainey <ggainey>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Griffin Sullivan <gsulliva>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: dalley, ggainey, mawerner, pmendezh, pmoravec, rchan, ttereshc, zhunting
Target Milestone: 6.10.0Keywords: Performance, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:13:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Grant Gainey 2021-09-13 16:15:02 UTC
Description of problem:

Promoting a CV, with filters, for a large repo can take a very long time, a large amoutn of memory, and occasionally will fail with an assert that kills a copy task.

All of these symptoms are caused by Pulp3 attempting to do repo-closure for the destination repository, once **per rpm** being copied.

There are a number of BZs opened against the various symptoms; this report is to consolidate everything against the single-fix.

Version-Release number of selected component (if applicable):

Sat6.10.snap-recent:
- python3-pulp-rpm-3.14.0-1.el7pc.noarch
- katello-4.1.1-1.el7sat.noarch

How reproducible:

Steps to Reproduce:

Best reproducer can be found at https://bugzilla.redhat.com/show_bug.cgi?id=1934545

Actual results:
Copy takes many hours, 20Gb of memory, and at least one copy-task throws an assert and fails.

Expected results:
Copy takes 10s of minutes and 2-5Gb at most, and succeeds.

Additional info:

Comment 1 pulp-infra@redhat.com 2021-09-13 17:14:02 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 2 pulp-infra@redhat.com 2021-09-13 17:14:04 UTC
The Pulp upstream bug priority is at Urgent. Updating the external tracker on this bug.

Comment 3 Grant Gainey 2021-09-13 17:42:22 UTC
*** Bug 1934545 has been marked as a duplicate of this bug. ***

Comment 4 Grant Gainey 2021-09-13 17:43:13 UTC
*** Bug 1965936 has been marked as a duplicate of this bug. ***

Comment 5 pulp-infra@redhat.com 2021-09-14 15:08:23 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2021-09-22 21:06:55 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2021-09-22 22:07:57 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 8 pulp-infra@redhat.com 2021-09-23 00:28:44 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 9 Daniel Alley 2021-09-24 14:46:49 UTC
There are some excellent reproduction / verification steps for this issue at https://bugzilla.redhat.com/show_bug.cgi?id=1994256.  I suggest copying the relevant information from that issue over here and closing it as a duplicate.

Comment 10 Griffin Sullivan 2021-10-08 19:20:39 UTC
Verified in 6.10 snap 22

Publishing a content view with large repos and filters succeeds in 10s of minutes and uses a low amount of memory. 

Steps to Reproduce:
1. Sync RHEL8 BaseOS and AppStream repos
2. Create and publish above described CV, e.g. via below hammer commands (customize the repository IDs 6 and 7 to match your RHEL8 repoIDs):

hammer content-view create --name mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --repository-ids=6,7 --solve-dependencies=yes
 
hammer content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 
hammer content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_errata --inclusion=true --type=erratum
 
hammer content-view filter rule create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --content-view-filter=include_errata --date-type='updated' --end-date='2019-01-01'
 
hammer content-view filter create --organization-id=1 --content-view=mycv_RHEL8_yes_include_2019-01-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 
hammer content-view publish --name=mycv_RHEL8_yes_include_2019-01-01 --organization-id=1 --async

3. Check the CV publish outcome

Expected Results:
Content view publish succeeds without warnings or errors after around 15 minutes.

Actual Results:
Content view publish succeeds without warnings or errors after around 15 minutes.

Comment 11 Pavel Moravec 2021-10-08 20:04:45 UTC
Great work!

Comment 14 errata-xmlrpc 2021-11-16 14:13:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702

Comment 15 Grant Gainey 2022-03-03 21:34:22 UTC
*** Bug 1995232 has been marked as a duplicate of this bug. ***