Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1965936

Summary: Huge memory usage when using CV filters in Sat6.9 with pulp-3
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.9.0CC: dalley, ggainey, myarboro, rchan, ryandeussing, ttereshc
Target Milestone: 6.10.0Keywords: Performance, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
The issue is observed when dependency resolution is enabled. It is not advised to use dependency solving in 6.10 Beta due to performance and memory usage concerns.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-13 17:43:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2021-05-31 08:01:22 UTC
Description of problem:
Playing with pulp-3 in Sat6.9, there is a trivial reproducer when publishing a CV with a typical CV filter AND depsolving enabled causes 'rq' process/broker to consume arbitrary huge amount of RAM.

This can easily lead to OOM killing 'rq' and CV publish fail.

The CV details:
- use just RHEL7 repo in the CV
- enable dependency solving (this isnt required for this specific minimalistic scenario, but assume one having a CV with multiple repos incl. RHEL7 - then depsolve makes sense)
- apply filters:
  - include all RPMs with no errata
  - include all errata published before 2021-05-01
  (- include all modules until the same date - this can be almost surely ignored)

Publishing this CV, 'rq' consumed 21G RAM until OOM killed it:

May 28 22:23:47 pmoravec-pulp3 kernel: Killed process 10890 (rq), UID 1000, total-vm:21678888kB, anon-rss:21179192kB, file-rss:0kB, shmem-rss:0kB

Disabling depsolve=true, no such issue happens.


Version-Release number of selected component (if applicable):
Sat6.9.2
libsolv-0.7.17-1.el7pc.x86_64
python3-rq-1.5.2-2.el7pc.noarch
python3-pulpcore-3.7.6-1.el7pc.noarch
python3-pulp-rpm-3.10.0-1.el7pc.noarch


How reproducible:
100%


Steps to Reproduce:
1. Sync RHEL7 repo from CDN
2. Create and publish a CV like described above, or with hammer commands in detail:
 content-view create --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --repository-ids=1 --solve-dependencies=yes
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --content-view-filter=include_errata --date-type='updated' --end-date='2021-05-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd

 content-view publish --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --async
3. Wait few hours and check /var/log/messages .


Actual results:
'rq' will consume 20G+ memory.


Expected results:
Much less memory requirements of the process / broker.


Additional info:
There is some additional dependency solving problem about (wrongly) unsatisfied dependencies, that *might* be related to this - I will file that in a moment.

Comment 1 Pavel Moravec 2021-05-31 08:21:20 UTC
(In reply to Pavel Moravec from comment #0)
> Additional info:
> There is some additional dependency solving problem about (wrongly)
> unsatisfied dependencies, that *might* be related to this - I will file that
> in a moment.

https://bugzilla.redhat.com/show_bug.cgi?id=1965942

Comment 2 Tanya Tereshchenko 2021-06-08 12:00:45 UTC
Daniel, Grant, I think this is one more item to investigate for the dep solving area.
Thank you!

Comment 3 Daniel Alley 2021-06-09 20:00:28 UTC
We look into this within the next few days.  One thing that does stick out is that the past 2 releases of libsolv (0.7.18, 0.7.19) have multiple fixes for memory leaks, so a good first step would be to see if it can be reproduced with a patched build.

https://github.com/openSUSE/libsolv/commits/master

Comment 7 pulp-infra@redhat.com 2021-09-10 16:11:20 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-10 16:11:22 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 Grant Gainey 2021-09-13 17:43:13 UTC
Closing as a dup, this is one symptom of an underlying issue to be resolved.

*** This bug has been marked as a duplicate of bug 2003764 ***

Comment 10 pulp-infra@redhat.com 2021-10-11 20:08:59 UTC
The Pulp upstream bug status is at CLOSED - DUPLICATE. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2021-10-11 20:09:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.