Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1965936 - Huge memory usage when using CV filters in Sat6.9 with pulp-3
Summary: Huge memory usage when using CV filters in Sat6.9 with pulp-3
Keywords:
Status: CLOSED DUPLICATE of bug 2003764
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.0
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: 6.10.0
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-31 08:01 UTC by Pavel Moravec
Modified: 2021-10-11 20:09 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
The issue is observed when dependency resolution is enabled. It is not advised to use dependency solving in 6.10 Beta due to performance and memory usage concerns.
Clone Of:
Environment:
Last Closed: 2021-09-13 17:43:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 9335 0 High CLOSED - DUPLICATE Huge memory consumption when performing depsolving-enabled copies 2021-10-11 20:08:57 UTC
Pulp Redmine 9378 0 Normal NEW Backport memory and performance fixes for dep solving to 3.14.z 2021-09-10 16:11:19 UTC

Description Pavel Moravec 2021-05-31 08:01:22 UTC
Description of problem:
Playing with pulp-3 in Sat6.9, there is a trivial reproducer when publishing a CV with a typical CV filter AND depsolving enabled causes 'rq' process/broker to consume arbitrary huge amount of RAM.

This can easily lead to OOM killing 'rq' and CV publish fail.

The CV details:
- use just RHEL7 repo in the CV
- enable dependency solving (this isnt required for this specific minimalistic scenario, but assume one having a CV with multiple repos incl. RHEL7 - then depsolve makes sense)
- apply filters:
  - include all RPMs with no errata
  - include all errata published before 2021-05-01
  (- include all modules until the same date - this can be almost surely ignored)

Publishing this CV, 'rq' consumed 21G RAM until OOM killed it:

May 28 22:23:47 pmoravec-pulp3 kernel: Killed process 10890 (rq), UID 1000, total-vm:21678888kB, anon-rss:21179192kB, file-rss:0kB, shmem-rss:0kB

Disabling depsolve=true, no such issue happens.


Version-Release number of selected component (if applicable):
Sat6.9.2
libsolv-0.7.17-1.el7pc.x86_64
python3-rq-1.5.2-2.el7pc.noarch
python3-pulpcore-3.7.6-1.el7pc.noarch
python3-pulp-rpm-3.10.0-1.el7pc.noarch


How reproducible:
100%


Steps to Reproduce:
1. Sync RHEL7 repo from CDN
2. Create and publish a CV like described above, or with hammer commands in detail:
 content-view create --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --repository-ids=1 --solve-dependencies=yes
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --content-view-filter=include_errata --date-type='updated' --end-date='2021-05-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd

 content-view publish --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --async
3. Wait few hours and check /var/log/messages .


Actual results:
'rq' will consume 20G+ memory.


Expected results:
Much less memory requirements of the process / broker.


Additional info:
There is some additional dependency solving problem about (wrongly) unsatisfied dependencies, that *might* be related to this - I will file that in a moment.

Comment 1 Pavel Moravec 2021-05-31 08:21:20 UTC
(In reply to Pavel Moravec from comment #0)
> Additional info:
> There is some additional dependency solving problem about (wrongly)
> unsatisfied dependencies, that *might* be related to this - I will file that
> in a moment.

https://bugzilla.redhat.com/show_bug.cgi?id=1965942

Comment 2 Tanya Tereshchenko 2021-06-08 12:00:45 UTC
Daniel, Grant, I think this is one more item to investigate for the dep solving area.
Thank you!

Comment 3 Daniel Alley 2021-06-09 20:00:28 UTC
We look into this within the next few days.  One thing that does stick out is that the past 2 releases of libsolv (0.7.18, 0.7.19) have multiple fixes for memory leaks, so a good first step would be to see if it can be reproduced with a patched build.

https://github.com/openSUSE/libsolv/commits/master

Comment 7 pulp-infra@redhat.com 2021-09-10 16:11:20 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-10 16:11:22 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 Grant Gainey 2021-09-13 17:43:13 UTC
Closing as a dup, this is one symptom of an underlying issue to be resolved.

*** This bug has been marked as a duplicate of bug 2003764 ***

Comment 10 pulp-infra@redhat.com 2021-10-11 20:08:59 UTC
The Pulp upstream bug status is at CLOSED - DUPLICATE. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2021-10-11 20:09:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.


Note You need to log in before you can comment on or make changes to this bug.