Bug 1965936

Summary: Huge memory usage when using CV filters in Sat6.9 with pulp-3
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED DUPLICATE QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.9.0CC: dalley, ggainey, myarboro, rchan, ryandeussing, ttereshc
Target Milestone: 6.10.0Keywords: Performance, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
The issue is observed when dependency resolution is enabled. It is not advised to use dependency solving in 6.10 Beta due to performance and memory usage concerns.
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-13 17:43:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2021-05-31 08:01:22 UTC
Description of problem:
Playing with pulp-3 in Sat6.9, there is a trivial reproducer when publishing a CV with a typical CV filter AND depsolving enabled causes 'rq' process/broker to consume arbitrary huge amount of RAM.

This can easily lead to OOM killing 'rq' and CV publish fail.

The CV details:
- use just RHEL7 repo in the CV
- enable dependency solving (this isnt required for this specific minimalistic scenario, but assume one having a CV with multiple repos incl. RHEL7 - then depsolve makes sense)
- apply filters:
  - include all RPMs with no errata
  - include all errata published before 2021-05-01
  (- include all modules until the same date - this can be almost surely ignored)

Publishing this CV, 'rq' consumed 21G RAM until OOM killed it:

May 28 22:23:47 pmoravec-pulp3 kernel: Killed process 10890 (rq), UID 1000, total-vm:21678888kB, anon-rss:21179192kB, file-rss:0kB, shmem-rss:0kB

Disabling depsolve=true, no such issue happens.


Version-Release number of selected component (if applicable):
Sat6.9.2
libsolv-0.7.17-1.el7pc.x86_64
python3-rq-1.5.2-2.el7pc.noarch
python3-pulpcore-3.7.6-1.el7pc.noarch
python3-pulp-rpm-3.10.0-1.el7pc.noarch


How reproducible:
100%


Steps to Reproduce:
1. Sync RHEL7 repo from CDN
2. Create and publish a CV like described above, or with hammer commands in detail:
 content-view create --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --repository-ids=1 --solve-dependencies=yes
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --content-view-filter=include_errata --date-type='updated' --end-date='2021-05-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_yes_include_2021-05-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd

 content-view publish --name CV_RHEL7_yes_include_2021-05-01 --organization-id=1 --async
3. Wait few hours and check /var/log/messages .


Actual results:
'rq' will consume 20G+ memory.


Expected results:
Much less memory requirements of the process / broker.


Additional info:
There is some additional dependency solving problem about (wrongly) unsatisfied dependencies, that *might* be related to this - I will file that in a moment.

Comment 1 Pavel Moravec 2021-05-31 08:21:20 UTC
(In reply to Pavel Moravec from comment #0)
> Additional info:
> There is some additional dependency solving problem about (wrongly)
> unsatisfied dependencies, that *might* be related to this - I will file that
> in a moment.

https://bugzilla.redhat.com/show_bug.cgi?id=1965942

Comment 2 Tanya Tereshchenko 2021-06-08 12:00:45 UTC
Daniel, Grant, I think this is one more item to investigate for the dep solving area.
Thank you!

Comment 3 Daniel Alley 2021-06-09 20:00:28 UTC
We look into this within the next few days.  One thing that does stick out is that the past 2 releases of libsolv (0.7.18, 0.7.19) have multiple fixes for memory leaks, so a good first step would be to see if it can be reproduced with a patched build.

https://github.com/openSUSE/libsolv/commits/master

Comment 7 pulp-infra@redhat.com 2021-09-10 16:11:20 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-10 16:11:22 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 9 Grant Gainey 2021-09-13 17:43:13 UTC
Closing as a dup, this is one symptom of an underlying issue to be resolved.

*** This bug has been marked as a duplicate of bug 2003764 ***

Comment 10 pulp-infra@redhat.com 2021-10-11 20:08:59 UTC
The Pulp upstream bug status is at CLOSED - DUPLICATE. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2021-10-11 20:09:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.