Bug 2229788

Summary: Memory Leak occurs when making repeated large queries against Pulpcore API service
Product: Red Hat Satellite Reporter: wclark
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Shweta Singh <shwsingh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.13.0CC: addubey, ahumbe, dkliban, ggainey, iballou, osousa, pcreech, pdwyer, rchan, saydas, shwsingh, vsedmik
Target Milestone: 6.14.0Keywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulpcore-3.22.9,pulpcore-3.28.5,pulpcore-3.21.12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2231496 (view as bug list) Environment:
Last Closed: 2023-11-08 14:20:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
RHEL 8 Hotfix RPM for Satellite 6.13.3 none

Description wclark 2023-08-07 16:40:07 UTC 
The issue occurs when making repeated large queries against the API service. The cache for the Pulpcore API access policy holds references to entire instances of views.

The caching mechanism for the Pulpcore API access policy was introduced in Pulpcore 3.20 and the first Satellite release in which it appears is Satellite 6.13.0
Comment 1 wclark 2023-08-07 17:10:14 UTC 
I created this BZ as a spin-off from from 2122872 to avoid overloading that BZ, which was originally created for a Satellite version that did not have this issue, and already tracks fixes in other components (satellite-installer, via puppet-pulpcore)
Comment 2 Sayan Das 2023-08-08 11:18:37 UTC 
I am not sure about what everyone else feels about handling this scenario via new BZ but even if we are tracking this just for the lru_cache fix, the BZ should be having Version set to 6.13.2 or 6.13.3 but not 6.14.0 .. 

This was an issue reported on 6.13 only by >= 4 customers for sure. 

And perhaps those cases should be identified from the other BZ and attached here as well.
Comment 4 Ian Ballou 2023-08-10 18:50:40 UTC 
Created attachment 1982843 [details]
RHEL 8 Hotfix RPM for Satellite 6.13.3

A hotfix RPM is now available for Satellite 6.13.3 on RHEL 8.

Installation instructions:

1. Take a backup or snapshot of the Satellite server.

2. Download the hotfix RPM python39-pulpcore-3.21.9-2.HOTFIXRHBZ2229788.el8pc.noarch.rpm from the attachment.

3. # dnf install ./python39-pulpcore-3.21.9-2.HOTFIXRHBZ2229788.el8pc.noarch.rpm --disableplugin=foreman-protector

4. # satellite-maintain service restart
Comment 9 Shweta Singh 2023-08-22 14:06:16 UTC 
Verified.

Version Tested: Satellite 6.14.0 Snap 12.0

Verification Steps:
1. Enable few large repos and sync all at once. 
2. Monitor memory usage by pulpcore gunicorn process.

Result:
1. The spike in memory usage comes down once the syncing of repos get completed.
2. The caching has been removed from pulpcore.
Comment 13 errata-xmlrpc 2023-11-08 14:20:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818