Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1948258

Summary: pulp3: RQ invokes OOM while syncing rhel-7-server-rpms causing the sync task to fail with "Katello::Errors::Pulp3Error : None" error every time
Product: Red Hat Satellite Reporter: Sayan Das <saydas>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Tasos Papaioannou <tpapaioa>
Severity: high Docs Contact:
Priority: medium    
Version: 6.9.0CC: dalley, egolov, ehelms, ggainey, iballou, jsherril, ktordeur, osousa, pmendezh, rchan, smallamp, swadeley, tpapaioa, ttereshc, vsedmik
Target Milestone: 6.10.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-pulp-rpm-3.13.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:10:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayan Das 2021-04-11 06:46:59 UTC 
Description of problem:

Pulp3 is unable to handle the sync of repositories with big metadata and RQ invokes OOM at some point during the same which silently kills the Sync Task with the error "Katello::Errors::Pulp3Error : None"


Version-Release number of selected component (if applicable):
Satellite 6.9.0 Latest Snap [ 19.1 ] with pulp3 enabled
Katello 4.0 + Pulp3


How reproducible:
Always 

Steps to Reproduce:

1. Install the latest snap of Satellite 6.9.0 with 2 vCPU and ~12 or ~15 GB of ram

2. Enable pulp3 on the same setup

3. Enable the "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" repository and attempt to sync the same.


Actual results:

While satellite is idle and only that sync is happening, after some time the sync task will fail with the error "None" or to be specific "Katello::Errors::Pulp3Error : None" as RQ invokes OOM. The OOM logs will be attached to the Bugzilla.

Part of what RQ logs in Syslog,
~~
kernel: Out of memory: Kill process 7509 (rq) score 374 or sacrifice child
kernel: Killed process 7509 (rq), UID 1000, total-vm:6863868kB, anon-rss:5203516kB, file-rss:212kB, shmem-rss:0kB
pulpcore-worker-1: pulp: rq.worker:WARNING: Moving job to FailedJobRegistry (work-horse terminated unexpectedly; waitpid returned 9)
~~


Expected results:

The repository should get synced even if it takes more than the expected amount of time.


Additional info:

The same scenario with pulp 2, will take around 8 hours to sync the repo but It will be finished and we will never get to see any OOM but with pulp3 that is not the case. 

I will add some more info in the private comment of the Bugzilla.
Comment 6 pulp-infra@redhat.com 2021-04-12 22:13:04 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 7 pulp-infra@redhat.com 2021-04-12 22:13:05 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 8 pulp-infra@redhat.com 2021-04-13 14:14:01 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 9 pulp-infra@redhat.com 2021-04-13 15:12:08 UTC 
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Comment 12 pulp-infra@redhat.com 2021-05-14 11:07:41 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 13 pulp-infra@redhat.com 2021-05-14 12:12:06 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 14 pulp-infra@redhat.com 2021-05-18 09:09:34 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 15 pulp-infra@redhat.com 2021-05-18 14:15:37 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 18 pulp-infra@redhat.com 2021-06-01 17:22:59 UTC 
Requesting needsinfo from upstream developer ttereshc because the 'FailedQA' flag is set.
Comment 21 pulp-infra@redhat.com 2021-06-10 18:25:50 UTC 
Requesting needsinfo from upstream developer ttereshc because the 'FailedQA' flag is set.
Comment 24 Daniel Alley 2021-06-10 22:15:47 UTC 
One last question.  Are these just syncs or syncs + publishes?
Comment 31 Stephen Wadeley 2021-06-16 20:02:18 UTC 
*** Bug 1972769 has been marked as a duplicate of this bug. ***
Comment 33 pulp-infra@redhat.com 2021-06-21 17:23:12 UTC 
Requesting needsinfo from upstream developer ttereshc because the 'FailedQA' flag is set.
Comment 34 Tanya Tereshchenko 2021-06-29 10:46:48 UTC 
Resolved in pulp_rpm 3.13
Comment 35 pulp-infra@redhat.com 2021-07-26 07:07:55 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 36 pulp-infra@redhat.com 2021-07-26 07:07:56 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 37 Tasos Papaioannou 2021-07-30 15:45:38 UTC 
Verified on 6.10.0 snap 11.0.

I was able to sync several repos simultaneously, with no errors:
 				
Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7Server
Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server
Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64
Red Hat Satellite Tools 6.9 for RHEL 7 Server RPMs x86_64
Red Hat Enterprise Linux 8 for x86_64 - AppStream RPMs 8
Red Hat Enterprise Linux 8 for x86_64 - BaseOS RPMs 8
Red Hat Satellite Capsule 6.9 for RHEL 7 Server RPMs x86_64
Comment 38 pulp-infra@redhat.com 2021-10-12 14:09:53 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 39 pulp-infra@redhat.com 2021-10-12 14:09:55 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 42 errata-xmlrpc 2021-11-16 14:10:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702