Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2223165 - Refresh Alternate Content Source fails with HTTP status code 502 error
Summary: Refresh Alternate Content Source fails with HTTP status code 502 error
Keywords:
Status: CLOSED DUPLICATE of bug 2219885
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Alternate Content Sources
Version: 6.14.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-16 04:32 UTC by mithun kalyat
Modified: 2023-08-03 17:09 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-07-26 16:11:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description mithun kalyat 2023-07-16 04:32:50 UTC 
Description of problem:

Configuring alternate content sources on the Capsule server to sync directly from  CDN fails with below error:

1 subtask(s) failed for task group /pulp/api/v3/task-groups/eb69bc9d-88b4-4ad6-8091-77e02c06389a/.
Errors:
 {"reason"=>"Killed by signal 9."}

Eventually:

Error message: the server returns an error
HTTP status code: 502
Response headers: {"Date"=>"Sat, 15 Jul 2023 19:06:50 GMT", "Server"=>"Apache", "Content-Length"=>"341", "Content-Type"=>"text/html; charset=iso-8859-1"}
Response body: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

Capsule is then unavailable:

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([RestClient::Exceptions::OpenTimeout]: Timed out connecting to server) for Capsule https://capsule.example.com:9090/logs)

[root@mkalyat-sat ~]# hammer alternate-content-source info --id 2
ID:                            2
Name:                          test
Label:                         test
Description:                   
Content type:                  yum
Alternate content source type: simplified
Products:                      
 1) Id:              10
    Organization ID: 1
    Name:            Red Hat Enterprise Linux for x86_64
    Label:           Red_Hat_Enterprise_Linux_for_x86_64
Smart proxies:                 
 1) Id:              2
    Name:            capsule.example.com
    URL:             https://capsule.example.com:9090
    Download policy: on_demand


Version-Release number of selected component (if applicable):

Satellite 6.14

How reproducible:


Steps to Reproduce:

   Content --> Alternate Content Sources --> Add Source --> Simplified --> Content type 'yum' --> Select the external Capsule 

Then:

  Content > Alternate Content Sources --> Test_source --> Refresh

Actual results:

Refresh fails and Capsule went down.

Expected results:

Refresh should be successful.
Comment 4 Brad Buckingham 2023-07-17 12:09:38 UTC 
Is this a regression in behavior from Satellite 6.12?  Thanks!
Comment 5 Ian Ballou 2023-07-20 18:28:16 UTC 
Mithun,

An alternate content source refresh in Pulp is more or less the same thing as 'n' on_demand sync tasks happening at the same time on the capsule itself. If a user has every product in an ACS on a capsule, that capsule is thus essentially going to be "syncing" every product at refresh time. The only difference here is that the refresh doesn't download any content to disk ever. Downloads only happen during that actual capsule sync.

Long story short, refresh is going to be a pretty resource-heavy task especially with many products at once.

If you count up all of the repositories that are being refreshed on the capsule and consider the resources taken up if you were to sync that many on a Satellite at once, does it more or less match up? If so then I think we're all good here.
Comment 6 Ian Ballou 2023-07-20 18:35:32 UTC 
To add to my Comment 5, ACS refresh on a capsule shouldn't take more RAM than a Library capsule sync.
Comment 9 Ian Ballou 2023-07-25 13:25:42 UTC 
It makes sense to me that refreshing RHEL 8 + RHEL 9 repositories would OOM on a capsule with only 7 GB of RAM.  I'm going to test that we can refresh those repositories with the minimum suggested RAM on a capsule. If we can, I will close this BZ out as NOTABUG.
Comment 10 Ian Ballou 2023-07-25 20:51:28 UTC 
I wasn't able to refresh these 4 repositories with 12 GB of RAM.  Now I'm questioning though if a capsule could even sync those 4 repos together with only 12 GB RAM.

Daniel, can you (or someone from Pulp) help give us the final word on if it's unreasonable to expect a machine with 12 GB of RAM to refresh 4 sizeable ACSs at the same time? The 4 repos are RHEL 8 AppStream + BaseOS and RHEL 9 AppStream + BaseOS.
Comment 11 Daniel Alley 2023-07-25 23:54:43 UTC 
You can also run this a couple of times throughout the process until the failure, to see which processes are using up the available memory.

printf "%s\n\n" "`top -b -o +%MEM | head -n 22`" >> memory_log.txt
Comment 12 Daniel Alley 2023-07-25 23:57:18 UTC 
>> Daniel, can you (or someone from Pulp) help give us the final word on if it's unreasonable to expect a machine with 12 GB of RAM to refresh 4 sizeable ACSs at the same time? The 4 repos are RHEL 8 AppStream + BaseOS and RHEL 9 AppStream + BaseOS.

It depends on how much other stuff is running in the background I suppose.  Full satellite has a minimum memory requirement of 20gb. I'd like to see some hard numbers first w/r/t how much memory Pulp is using during the refresh, re: my previous comment.
Comment 13 Ian Ballou 2023-07-26 16:11:37 UTC 
I'm closing this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2219885

Both are in regards to refreshing repositories on capsules and running out of RAM.

*** This bug has been marked as a duplicate of bug 2219885 ***
Note You need to log in before you can comment on or make changes to this bug.