Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1625307

Summary: Satellite swaps for large scale with current tuning configuration for Postgres
Product: Red Hat Satellite Reporter: sbadhwar
Component: InfrastructureAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: high    
Version: 6.4CC: bkearney, ehelms, inecas, mmccune, sshtein, zhunting
Target Milestone: UnspecifiedKeywords: Performance, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-05 16:45:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
PostgreSQL high memory usage
none
postgresql configuration none

Description sbadhwar 2018-09-04 15:38:16 UTC
Created attachment 1480829 [details]
PostgreSQL high memory usage

Description of problem:
During soak testing for 6.4, we encountered that Satellite started to swap with approximately 35K hosts.
On a further investigation, it seems like PgSQL seems to consume quite a lot of memory(max usage 20G) even with very basic workload running on Satellite(27 hosts registering in concurrent).

The Satellite VM is powered on 20 cores and 24 GB RAM, with hosts being registered to Capsules.

Attaching a screenshot of PgSQL memory usage
Version-Release number of selected component (if applicable):
Satellite 6.4 Snap 18

Comment 1 Mike McCune 2018-09-04 16:10:56 UTC
Can you attach the custom-heira.yaml that you have defined for this instance?

Comment 2 Mike McCune 2018-09-04 21:35:44 UTC
as well as your postgresql.conf .. that would be helpful.

Comment 3 sbadhwar 2018-09-05 03:18:49 UTC
(In reply to Mike McCune from comment #2)
> as well as your postgresql.conf .. that would be helpful.

Hello Mike,
The custom-hiera.yaml is blank for our setup, since we did not perform any kind of tuning to our Satellite manually.
On a discussion with Ewoud, we can see that Satellite now ships with some of the performance guide tunings been applied by default. These tunings ship under the /usr/share/foreman-installer/config/foreman.hiera/tuning.yaml file, whose contents look like this:
[root@sat-s64-vm1 ~]# cat /usr/share/foreman-installer/config/foreman.hiera/tuning.yaml
---
lookup_options:
  postgresql::server::config_entries:
    merge: hash

apache::mod::passenger::passenger_max_pool_size: 12
apache::mod::passenger::passenger_max_instances_per_app: 6
apache::mod::passenger::passenger_max_request_queue_size: 250
apache::mod::passenger::passenger_stat_throttle_rate: 120
apache::mod::passenger::passenger_max_requests: 10000
apache::mod::passenger::passenger_min_instances: 1
apache::mod::passenger::passenger_max_preloader_idle_time: 0

postgresql::server::config_entries:
  checkpoint_completion_target: 0.9
  max_connections: 500
  shared_buffers: 512MB
  work_mem: 4MB

Also, attaching the postgresql.conf with the bug.

Comment 4 sbadhwar 2018-09-05 03:20:14 UTC
Created attachment 1480960 [details]
postgresql configuration

Comment 5 Mike McCune 2018-09-06 15:51:14 UTC
These tunings are correct.

The 20G rss seems high if the overall traffic is low but will examine the specifics in detail.

Comment 7 Eric Helms 2020-11-05 16:45:30 UTC
This soak testing was performed on Satellite 6.4, since then we have added additional tuning configuration and moved to PostgreSQL 12.