Bug 1312098

Summary: [RFE] Satellite 6 should ship with larger default configurations to support large client installs
Product: Red Hat Satellite Reporter: Mike McCune <mmccune>
Component: InstallationAssignee: Chris Roberts <chrobert>
Status: CLOSED ERRATA QA Contact: Jan Hutaƙ <jhutar>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.0CC: andrew.schofield, bkearney, cdonnell, chrobert, gkonda, jhutar, kdixon, ktordeur, millard.matt, mvanderw, peter.vreman, psuriset, rbeyel, riehecky, sbadhwar, sreber, stbenjam, sthirugn, vanhoof
Target Milestone: 6.4.0Keywords: FutureFeature, Performance, PrioBumpGSS, Triaged, UserExperience
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: https://projects.theforeman.org/issues/22513
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 15:26:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1312680, 1440879    
Bug Blocks: 260381, 1122832, 1385841    

Description Mike McCune 2016-02-25 17:54:14 UTC
Right now we have a pre-set limit of 225 Content Hosts for QPID which is far too low, see the install guide section:


https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/Installation_Guide/sect-Red_Hat_Satellite-Installation_Guide-Prerequisites.html#sect-Red_Hat_Satellite-Installation_Guide-Prerequisites-Large_deployments

"""
Increasing the Maximum Number of Client Connections
With more than 225 content hosts, qpidd reaches the maximum number of client connections. To increase it, first establish the new value of the limit that is calculated as: 
"""

we should have our installer configure the various system level values to increase the defaults so customers don't hit arbitrary limits with reasonable # of clients.

Ideally we can support up to 30k clients without requiring manual reconfiguration.

Comment 3 Brad Buckingham 2017-06-01 13:57:08 UTC
Hello Pradeep,

Based upon your testing, do you have any specific recommendations on the configurations/properties to tune and their values to support addressing this request?

Thanks!

Comment 5 Stephen Benjamin 2017-06-02 14:03:43 UTC
The 6.1 docs linked in comment 0 are a bit out of date.  6.2 has different requirements, documented at  https://access.redhat.com/documentation/en-us/red_hat_satellite/6.2/html/installation_guide/large_deployment_considerations.

Each item is detailed below. Some are entirely possible in the installer.  The postgresql configuration is complicated, see item #3.  I would not expect that to make 6.3.


1. "Increasing the Maximum Number of File Descriptors for Apache"

Not available. Maybe expose in puppet-foreman? But installer won't wipe out these changes if manually made for now as changes are made via systemd drop-ins.


2. "Increasing the Maximum Number of File Descriptors for qpid"

   PR (pending) https://github.com/Katello/puppet-qpid/pull/64

   Custom hiera usage:

   ::qpid::open_file_limit: 65536

3. "Increasing the Shared Buffer and Work Memory" (postgresql)

   Complicated, because of how puppetlabs-postgresql works.  I wouldn't expect this to make the installer for 6.3

   Other BZ's: https://bugzilla.redhat.com/show_bug.cgi?id=1440879, https://bugzilla.redhat.com/show_bug.cgi?id=1449707,    https://bugzilla.redhat.com/show_bug.cgi?id=1421636

   Upstream discussion: https://groups.google.com/d/msg/foreman-dev/mKELbDvo-TQ/kYRIHYZQDAAJ


3. "Increasing Concurrent Content Host Registrations"

   Partial. See https://bugzilla.redhat.com/show_bug.cgi?id=1449697


4. "Increasing the maximum number of open files for qdrouterd"

   PR (pending) https://github.com/Katello/puppet-qpid/pull/50

   Custom hiera usage:

   ::qpid::router::open_file_limit: 3160

Comment 7 Pradeep Kumar Surisetty 2017-06-07 03:48:45 UTC
1) Tuned profiles

    If its RHEL6, install tuned and setup throuput-performance profile
    If its RHEL7, its taken care
2)  Apache Configuration : Passenger config
    It depends on workloads. 

3) Increase apache open files



4) Increase qpid open files

5) Increase content_action_accept_timeout
 in Administer -> Settings -> Katello
 to more than default (20seconds) so clients have more time to answer

6) Increase qdrouterd open files

7) increase postgresql tunings
   shared_buffer
   max_connections
8) httpd_foreman_ssl_katello_keep_alive: 'On'
9) httpd_foreman_ssl_katello_keep_alive_timeout: '5'
10) httpd_foreman_ssl_katello_max_keep_alive_requests: '0'
11) httpd_foreman_ssl_passenger_max_requests: '1000'
12) httpd_foreman_ssl_passenger_max_preloader_idle_time: '0'
13) httpd_passenger_passenger_max_pool_size

    This needs good capacity planning. increasing it more can cause more CPU usage. For puppet workloads also, it will affect

14) postgres_log_min_duration_statement: '500'
15) Increase smart_proxy open files

Comment 8 Chris Roberts 2017-07-26 16:28:50 UTC
Based on Comment #5 moving this out of Sat 6.3 as its not possible.

2 PRS will be taken out of this 1 merged and another in the works see comment #5

Comment 11 Brad Buckingham 2017-08-08 11:00:02 UTC
*** Bug 1472587 has been marked as a duplicate of this bug. ***

Comment 12 Brad Buckingham 2017-08-08 11:01:20 UTC
Bug 1472587 has been marked as a duplicate of this bugzilla. 
 When addressing this one, please ensure to also cover the configuration parameters specified in: https://bugzilla.redhat.com/show_bug.cgi?id=1472587#c0

Comment 15 Bryan Kearney 2018-01-18 19:26:03 UTC
*** Bug 1244112 has been marked as a duplicate of this bug. ***

Comment 17 Stephen Benjamin 2018-04-26 14:37:48 UTC
*** Bug 1440879 has been marked as a duplicate of this bug. ***

Comment 18 Julio Entrena Perez 2018-06-28 10:57:42 UTC
*** Bug 1440879 has been marked as a duplicate of this bug. ***

Comment 19 Rich Jerrido 2018-07-05 15:16:34 UTC
*** Bug 1326260 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2018-10-16 15:26:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927