Bug 2002693

Summary: increase pulpcore-api worker count and make it configurable
Product: Red Hat Satellite Reporter: Justin Sherrill <jsherril>
Component: InstallationAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: gpayelka, gtalreja, pcreech
Target Milestone: 6.10.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-installer-2.5.2.8-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:13:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Justin Sherrill 2021-09-09 14:03:44 UTC
Description of problem:
we currently configure the api service to only have a single worker:

/etc/systemd/system/pulpcore-api.service

ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application \
          --timeout 90 \
          -w 1 \


Generally this is fine, but we've seen some conditions where this isn't enough and can lead to errors.  I think we should likely increase the default to something higher (4?) and make it configurable.

Comment 1 Eric Helms 2021-09-09 15:26:05 UTC
Created redmine issue https://projects.theforeman.org/issues/33445 from this bug

Comment 2 Bryan Kearney 2021-09-09 16:05:51 UTC
Upstream bug assigned to ehelms

Comment 3 Bryan Kearney 2021-09-09 16:05:54 UTC
Upstream bug assigned to ehelms

Comment 4 Justin Sherrill 2021-09-17 19:54:32 UTC
*** Bug 1995920 has been marked as a duplicate of this bug. ***

Comment 8 Devendra Singh 2021-10-26 15:38:15 UTC
Verified on 6.10 Snap24.

Verification points:

1- Updated the pulpcore api_service_worker_count in the custom-hiera.yaml file.

# cat /etc/foreman-installer/custom-hiera.yaml 
---
# This YAML file lets you set your own custom configuration in Hiera for the
.....................
.....................
#   shared_buffers: 1024MB
pulpcore::api_service_worker_count: 10

2- Ran the satellite installer and value gets updated successfully from default to 10
# cat /etc/systemd/system/pulpcore-api.service
[Unit]
Description=Pulp API Server
................
................
ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application \
          --timeout 90 \
          -w 10 \
          --access-logfile - \
          --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
ExecReload=/bin/kill -s HUP $MAINPID
ProtectSystem=full

3- Remove the changes from custom-hiera.yaml file and ran the installer again and the worker count was successfully restored to their default count
# cat /etc/systemd/system/pulpcore-api.service
[Unit]
Description=Pulp API Server
.......................
.......................
ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application \
          --timeout 90 \
          -w 5 \
          --access-logfile - \
          --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
ExecReload=/bin/kill -s HUP $MAINPID

4- Checked the API service worker count update in the pulpcore->manifest->init.pp file

# grep api_service_worker_count /usr/share/foreman-installer/modules/pulpcore/manifests/init.pp 
# @param api_service_worker_count
  Integer[0] $api_service_worker_count = min(4, $facts['processors']['count']) + 1,

5- Verified it in the latest foreman-installer version.

# rpm -q foreman-installer
foreman-installer-2.5.2.10-1.el7sat.noarch

Comment 11 errata-xmlrpc 2021-11-16 14:13:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702